I think that while you might feel you are over thinking this, I do believe that it is good to look at both, very polar scenarios. Personally, I would not go on the trip.
Is there any reason and way that you could turn the trip into a mini vacation for yourself, travelling alone, meeting your kids and H for the time at the park, and then spending some extra time in the area alone, as a tourist?
