SS I know all about being alone and away from family and it [censored].
Here's the thing. You are mind reading and predicting the outcome of the trip and you're taking his feelings (or what you think are his feelings) into consideration. Even to the point that you'd put something that would help you heal second to his feelings.
I think you should go. My therapist friend who isn't my IC said even being in a different place is good for our minds.
We don't really know what will and won't make them withdraw. And I think that doing something (or not doing something) thinking it will keep them closer is attempting to control outcomes.
