Thank you all. I feel better to hear that my anxieties are reasonable. Its bad enough having the boundaries of trust tested without being put in the role of obsessive posessive husband.
I have now been invited along on the trip. Which is fine until the next time. I think my stand may be that we can be friends as a couple with him and his wife as a couple.