Snodderly pointed out to me when I had a sitch along the same lines was that old saying, "He doesn't want you, but he doesn't want anyone else to have you either."
If he doesn't like the fact that you're out with others, well then that's his problem. Let him twist in the wind worrying about it.
