Sounds about right he'll stay if there's no one there to make him feel awkward/guilty. Possibly mention to your parents that they should try and make him feel welcome, he's bound to be nervous to see them. It could go a long way in making him realise that you are not making him out to be 'bad'. If he gets on well with them he's perhaps missing them too? Take this as an opportunity.
My H too retreats once he feels we've had a bit too much fun. Perhaps they then feel bad that they are letting the OW down - hee hee. Perhaps the tables will turn and you will be the OW!
Although to be honest I'd be concentrating on my surgery and not worrying about anyone else at all. Good luck
