Are you sure that was his boss calling that he had to rush right back out again after that call. And even more that he had to take the call outside? Maybe some of his anger could be coming from the fact that he hasn't ended it. And the OW could be very well pulling him in another direction. He thinks he is above himself right now and can have both for right now? Just a thought I had but his behavior seems very strange. Maybe he came home to see if there was any consistency to what you just told him. I think that was a test on his part to see if you hug or if you kiss him. To him I guess it showed him all was okay, and you didn't mention it while he was there. So to him maybe you were just venting and now you were over it. So he was free again to do what he wanted.

If this is something you honest;y want I think you will have to try a different approach to get him to leave. He may have gotten use to you saying it but not backing it. Maybe you could write him a letter and ask it of him. Leave the note there for him and take the kiddos and stay at your parents house for a night. Ask him to be please be out by tomorrow. That would give him a whole day to pack and you and the kid's wouldn't be in any direct line of his anger in that moment.
Maybe you could let him know that you feel you should seek legal advice because you have asked him to leave and he won't. That resort would be the last one I would use. Because that resort tends to put a couple at war right from the begining.
In the letter you could tell him all you have stated here. That you are just ready to move on. You want and need to find happiness and someone that would appreciate you and love you and want to be there with you and the boys.

I would also check his phone records if possible. His calls and recent behavior is very questionable. He may very well be lying about it right now. In my opinion you have every right to know if he is still being unfaithful.