Next time try & chill in the car until the kids come out & jump in with you. Don't even go in and give him a look-see so to speak. Jump on the phone with someone while you are waiting for your kids to come out. If he asks about why you didn't come in, just say that you had an important call to finish & a lot of things going on. You can have a lot of things going on each time you come to pick up the kids, which will prevent you from having to go inside & have any contact with you at all.
IMO besides the embraces, you going in to make any face-time what-so-ever is just spinning you in the wrong direction!
Now repeat after me:
I am a good person! I will not go inside & make contact with him at all! I will sit in my car & wait for kids to come out! I will drive off & enjoy time at my place with kids! I am a good person!