I hope things work out with you and the house stuff. I don't think I would bring someone to the soccer game. It is just asking for trouble. Just go and have a good time and talk to the other parents and just ignore him if you have to. Let him know you are out and having a good time.
It is good you are letting the kids spend time with their grandfather. As long as you are welcome in his house, continue going. Your H can't stop that. You have built a relationship with him and it is good to keep it. My MIL and I are going Christmas shopping soon. We talk on the phone and she has even told me no matter what happens, I will always be a daughter to her. Brought tears to my eyes. I told her I still have not given up on her son yet.
I hope you have a good day. Try and do something fun for yourself today.