So today he called and asked if he could come over this evening and hang out with the kids. He also mentioned we could go out to dinner. He later retracted that and said to not tell the kids so that he can make sure he will be here with enough time to go to dinner.
Why is it cool for him to just come and go as he pleases? In my opinion, there should be a clear schedule for when he is over. That will help for you and for the kids to have that clear expectation. Then, next time when he is over, how about you plan to go out somewhere so you arent spending time with him during the "kid-time"?