I struggle with this too. Our D should be final Sept9th and the house I live in is the house stbxh grew up in. Even though he now hates it and says the house is a "sad" place, he will come in, make himself a nap and lie down on the couch.
I do think you can say "I'm so glad you and the kids had a great visit! I am uncomfortable with you coming on the house when I'm not here. I hope you understand."
And he should. However, he may get snippy and say I've been here before, Yada, Yada. Ignore it and just say "I understand. Please know this just makes me feel more comfortable. I'm sure you can respect that."
3 kids BD 12/15/13 (IDKIILY. ) Rope dropped Cirque du Soleil style D final 9-9-14 "Some people are born on 3rd base and go through life thinking they hit a triple." Barry Switzer