And you know him best, so you are probably right! I think you should change the locks on all the doors and then wait and see what he does. If/When he shows up again, act surprised he's there and then tell him that he needs to call and make pre-arrangements before coming over. Tell him that you need your space and time to heal and work on you and that you can't relax if you are nervous that he is going to be showing up unannounced all the time.
Mind you, be friendly and tell him you don't mean to exile him from your son, but remind him that he doesn't have to worry about you dropping in on him un-announced at the OW's mother's place, so you deserve that same respect.
And then, when/if he calls and comes over, make it a point to be gone!! Out of the house. Even if you just drive to the store, it doesn't matter. Just be gone, and don't tell him where you were.
In fact, his meetings with your son should not be in the house. That house needs to be your space exclusively! Claim it! Make it yours! Be calm and kind, but direct and stand your ground!
I actually met sandycay in person last evening for coffee and we talked for 3 hours! She is a very smart lady, and this is pretty much what she did to with her H, and has been very successful!!
[[[[[[BM]]]]]]
TJ
Me45,H49 D24,S18 M26,T28 Bomb 3/19/08 Sep 6/23/08 EA/PA with Secretary 2007-8 3/2009 H moved in w/OW2 7/2009 Let him go w/Love. 8/2009 Legally Sep'd