First of all, good for you for finding strength to move on and be happy.
Second, I would just let him know that since he moved out, you would like to be consulted/informed before he drops by. He moved out, he should not have the right to come and go as he pleases unless you have communicated to him that it is ok for him to do so..
"Everyone you meet has baggage. Find someone who loves you enough to help you unpack." ¤Formerly DelinquentGurl¤