Because for all intents and purposes, you KNOW he was there. So you were trying to verbally tell him boundaries that would be extremely difficult to enforce, unless you had neighbors call you about his presence on your property and then called the cops for trespassers.
So as snod said, change locks to protect the home. Otherwise, the confronting will likely just lead to projection and spew, at this time. Those results will do one of two things for you, maybe... put you on the roller coaster or put you in WAS mode where it is just another reason why you do not want to be with him.
If you'd rather be in neither of those modes... then when you receive info regarding him being an idiot... just consider it... file it away for possible reference at some later time (if you want) and then... forget about it...
