It feels to me, from a long ways away, that he is comfortable with the status quo, and therefore is comfortable with you, as long as you seem to be accepting things as they are.
Enforcing the boundary was clearly a good thing. I do think you are right, though, if you don't follow up on it, it will just "go away". But what can you say that would be useful? He'll get defensive, make excuses, and most likely, lie. Then where are you? Grrrrr......
