Sounds like he's trying to get you hostile. Just let him say whatever he "needs" to say. Just don't get into a confrontation with him about this. He may change his tune tomorrow or next week. If you confront him and it turns into a big hostile situation, he'll just use it to justify everything. The guilt for saying what he said will eventually kick in with him.
That's just my opinion and maybe I'm way off though. Don't let him walk all over you but also, don't let him fuel a fight. NOT fighting with him will annoy him even more.
