Also - this part struck me: “ One time I wasn’t home and he got into an argument with my ex and punched a whole in the bathroom door. Another time he physically went after her and her dad held him back. Always happened when I wasn’t around. Of course, he wouldn’t dare do that in front of me. ”
I know this sounds far fetched, but did her father confirm this story about her brother? Or could she possibly have made parts of it up?
What your ex is doing right now is either A) mentally ill - paranoia or B) sociopathic narcissism. You might want to read The Sociopath Next Door and see if it turns on any light bulbs about her past behavior, before your divorce.