I feel your anger. When my W was still seeing the OG, she had taken the kids into the office one day when they were off. I found out that the OG was with the kids and playing with them. I flipped. Told her that is was completely irresponsible to be introducing them to him.
You definitely have a right to be angry. He listened when you expressed your anger about your kids seeing the OW. You should make the same point here.