I'm glad your h turned up at your S17's university dinner. As for him going deeper into the hole, you didn't cause it. They tend to pop out and emotions remind them of what they had and then the scurry right back into the role. You aren't evil and if he's saying that you look mad, that sounds to me like projection and guilt on his part.
Keep the focus on you and your children. Leave him in the rabbit hole.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.