It's OK to be mad. In fact, I think sometimes it's constructive. The tough, yet oh so important, part is that you can't act on that anger. It can do a lot of damage.
So, this is the part where it's most important for you to get your own life and not worry about what he's doing. Time to focus on you some more.