When we care about somebody, even if they have not behaved well or have treated us badly, it is natural to feel *something* when you know they are having a hard time.
I do think this is where boundaries become very important. It is unreasonable to be used as an emotional sounding board by a WAS when they need support and be left to flounder when they are feeling okay.
It is a good thing both you and your H are in IC. IC is not a magic fix and is a very slow process. You cannot fix your H's issues and transgressions and he cannot fix yours. You both need to give one another space to do the individual work.
I understand the anger side of it. My H could have cared less when I was beyond hurt, shocked and a million other things. In fact, he became more cruel. But when he needed me and I wasn't there ASAP he got very angry.
It might take a while and a bit of trial and error but with the help of both your IC's, space, detachment and boundaries things might not seem so one sided and hazy.