I bet this feeling is very common among people on the board. I also thought my H was always fine bc he never communicated anything negative to me. But he wasn't.
But you know what? You didn't know at the time. You did what you thought you needed to do. And your H is the one who went outside the M, not you. He also played (plays a part.) So don't be so hard on yourself. I tend to be really hard on myself too.