You are a beautiful, warm, loving lady who often strives to understand the motivation of others.
We have many similarities. I used to do the same because I see the best in people and have trouble imagining the worst.
For example, you're still holding onto the guilt of the Hawaii incident which was more a highly inappropriate reaction by your spouse. You carry the blame and guilt because of his guilt and deceit. And it was just a statement, probably offhand.. a question about children, right? Step aside and look at it with a different perspective. HE was in the wrong. You are sensitive and caring of another's needs. It's like my former spouse. Even though I kept asking him if something was wrong, he said it was all me. Everything was fine as far as he was concerned. Liar.
Consider seeing a marriage counselor. Tackling long term infidelity and working through the process takes a third party. If you had a broken bone would you fix it yourself or go to a doctor, a professional. All this has to be processed in time. Why not go to someone who has the tools to help you do it right? Seeing a third party will also increase the accountability of your spouse.
The mistress, lover, your former friend is angry because your spouse wants you over her. She feels rejected, angry and lied to (!) by your spouse her long term lover. Her reaction indicates that at least at times he's tried to pull away.. unsuccessfully. He lacked a true commitment to your marriage while he seems to love you. No way in hell would I have a meeting with her. Sheesh.. spew queen, liar and sloppy seconds to boot.
This may sound harsh but you are the one who needs to change if you do not want to be subjected to continual pain. I did, over time, with the help of my counselor. One word... boundaries. Learning to know and listen to my intuition, gut feel. Keep myself in a healthy place first which in turn leads to a healthy perspective. Like when my sixteen year old daughter lashes out because of anxiety or fear, I don't take the guilt. I'll try and talk but when it doesn't work I just stay quiet and watch the guilt bombs land nowhere near me.
You are not a victim. You have rights and expectations. You are worth it, all the rainbows and rescued kittens in the world.
