I think the letter is a great idea. It will give you the chance to tell him how you are feeling and will put the ball in his court as to how he wants to handle it.
You are right about not being able to see both sides until you find yourself in the middle of it. I have always valued my honesty, but as you can see, I did not work hard enough to uphold it.
As for him not feeling guilty and see the letter as an approval for his behavior, I am sure that will not be how he will take it. No matter what, I believe he is confused, feeling guilty, and must be upset with himself. It is just easier to believe you made him the way he is so he is directing all of that towards you. The letter could help in diffusing some of that.
