I know how you feel about starting over, even with H. Its like, we know what they are capable of when they become unhappy, so will it happen again? Plus, a different person. Probably someone divorced...that's scary too.
In the end, we will always have a jaded sense of trust, because of being betrayed. I guess we have to tackle those things when we feel more ready.