If your H elects to not make an effort to see you, it's because he is ashamed in my opinion. If I was carrying on an affair, the last person I would want to see is the person I was cheating on.
Remember, this one person does not define your worth.
Continue to act with dignity and grace and you can't go wrong.