Who is unhappy, you or your H? Both? I understand the difficulties caused by an affair. However, they can be overcome. Often times the guilt and shame associated with an affair are a difficult phase for the betrayer to handle. They can become depressed.