Did your husband have any symptoms of low self-esteem and depression before the affair? It sounds like he may have bigger problems than the affair. It's a shame he won't go into therapy. My guess is he really needs it. If he is verbally or physically abusive, or using a great deal of alcohol to self-medicate, you should seriously consider if there's a safety risk for you and/or your daughter and separate. Do not take a chance if there is any abuse going on.
There is no arriving, ever. It is all a continual becoming.