Gmum - Please get a Lawyer. Your H is doing his best to make sure he is protected, no matter what. Next, he demeans you by implying you'll become lazy living off of him. So, his lifestyle may get to improve, but his D's doesn't? Only if she visits him?
You're married - the debt belongs to you both - but he's forgotten that right now. Of course it's your fault, just like him finding OW is your fault.
He won't quit his job. That would be ridiculous. This is intimidation to get you to stop what you're doing. It's easy to see from out here. Not so easy when you're in the storm. If you get a L, and he does quit his job...you'll still be protected, legally. Lots of WAS try this trick. They all wind up regretting it. Some are forced to work 2 or 3 jobs to make up for the income they lost by quitting a job in order to avoid having to pay more. Judges do not like it.
You need a lawyer to protect you and your daughter. I'm not kidding. WAH is not looking out for either of you. He's looking out for his future with OW.