"he thought he should have a say on who lives on the house"
Uh, NO.
Nor should he use your home for his parenting time. You have no idea how unhealthy this plan is. Ugghhh. It is not going to work.
He should not have free access to your home. He should not have a key for it even. Do you have one for his apartment?
He does not get to have an interest in your personal life or your financial decisions. And vice versa. He does not get to have access to your home, and vice versa. This is a divorce. You are not going to be friends anytime soon, if ever.
I'm with Fig, sell the house. Or, forget the loan and put it in your agreement that he doesn't have to pay the first $20,000 of child support in lieu of him getting off the deed for your house NOW and all of his stuff off your property NOW, being sure to specify how many months this is and the exact date on which he will have to start making support payments.
That way, it won't matter for a long time whether or not he loses his job or not, he won't be sending you child support anyway.
