By the time I have backed out the cost of selling, my retirement and the principal that I have paid down he will get very little equity
The cost of selling - good if you can get it, but not how it is usually calculated. Just a heads up.
Your retirement - again, if you were married at the time that you cashed in your retirement for the down payment, in my community property state at least, that would have been considered a joint decision and the money converted to community property once it was mingled into a joint asset (would be different if that occurred before marriage). Again, good luck with that but it may not happen.
The principal you have paid down - was he contributing anything to support during that time? If so, an argument might be made that the money he was paying was for family support and therefore partially applied to the mortgage.
Not saying any of this is fair or right, and I'm certainly not a lawyer - just a heads up that none of these issues are clear cut.
