I have to agree with you that my H is not better off either. She has a nice house that she bought and renovated and and our house is still a work in progress (no cabinets in the kitchen, bathrooms only partially done, no walls painted etc.)
But, when she puts him out he has to bring his clothes in garbage bags (after two years) and he carries them on the back of his truck like that for days until she lets him come back even though he never fulfils whatever ultimatum she lays down, so who is the fool me or her? At least I have papers on him which is a whole lot more than her.
I am sure that H misses your son. I guess I am lucky that my H is so attached to S3 that as long as we are not travelling there has not been a single day that he has missed seeing him. (except when we went to a hotel during a hurricane). He is also always here for holidays like Christmas etc.
Once you never prevent H from seeing S4 and keep doing well on your own I am sure that when H comes out of his fog he will be seeing you with different eyes.