I think that depends on how you feel about having him in your house and how he would act in your house.
Given that you are very far along with all this (that is, your X will not act entitled), I think it would be all right if you did. As recently as a month ago, X made himself quite at home to the point of trying to change the channel on my TV. I was offended by that. You don't walk into someone else's house and change their TV!
As far as invite him, it seems to me that he needs to make some effort to mend things w/son and it is his responsibility to figure out how to make these arrangements. If he ASKS, then you could agree, for the sake of your son. But I wouldn't volunteer. It should be clear that you are doing him a FAVOR.
M: 16 years Bomb 4/07 OW 20s long gone Divorced 11/09 I remarried New Guy Cooperative r w/X regarding D