Sorry aliveandkicking for getting some details wrong.

So you are in a better position...he essentially abandoned the house.

You did right to usher him out. If he's coming to get his stuff, or coming to get the kids, he should be ringing the doorbell and waiting to be invited in. You can still sound friendly (and you should), but be aloof....more like friendly is your thing, rather than you are turning it on for him. Try to smile before you answer the door so it's already on your face.
And let him draw his own conclusions and own these emotions of his. Is it your problem that he thinks something is wrong? Is it your fault that he gets upset when you have other things to do? If he's upset that you aren't doting on him or being standoffish, let him stew in it..don't do or say something to show him that you are still on the hook.

You cannot be lonely if you like the person you're alone with. Dr. Wayne Dyer