You can't be any other way. It is pretty unacceptable behaviour of him to turn up that late and not to reply. It does sound like he's pretending you don't exist because that makes life easier for him. And if he is as family oriented as you say, it will NOT be easy for him to tell the kids about the OW. He must be feeling horribly guilty. As he should. But it's not your problem. I think I would probably go round him and ask the kids directly what they're up to or what time they are coming home etc. They're old enough to tell you, he's clearly too immature.
And I sooooo feel you on the hugs. Not being able to hug my friends is really horrible, we usually say hello and goodbye with big squeezy hugs and I am missing those so much. I think half the time I want a new man just so I can have some real human physical contact...