Meganna, I think you could benefit by distancing from him before he leaves...maybe, I'm going to do the dishes/load the dishwasher while you are here with the kids, just holler when you're headed out. Or sort laundry, or read a book in your room, or go for a walk, or take a shower.
H wants them to think he's at work because (my opinion) he doesnt want to face them asking about him leaving. I think you can just say he's at work when he's at work or that he's at a friend's house and then tell them when he will come see them. A calendar with some simple marking can help them understand when he is coming - I use this with my kids after it was suggested by a family member dealing with a similar issue.
Maybe with the hug he is trying to detach, maybe he forgot, maybe he loves you less, maybe he thinks you hate him, maybe...maybe...maybe...don't worry about it. Get hugs from your kids and family and let him have the space he wants. It'll be ok someday, maybe soon. Try to stay strong.
Faith is, at one and the same time, absolutely necessary and altogether impossible. --Stanislaw Lem