One thing I'm noticing is that he doesn't look me in the eye as much when we talk. I don't know if that means anything, it's just odd.
I had that same thing. I kind of didn't think about it much, but he didn't look at me, would look anywhere else really, for about 18 months. Just this past month, he's started looking at me in the eye and talking to me like a normal person, not so angry or distant.
I think trying to figure out what's in their head is a waste of time, and I wouldnt even want to know. My H told me when he started the affair, he was thinking things were way worse than they actually were. I guess they do that so they can feel less guilt maybe. So even if your H did tell you some stuff or that negative type of stuff, with time and perspective he may feel a bit differently down the road. My H also tried for a week or 2 to say he never loved me, but that's crap too. They all say that. Of course that's not true, and please don't even think that for a moment.
So focus on you, and yeah, I think you should enforce your boundaries too. Take care of your children by packing food or snacks with them, but don't be driving over to help parent when he's there. That's his job. And if he is neglectful, I would give him a warning too, but that's something that should make you cut down those visits. And keep track of that kind of stuff, take notes in case you ever need them in the future.