I am not sure what to tell you so I have been thinking about it for a while before posting. I do know that men (and maybe women too, I am just thinking of men here) sometimes get in pi$$ing contests with each other, esp. with coworkers.
What I mean is, he may have said some of that stuff just to look "tough" in front of his coworker. Doesn't make it right, but doesn't mean he doesn't really want to be with you, either. I am sure his coworkers knew he had moved out and was with OW, right? And I am sure he told them quite a story to justify that to himself...so he is trying to downplay moving back home.
Don't know if that makes sense, I just mean a lot of times guys have to make excuses for their actions to try and save face in front of other guys...
And his comments about the other affair at work? In our minds it would be the appropriate time to say, "I made that mistake and it was a horrible decision, (co-worker) should stop now" or something like that. But since HE had an affair he prob. feels he would be a hypocrite to judge the other person. (My H said as much once when I was criticizing OW, he said it was hard to get mad at her for it b/c he had done the same thing)
Anyway maybe I am way off the mark here but sometimes guys just b.s. with each other....