Doodles, I agree with you. There is a whole world of grey in between "I cannot turn my back on my son" and allowing them both in his house, and even entertaining them.
There are ways he can keep his relationship with his son, and not condone his affair. That's just a load o' crap he's giving you.