Your showing him it was ok without him because it WAS ok.
The only reason you say "That makes me feel like I'm telling him it was OK to be off with OW and not with his kids if that makes sense?" - Is because your heart is beginning to creep in again.
I assume you didnt marry a complete idiot, so he will know what he should be doing. Its not your job to make him realise it. Thats his
