He once said to me that since he wants a "white picket fence family"
Just one hard truth here to think about:
What if, affair aside, this is really just a biological clock issue for him? Maybe when he was first with you, he was willing to concur with your decision not to have kids, because he was young and didn't feel the drive himself. But now that he's older, he's finding that he really DOES want kids and family life.
If so, that's a pretty impassable difference. And in reality, it wouldn't be fair to keep him in a relationship with you if it meant giving up his (newfound) dream of having kids.
I understand that the WAY he's gone about this, with the affair and all, is horrible and painful. But if you love him, and he wants kids and you don't, wouldn't you want to set him free to do that?
(Not discounting, btw, that this is all just mlc and rescuer syndrome, but maybe not. I had a friend here who was happily married for many years to a much younger man. She chose not to have kids and he seemed totally on board with that. Once she reached about 52 (still stunning and quite youthful)and he was about 37, he left to marry and have kids with a younger woman. Sometimes that biological clock is a powerful pull).