OrangeK, too bad your wife is so immature and selfish to do this to her child. That other man won't be around forever and it's going to be so hard on your son when she switches partners again.
I was remembering how my mother was a really bad mother when I was a kid. My parents were married but my mother always had other people helping with my brother and I or she'd just leave us unsupervised. She was pretty abusive and just generally unstable. I no longer speak with her aside from general conversation in group settings. My father was nice, but I blame him a lot for staying quiet when he saw my mother treating us bad. He was afraid of my mother and he'd let her do whatever she wanted to avoid a fight. So he enabled her abusive behavior.
The point is that you want to make sure when you grow up that your son sees you as someone who defended him and fought for his best interests. It doesn't matter if your wife gets mad at you or says she doesn't care. It's still worth fighting for your son. When he's with you I think it's good to do everything you think is right to show him what a stable home and loving parent is. He's an innocent child who never asked to come into this world. You're his best hope in life.