"Controlling" is the battle cry of every wayward wife (and many wayward husband's too).
Like I said about your EA. It's the side show. A magician's parlor trick with a slight of hand to distract you from focusing on the real problems in your marriage that she has no intention of discussing, changing or accepting responsibility for.
Read up on the term "gas lighting". Essentially, it's a distraction technique whereby they try to act like and convince their "victim" that they are crazy.
Another way to say it. Your wife's best defense is a good offense. By focusing on your "controlling" behavior and taking advantage of your desire to fix things and make your home and family a happier place, your wife manipulates you into allowing her to do whatever she wants lest you be deemed a "controlling jerk". Your continuing guilt from your EA likely made this an easier manipulation.
What should you do about being called controlling? Open the cage door. She's free to go. If you truly are "controlling" you obviously aren't very good at it. It's a lie and you just don't need to believe it, buy into it or let it make you feel guilty anymore to such an extent that you allow HER to control you.
I'm not saying you have to fight her over the issue. You could even sit there and validate it. "So you think I'm controlling...How?, that is an interesting concept". Anything to keep her talking and while you meet her emotional needs for conversation.
Why can't YOU insist on 50/50 and a 4 day/4 day rotation???? Seems the courts would give it to you in most states these days. Stick to your guns and fight for it.
The internet is 90% complaining and entitlement and I hate it because I deserve better!