After putting some thought into it (and talking to BrandNewDay) I'm going to compromise and let her have him Monday-Wednesday-Friday and any weekend day she wants.
And, instead of her coming to the door each of the mornings to drop him off, she can open the side gate and let him into the yard. We have a dog door so he can get in on his own.
There's no reason I need to see her each morning. It's not healthy for me.
I want him to have the contact, it makes him happy to see her.
However, I think this was all about setting boundaries for me.