I don't have it in me to tell him to leave when he drops her off but at least I could pick her up and not worry about the whole thing of him staying at our house.
If this is what works to get you what you need, do it. It almost sounds like you think it weak not to be up in his grill about it, it isn't. You will get to the point where you will say what you need to, until then, you are still taking care of yourself. I have been known to practice saying what I need to say in the mirror. It may sound foolish, but it's gotten me to the point where I can address most things directly and if not pleasantly at least in a civil manner.
Go get your baby girl and play ball. They don't stay small nearly long enough.