You need to stand firm about visits taking place during the scheduled time. When he contacts you regarding visits that are not during his scheduled time you need to tell him you will be available during his next scheduled visitation, and give no excuses for why he can't visit when he wants to. Hopefully he will eventually get it! Don't worry about him saying you are unreasonable or keeping baby from him. He has a schedule, he can take advantage of it. No judge in the world would expect you to be available at his whim.
"You can't stop the waves, but you can learn to surf." Jon Kabat-Zinn
