If you allow him to keep calling the shots and jerk you around now, you could end up with a sitch where he comes and goes as he pleases.. shows up at your new place... guilts you into letting him stay when he wants and then disappears again.
Oh my, how scary. This has become my world. Don't let it happen to you. Its harder to 'undo', I can soooo promise you that. Ask him to figure out what days he wants to see D4, and make a schedule. Did you see an atty about temporary custody agreement yet? That way you could get the daycare payment in writing.