Melody, you do not have to let your daughter go with him if he will not tell you where he is taking her. By the way, 9pm for a return time for a 4 yr old seems awfully late to me. He's acting like an infant and lashing out at you using your D.
You might need to go ahead and file a separation to stipulate some things like visitation, finances, etc.
