"I had to drive across the city to take the kids back after they had been only for 4 hours with him and today ..."
K, I'm wondering why you agreed to do this drive across town - to me it seems to enable a situation you trying to change. What I've learned is to establish good communication you also need to set some boundaries but perhaps I've missed something.