Get him to say -- straight up yay or nay -- on Saturday Night -- make him say it, confirm it, so you both can know explicitly that he is being an azz or not.
If he refuses to give you a straight answer then kindly tell him he can wait until the normal scheduled time to pick them up -- and no earlier.
Sheesh! If this guy can't understand decent common courtesy and fair play ...