A bully is successful only when someone tries to "appease him/her". Bullies are only neutralized when they are punched in the "snot locker". Emotionally punch him in the "Snot Locker"
Set your own agenda. Set your own timetable. Ignore the irritating voice of your XH in your ear. He IS a bully. An abusive, small, petty man. Screw his timetable or his wants or his needs. When was the last time he gave YOU that courtesy? Oh yeah...never.
A social worker at my school has a great phrase for people like your XH and my STBXW. She calls them "Injustice Collectors".
They pick up injustices or times that they have been "wronged" and put them into a container, like pebbles from a beach. They never get rid of any of the pebbles and the container gets heavier by the day.