The way to not be controlling but have boundaries is to stay completely out of what he does on his own time and not dictate a schedule or terms, but ask that he does what he said he would.
For instance now, you could text him and ask, "I thought you were coming over this morning - did I misunderstand?" and see what his reply is - it's a non-accusing question. If he says he changed his mind or is disrespectful or flaky, you can draw a boundary by saying, "I'm sorry, but that doesn't work for me. In the future, please be here when you say you will or let me know if you can't. Thanks!"
It's a perfectly legitimate request that you would ask of any friend or family member, right? It's rude and inconsiderate to let people wait around. But you don't say that to him - just look at what you need and express that.
Now if he's used to disrespecting you, you may get fallout from this - but don't let him bully you. He's not there, right? If he gets rude or angry, you can say "Let's talk about this later," or if you don't feel up to that, "Sorry, gotta go, child needs me!" If he argues, you can repeat "Please be here when you say you will or let me know if you have to change it, that's all I'm asking. Thank you." And you can repeat that until he gets so bored that the conversation ends.
I know there's some posts here about boundaries, but I don't have the link - anyone?
M 16 yrs, WH62, P54 3 adult blended kids EA 11/13, BD1 6/14 PA fall 14, BD2 2/15 Piecing 2015, BD3 12/15 Separated 4/16 WH moved OW in 5/16 Divorced 6/15/17