You are dealing w/a man/child. He's reliving his childhood and it's tough dealing w/a man who is acting out. From this day forward, mark the things that you do not want him to touch. I know this sounds crazy, but you've got to identify those things and I would even go as far as taking photos so that he can't come back and say "I didn't see any notes on them".
Choose your battles. Yes, you were upset by what he did last night.
Also, keep your answers short and sweet. When he asked you to sleep elsewhere, respond "h, no, I am not sleeping elsewhere. You have two options do the project here or go elsewhere". That is a firm, but simple response. Stand firm and you do not need to explain yourself to him. He already knows the answer.
Also, if it is time for your child to go to bed, why would he even entertain the thought of giving her sweets at that time of night? Your daughter should have taken a bath and had her pjs on and ready to head off the bed.
So, I will say this again, dig deeper for patience, do not engage w/him any more than you have to and choose your battles wisely. He's doing everything he can to get you to argue w/him and he, in his own way, gets a thrill out of your being upset.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.