Gosh, that sounds very hard, maybe for both of you. Have a big virtual hug (())

Positives:
he wanted to spend time with you and as a family
he's expressing willingness to work on his own stuff
you had some positive interactions in there along with the scary stuff
you repaired ok a few times by the sound of it

It sounds like it might have been too much for both of you to have so much time together, especially when he is so exhausted.

Things will get better, I have a lot of hope for you! Hang in there, look after yourself and be patient x