I'm seriously so scared that he's going to want to leave again now.
I think that may be the root of the problem. And trust me, I can relate. I don't live in fear H will leave anymore (more like wish he would, but that's my issue), but I do always think that he could do it again. No matter how much he says otherwise. And for you, living in fear is not going to do you any good. You cannot control him, right? If you want to apologize, then apologize. If you really have issues you need to discuss further with him, then do that. Only you know what is best for you.