Do not beat yourself up over it. I think what he did was really shifty and snake-like... You were asleep. You were vulnerable, as we all are when we're sleeping. He knew that. He took advantage.
We all want things to go back to "normal" which they never ever will..... But it is totally understandable that when you miss those old days, and you're half asleep, and you have the familiarity of your own H coming toward you wanting you... I think anyone in our shoes would understand it's hard to resist the moment.
But, yeah, a boundary has to be there or it will continue. And then you won't ever sleep well because you'll lie awake and wonder "is he going to come tonight" and have anxiety and fear...that's the emotional/mental terrorism I dealt with a lifetime ago. Don't do it.