You are right! You have alot of history with your H. Those memories are there in his brain. They might be hidden right now but they do surface. Things will trigger them.
You are doing better than I was at this point. I still had maybe melt downs. The ow can't give him everything that he has had with you and your kids. He will miss it.