i think that lifeboating like that does put a strain on things... Its teh same as your H keeping a book of old girlfriend's phone numbers or something...
I know it sounds hard hearted, but it does not show 100% commitment... but I strongly suspect you started this after you were questioning his commitment which was also not 100%
Did you ever leave him to think he wans't welcome to it?
It's not like a secret, so it may not have done much damage...
