A quote I remember goes something like this: Privacy is what you want when you go to the bathroom, not in your marriage relationship.
Secrets are what rip a marriage apart, even if it nothing major. It fosters distrust, no respect, and resentment grows.

You are doing the right thing, it sucks, I know.