If you're planning on keeping it a secret from your husband, then it's not the right thing.
Everything you've said about how it's gotten to where it is, if you turn it around, sounds like a wayward spouse trying to justify an EA with the whole "We're just friends!" thing.
