I wish I could answer this for you, but I can't. You have to do what is best for you. I can only tell you about my sitch, which may be like yours or may be different. My H also told me he was talking to OW and it was "nothing". But my gut told me different. And it was right. Do you believe your H or is your gut telling you otherwise? I think so many times I didn't listen to my instincts because I wanted so badly for things to work out. Do some thinking and keep posting here. You may gain the insight your looking for!
Sorry I wasn't much help, but only you can make this decision!
WC Beth
God grant me the serenity,
to accept the things I cannot change,
To change the things I can,
And the wisdom to know the difference.
- Reinhold Niebuhr