SPM - It simply feels good . To break contact or give up the dream of being with this OM/OW is as hard and as painful as what you are going through.
Trouble is you dont have to go through it, that other person is probably waiting. Being torn between family and lust is tough but even tougher to give up the person who is presently giving you everything you THINK you need.
A little empathy might be required. Not acceptance but your spouse needs a soft place to land if you are asking them to give it all up.