I guess the way I see this OM is that...sure it was exciting as any new R is when you were on your own and seeing him for a time. You were still in that "honeymoon" phase with him...everything was still very new and really exciting. I know it was your reality at that time, but think about it....what phase in that R were you in? The infatuation/honeymoon phase. Perhaps forbidden was the wrong phrasing for me to use....but IMPO it's still comparing apples/oranges with your marriage.
Eventually, if you had pursued a long-term R with this OM that R would have settled in as well....all R's eventually do. Now if you two were at that point in your R and also trying to compare it to your M then I'd say you were comparing apples/apples. Right now though, there's no way in hell your H can compete...he's too familiar.
Don't know that anything I'm saying here is helping.