How do you feel about your H having an OW that he is leaning on? I would guess he feels the same way about you and your OM. As it has already been mentioned for the sake of the R you need to end it. By continuing it you are only dis-respect your H and your R and proving to your H that you are not really serious about trying to rebuild the R.
“Do you want to be RIGHT or want to be LOVED” “You have to have a life to share a life with someone” “When you stop resisting, you start learning”