It sounds like you already know the answer. You need to end the R. It is not healthy for you or for him (or his family/marriage)

It might not be a bad idea to seek counseling, to help you understand why you don't feel worthy enough for someone who is single.

Since most of us here are LBS's we could benefit from your insights as the OW, that could be cathartic for you as well.