I think you're allowing him to "cake-eat." You deserve better than to be his "bootie call."
I think he needs to make a decision, and you need to be less available to him and to work on yourself. Since he is with OW, I pray that you are using condoms when you are with him sexually? Please be careful.