Let me rephrase the confusing question.

Do you want to miss out spending time with OW/GF/new girl because of feelings for X?