Quote: Is it ok to say that I am only interested in a date with him when he is OW-free?
Depends on what you want. If you really are not particularly interested in reconciling with him, and wouldn't want him back unless he made near-miraculous changes - then fine, set that boundary,
The truth is, though, that it's the rare WAS who returns without some overlap - they seem to need to keep the OP in their back pocket as a fallback, or else the shame of admitting to the OP that they lied about their spouse and now want to reconcile makes it difficult for them to cut that final tie. So sometimes, setting that boundary makes it seem like an impossible task to the WAS.
Have you tried making him jealous yet? Perhaps you should tell him "sure, I'll go on a date with you. But this weekend's already booked up, I have a date on Friday and one on Saturday. How about next week?"
He doesn't have to know if some of those "dates" are evenings out with the girls