I find it interesting that you indicate he's asking you to admit you're holding out hope for the M.
Why it is interesting is because, you are getting a lot of standard "script" stuff that WAS will say, stuff that so many of us here have heard in our own sitchs. The point is though, that while we KNOW that they tell the LBS not to have hope, and they mostly feel any R talk is pressuring, it's almost like he's goading you to pressure him.
As far as cake eating, you may want to get clear with your boundaries about what you will allow in your life and what you won't.
There are those LBS who put up very strong boundaries and say no friendship and no real communication except regarding kids and financial. Others have no problem leaving the WAS to come and go as they please. It is completely up to you on what you want to do, so long as it is because that's what you want, as opposed to punishing or trying to force your H to decide to come back to the M.
You will hear others state that the LBS should do their best to "keep the road home paved and smooth". IOW, if the WAS feels it is too difficult to return due to resentments or strong boundaries, they won't.
If you are not sure that the OW is a deal breaker for you, then focus and work on yourself to become and even more awesome catch, that only a fool would leave. You can always change your tune at a later date.
What types of things would you like to get better at in your personal life?
