I wouldn't push him any more about leaving OW unless you are willing to stand by what you say. IMO if you keep pushing but don't actually make him stick to following through he gets the message that you don't mean what you say - better not to say it then so that when you do say things you are going to stick to he knows you mean what you say.
If one of his big worries is that things will be just the same as they were before, perhaps the best thing is for you to just carry on and just make very sure when he sees you that you are showing him you are changed by the way you behave. He does sound uncertain about OW but he has to be ready to make the break in order for that to happen; I think I was very lucky with the timing in my sitch that I believe my H was ready to call it a day really when he told me about his A.
I think your idea of going away this weekend and waiting for him to contact you is a good one. Personally, I would go with the expectation that he won't break it off this weekend - that way you won't fall too badly if he doesn't.
If he was really unhappy when he went away with OW before, that perhaps tells you something,; namely that prolonged contact with her doesn't benefit their R. I see that as a positive.
If you don't feel able to go the ultimatum route, (and I get the feeling that is not the route for you at the moment), then go with making sure all your interactions with your H are as positive as they can be - show him you are the better option.
Hang in there.
Saffie me 46 H 46 M in 1986 D20,D18,S16,D13 H's A 01/05 to 07/06 H recommitted to M 07/06 renewed vows 09/06 Going from strength to strength