It's all fun and fantasy to talk about ditching your marriage and running away - it's something else entirely to have your spouse ditch YOU and walk away.
I think Allen's right - OW won't be able to handle the blubbering WH and may try to send him right back to you.