I think that is asking entirely too much of anyone too soon. Perhaps if you both were happily remarried and OM was not the 3rd person in your marriage. Perhaps if several years had passed...

Stay strong. You are doing well to set your boundary.

M:25 years at BD w/ 2 daughters
BD: 5/14
Separated 6/14 - H moved cross country w/OW
D Final 9/17

“I can be changed by what happens to me. But I refuse to be reduced by it.”
― Maya Angelou