Hmm, I don't think I can give the best advice (I'm sure someone else will be able to), but is there any need for you to have contact with him?
Your kids are grown up, right? There is (what I would imagine) a super minimal amount of co-parenting? I might be wrong - I don't have children. But what kind of conversations do you need to have about the children - by text, email or in person? Can they go and visit him instead of him coming round? So you don't have to see him or speak to him?
Or is there any need for you to be friends with him? Do you respect him? The choices he makes and the person he is? Does he give you the same level of respect back?
Are there things that you didn't get round to doing for yourself the first time round? Things you'd like to try or do? Thinking now might be a good time to try those things.