Try to keep in mind that it is 50/50 - half weekends for you, half for him. Do not think about using it because OW. It really does not make any sense. Any day is a day for OW really.
That is not important now, your kids mental health is important and your own time to have a decent life is important too.
It's not about him and what he does with his time. Try to be fair and he will also see it. He has a person that is willing to destroy a family, cheat on her best friend. One day this will all play big time inside of his potato head.
Be honest and caring, I am sure he will think about this. You are not needy, not make an issue out of proportion, not interested in ruining his life. Be generous. What you give today will come back to you tomorrow.
Be patient...have patience... learn patience...
It will take some time for him to see all this, to see all what he did and is doing. You know already that his family will respect what he is doing just because he is an adult. But they do not agree with it.
He knows you do not want this but you will respect his mistakes.
At some point, their R will start to rotten, I have that almost at 100% sure. Let that play alone while you get better, more confident, more independent, happier.
That confidence that is building inside of you, will be the 1st thing that will attract him. Don't be available, he scares, mysterious. Let time work on your M right now.
Don't give him any reason to blame you for his life being a hell. Let him and the scum bag live hell by themselves and one day you will see the wet dog stand right before you feeling all remorseful.
Believe and wait. Be clean in your heart. Do what is best for your kids and yourself and respect your H's space.