Hi sis, I know where you are. You are better than having nothing and soooo far away from having what you need and want. At the same time, you are scared any kind of "radical" 180s could mean you lose even the little you get, which you need to hold on to emotionally, or even risk the possibility of the kids getting upset. And believe me, I am not talking from a very different position here.
I wouldnt go to his sister's house. I would be extra friendly to his family on the phone, call or something, but I would NOT use a silly excuse, just say the truth "I love you guys but things are a bit weird, I am sure you understand" and NOT go back on your word. Dont let anyone convince you to go. IMPORTANT : you do that NOT looking hurt or upset or sad or trying to make him feel guilty, show him what his choices are causing etc etc (you know the drill).
Set up the calendar, put your dates of appointments up and next time he comes, let him fill the rest.
Get some new clothes (with his card preferably), change your perfume, and FIND yourself. The Bbj that was confident and innocent and felt "safe" before things took a toll on you. Find your humor, your mojo, yourself. Be the way you want to be. Do in your house what YOU want to do. Forget him and what he likes or dislikes. Dont be hostile or mean. You have kids, you love him, your goal is to get him back the way he should be back. So, again IMO, you can always have dinner one night together, but when he says I will meet you at the karate, say fine, I need to go to get my hair done, so you stay with the kids and when he joins you, no matter how sweet he treats you, you leave, kiss him on the cheek and WALK OUT all bubbly and cheerful!! Make him wonder...
I will think some more. But again, that's only me and my Greek logic... love K