No #1 isn't the DB approach. You're not in a relationship with OM's sister. I suggest option: #4 Thank you for keeping S6 and D3 in mind, that was kind. I won't be able to swing that. Our apologies or if you must your #2 .
I personally think because you are working on creating boundaries and adhering to your custody agreement this should be a no, and for absolutely no other reason.
I understand that this is a crappy over all situation, but this is what is going on. Ignoring it doesn't make it go away. Dropping your emotional trauma on OM's sister accomplishes nothing. Even if you were to unload on her you'd feel better in the moment and the like sh!t after the fact.
OM's sister didn't cheat on you nor cheat with you exW. And I don't know enough about your sitch so maybe she even encouraged or supported them during the affair. Which I could see why you'd be wary or even a little hostile about her, but all that's pretty irrelevant now. You're not married any more.
What is relevant is your current situation. OM's sister has kids that your kids probably play with and are familiar with. They may even really enjoy playing together. It was a nice gesture. Is it weird? Yup. Probably a little too soon? Also, a yes. But at some point in the near future this will be your new normal. The question here is are going to turn every opportunity like this for your kids to have a good time into an emotional landmine for you because it's with OM's family or are you going keep working through this so you don't have a visceral reaction to a kid's birthday party invite and can make the decision solely based on the kids availability?
BL I think maybe it's time to head over to Surviving the Big D. I've done this all before, but it's been like 10 years for me, my kid is an adult. It's pretty active over there and there's a lot of people who went through or are going through what you're dealing with. That way you can get more voices to weigh in on this stuff.