I appreciate the advise, but its tough. Being invited to hang out with my kids and saying no is tough. Not to mention, when it comes up I'm betting she will tell the kids "I invited your father but he didn't want to come."
Why does that scare you? "Remember that talk about divorce? Mommy and Daddy are divorced and live separately. I love you and celebrate your birthday here. She loves you and celebrates your birthday there." If my ex offered this I would decline unless there were special circumstances.
If it's "normally your day", presumably she has a pick-up and/or drop-off time. For a simple family birthday, I would celebrate during your custody on that day, the day before, or the day after.
