Do whatever you are comfortable with. My birthday was WED and I did not invite my W to the party. I do not enjoy being around her and I realized that she does not deserve to be at my special day based on how she a treated me with her wayward ways.
If you think you can do it and be normal with engaging non-relationship conversation then do it. If not then maybe you take your d's out.