Hey B, have to say, I agree with everyone. She only gets to be this age once. And sorry he will feel awkward - his problem, not yours.
He could choose not to go to your house. I know that you are thinking it would be sad for your daughter.But I think she would love her party. And maybe he could step up and do something special with her.
Sweetie, do what you want for your daughter. You will be glad you did.