I think that your doing great! I think that you would want to treat him with basic courtesy. I wouldnt initiate conversation, I would play games with DD that he could easily join into.
Could you leave the invitation out where he would definately see it? Im not sure if you invited him he would come anyway, but if its his idea hes a little more likely to want to come.
Treat him like the guy you see in the grocery store! Not super friendly, but not nasty and mean either!
I guess I gave the wrong finger to the wrong man...