Well - I guess some of this decision depends on how you are feeling at the moment - if you don't feel strong enough to deal with him right now, then don't - I think a bad interaction is worse than no interaction sometimes.
If you DO feel strong enough to deal with him though - to go and Act As If and have a PMA and light that vibrant glow in your eyes - then it may be an opportunity to be the only one of the three women he emailed who is willing to accept his friendship. So far your lunches have been pretty good lately, right?