Don't assume. Maybe he had to get out of there because he just had a fight with the OW. Maybe he has diarrhea and really badly needed to go to the bathroom. Maybe, maybe, maybe - I suggest you Act As IF his mood had nothing to do with you. Act, don't react. Pick out a really nice place for lunch and treat yourself. Go even if he doesn't call you back. Later he may wonder who you went to lunch with!