Don't assume.
Maybe he had to get out of there because he just had a fight with the OW.
Maybe he has diarrhea and really badly needed to go to the bathroom.
Maybe, maybe, maybe - I suggest you Act As IF his mood had nothing to do with you. Act, don't react. Pick out a really nice place for lunch and treat yourself. Go even if he doesn't call you back. Later he may wonder who you went to lunch with!

Ellie