I wasn't going to start a fight, but I know what you mean: don't give him any reason to use as an excuse for not attending. I guess even if I mention casually and "friendly" that it was a long shopping trip he might take it as "where the he*# have you been?!"
