I guess I should have said something to him, maybe I will today like "you seemed like you were a little annoyed/frustrated yesterday" and see how that goes
Just a small point, but IMHO, it's better if you tell him how YOU felt rather than making a statement about how you thought he felt. Do you see how talking about his hypothetical feelings keeps you in hiding instead of exposing yourself?
So think back to the sitch and figure out exactly how you felt and state that to him without accusing him of doing anything wrong. And state some REAL feelings-- take the risk. Avoid "frustrated,"* "disappointed," "confused"-- stuff like that. Tell him you were angry, sad, afraid. Figure out what you DON'T want to admit-- and then tell him that.
Expose yourself, be vulnerable-- without accusing him. Be naked.
___________________ *re: "frustrated"-- a therapist once told me that frustration isn't a feeling; it's the blockage of feeling, and that stuck with me, so I resist using that term. Under frustration, you'll usually find anger, fear, sadness-- something like that, closer to the bone.