I know it hurts. I wish it didn't . . . but the hurt is a message that something's wrong.
First, you're right not to pester him, of course, but you can tell him how you feel. If you feel like you're being punished for coming home late, you can talk to him about that. It might not solve the problem, but you won't have to carry it around alone. If I were taking wild guesses (and I am!) I would say he probably didn't intend to say anything about you being late. He probably started out to tell you as nicely as he could that he wasn't in the mood, thinking he was going to do the right thing and be honest . . . he probably had the best of intentions. But because he was still carrying some resentment from you being late the night before, once he got rolling it had to come out. Part of him probably feels bad about bringing it out the way he did, and part of him is probably proud of saying something.
Now it's your turn to pick a better moment than he did and tell him how you feel about this. I know you weren't asking for advice, but that's my advice. You're the one in the situation, so if you don't follow it, it's probably because you see something I don't.
I'm glad you're venting here. This is a safe place to talk about these feelings with other people who have them, too.
If you do talk to him about how you feel, don't forget to talk about how much you like the cuddling and kisses. He needs to understand that you're talking about how you feel, not about whether he's a bad guy or how much he screws up.
And don't forget that there are people out here who don't know who you are or where you live, but are pulling for you both and thinking about you both.
