If y'all both agree that your marriage is on the rocks, you may want to sit down with him and see what each of you can do to fix it. If he pulls back and does not want to work on the marriage, I would not call that a smoking gun for the affair, but maybe another clue.
When I was deeply involved in my affair, I did not want to work on the marriage. Why would I? If I had a happy marriage why would I need the OW.
But on the flipside, men usually don't like to talk about emotional or marital problems anyway.
If my wife has flat out asked me, I would have lied. I was already living a lie. What is one more.
I know when my wife and I first started having problems years ago, I probably gave her clues that I was having an affair. I was not. I just did not want to be around her and I was immature. I did not want to be married and was just hanging with the guys.
However he did kiss a girl on the lips and if I recall you said he likes female companionship. If he is not in an affair, all the elements are there for one to flourish: Free time Unhappy marriage Females The formula is really pretty simple.
May or may not be what you wanted to hear, but I hope it helps. You will have to sift through all this advice and make the decision for yourself. Good luck.