Of course you have the right to know, but if you confront him, he'll just lie, and you still won't know. And you'll have tipped your hand that you're suspicious.
I know it's not easy to be counter-intuitive, but it's do-able.
You have to decide for yourself, but there is an advantage to waiting to find out more before you confront him.
You can't control where his focus is, ever. That's not your call.