(((((mindfull))))) Ick.... not enough to say anything for sure, but still creepy.
Checking the phone could be such a habit that day of the week doesn't matter, but he could have checked it before he went in. So why take it in with him? I don't know... could be nothing, and could be something. And certainly is icky.
If you say anything, you could almost write his answer for him... "One of the drivers needed something, I didn't want to disturb you, so I went to the bathroom to text him back." So I don't know how much you gain by saying anything.
I will say that you have known him a long time, if something seems odd, there's a good chance it is. But there are a lot of different flavors of odd, some aren't bad. I guess I've taken the long way to say, "I don't know!"