Full disclosure up front - I do not know your entire sitch however - Sorry but I see MAJOR red flags here of OW contact. Locking himself in the bathroom, texting late into the evening and not wanting you to see, sitting in the car by himself on the premise that the kids are on his nerves. This man is in contact with someone that he doesn't want you to know about and based on his past you have every right to question him on it. Do not apologize or feel bad because you questioned him - in my book it wasn't a mistake. I suspect he didn't want to see your phone because he would feel like he needed to reciprocate and he definitely doesn't want you knowing who he's in communication with.
