You shouldn't have to worry about trust and shouldn't have to walk on egg shells...
If you don't want to ask immediately about the phone password, than maybe the two of you should have a direct convo around trust...
ie: I want to trust, but I don't want to blindly trust.
Indicate that transparency is key.
Some would call it open and honest, but really it's about transparency. If he tells you every time he has a convo with her, then he's not hiding. You could promise the same.
