It doesn't sound like all this contact is doing you much good. He is confused. And he seems to want you to provide the answer to him. He's frustrated when you validate him because validation acknowledges and respects his feelings, but it doesn't make them your problem. You are leaving him, as well as you can, to navigate his own mid life crisis and he doesn't like it.
Half of me wants to tell you to disappear for a while. To go and get your own flat - or at least take a holiday for a couple of weeks - and leave him to care for his own children and deal with normal family life without you. I think he needs some kind of reality check and all this current situation is achieving is allowing him to wallow in his own self inflicted misery and blame you for it.
You know, if you are going to confront him and have these conversations, it might be more productive to make them specific rather than general and about the concrete rather than the abstract. I know my H responds much better to details and facts rather than emotions and concepts (perhaps a Mars Venus thing? I work in the arts, he's in medicine...). So you could say, 'why am I not allowed to see your flat?' or 'why do you have a second phone' rather than talking about trust and earning trust and feeling mistrustful. And if he gives you flannel, you could just again, tell him the truth in a very low key and factual way. 'I don't believe you and that means I need to go away now and consider my options' or something like that?
I am a talker. I go round the houses. I process by thinking aloud. I find I have better results if I do all that in my own time and go to my H with something clearer and more concise.
