Oh n2hcke, it's awful that you can't just feel comfortable in his company and that you are always on edge in case he drops another bomb on you.
For me, and I'm certainly no vet, it sounds like you are trying to mind read, which I am guilty of, and that's only going to eat you up. I'm not sure what the answer is to make you less suspicious of him except if he is telling you what he is doing and where he is going that sounds positive. Is it possible for you to ask if you can join in on some of these activities? Would that be a 180 for you?
Me - 47 H - 45 D-16 M - 6 years Separated - May 16
Don't leave me behind can't you see me I'm shining... (Years & Years - 'Shine')