I'm not surprised that distrust on your part feels like you are pushing him away from his side. This stuff can become a self-fulfilling prophesy. Why do people think they can be actively suspicious of the spouse and that it will have no affect?