Busting, you know what the stories here how difficult it is for WAS to give up OW. It would be too good to be true if your H could just walk away from this relationship with OW of almost 4 (?) years. According to his remarks about you treating him differently, he is working through something. He might be very frustrated because his life is not what he thought it would be, who knows.
I agree with Heather, keep doing what you are doing.
M:50 H:52 S28 (my S from previous marriage) M:17 + 3 BD: 06/12 S: 06/12 - H works in another state