Not having had this experience, and I am not talking about the sex with you ... but the other mlc stuff ... it is easy to say there is no easy answer.
I wish RCR RollerCoasterRider was back to posting to speak to this with her recent experience and extensive reading.
What is easy to read from articles like TheSixStages state that they first reconnect with the kids, old friends that they cut off in the beginning, etc. The spouse is usually the last.
Since you two had begun to be physical again, your H would have sounded like he wanted to return to parts of his old life. Maybe he tried and scared himself off ... for now. Don't over analyze this or anything he says ~ does.
The fact that he is still working on his R with your girl is positive. I say that aside from his R with you. It is good for both of them to restore what he has begun to miss. It is also good for him to feel he is safe and not rejected. Now it is up to him to decide who he is, who he wants to become again. You have shown him he is safe with you.
It sounds like at this point he is the only one that can scare himself away ... if it is true there is no OW. They do fight hard when they think they are losing their toy.
Remember that they live in a different time zone and he may have lapses of time where he doesn't realize he has not been in touch with you so much. Yes, he may just be remembering to call when he remembers he has not enjoyed your physical company lately. It is up to you to be available to that or not.
If you set limits, make demands, or imply conditions he may become withdrawn. He has avoided guilt a long time. If his journey is not yet complete, he will continue to do what he must to continue to avoid that guilt. If he returns before he has faced his actions I am afraid things may not go well for you for long. Be very patient and give him time between your visits to digest his own meaning of those times. Let him decide you are still willing to live without his company if that is what he wants.
Live your life without waiting on his next signal. Maybe he needs to get a "busy" signal from you every now and then.