I had a similar situation in that my H insisted on coming by every day to see the kids and even watched them in our house during the day while I worked. It was so confusing to me to see him interact with me like nothing ever happened, hang out with the kids on the carpet, do the diaper changes and bottles and then-walk out like nothing ever happened. A family but not a family. It was like having a wound reopened every time it happened.
About 4 months in, I insisted that if he wanted to spend more time around the kids he would need to take them out of the house (park, library, etc) and if they needed down time, he would have to take them to where he now lived-a one bedroom apartment down the street. Let me say, he was not happy about this at all. But it was a necessary boundary. Every time he would say "we" instead of me, make an offhand comment about a possible trip or watching a program together, it would seem like we had made progress and then, the next day-WAM-COLD HUSBAND HATH RETURNED. Once I started setting these boundaries, my PMA dramatically increased. I was able to separate that my H wanted the kids to reconnect with family life, not to reconnect with me. Understanding that distinction made me not jump to any wrong conclusions and allowed me to detach. I only see my husband now for a few minutes a day during child exchange but it is now H that is trying to stay for a few extra minutes to talk and chat and not me trying to prolong things. It has saved me from initiating any ILY discussions or "what are we doing here". Trust me-I know what you are going through but since I have started setting some boundaries ("you can't drop by unannounced-I am going out tonight", or "you can take them on Saturday but don't worry about dropping by until Monday-kids and I have plans..." things have been 100% better. Veterans-would love some feedback here but it appears that, like a lot of WAS, he wants the security of the family without the responsibility. Start making yourself less available to the drop bys and hangouts and I think this may be helpful.