Your nursing son shouldn't be away from you for 2 nights. Your supply will drop even if you pump. Not to mention, how many bottles has he had anyway, and what about latching when he gets back to you?
Ok, I think you should stroke him a bit about wanting to keep the kids for longer amounts of time, but ask him to maybe just plan one night at first, for the sake of the kid's comfort. Then offer that he can come see them anytime at the house and you could go run errands. I wonder if he debates on visiting because he doesn't want "R" talk when he is there. Get him there one time and show him its not tense, that you won't follow him around, etc. I bet he starts coming more. Our aliens don't like to be cornered.
I am sooo not an expert on this, but thought I would throw my ideas out.