I see your Pastors point, but only you can truly decide what you think is best for your sitch. On the one hand, it might show him all he'll be missing out on. Don't you think he's thought about that though? On the other hand, it is a connection. That slope can be very slippery and quite treacherous. I went for the other hand and in some ways it's better b/c of D14 (got some serious stuff going on there) and she needs to just "hang" in a safe place with him. I know that it's his responsibility etc, and it's looking to change soon, but I would just say be very cautious. It's can be really hard on you esp early on with all the yo-yo effect.
I know that this is early on for you operating out of fear is the norm, but you cna't make decisions based on your fears. What's the worse case scenario? Divorce? Well, you've already got a final date. That's in the works, as it were. Are you afraid he'll get mad and not come around? Sometimes our grief is the last connection we have with them. We're afraid to loose that last thread. Looking out for yourself though is the ticket to turning that thread into a rope.