ROMB, Concentrate on your GAL'ing. Hopefully there will be times where you are too "busy" for him to come and go as he pleases. Make yourself a little less available.
I don't know if you're already doing this, but you don't have to pick up the phone every time he calls. Maybe sometimes you do, and sometimes you don't. Most likely the times you don't pick up are the times you're not prepared to talk to him.
I agree with IaP, take care of yourself and your D first.
