If you think divorce is an easy way out, try something else that is easier. Do not be around when he is there, if he comes and catches you off guard, how about jumping in the shower getting dressed nice and say that you are going out to meet with someone. Go out have a nice time, call a friend, go shopping, grab a bite, anything but what you are doing which is total pursuing over and over again, which he labels as BS.
Time to get a plan a stick with it. You can totally turn this sitch around as long as you use your head instead of reacting to him or your current feelings.