Only you know when you have had enough. When your spouse refuses to even attempt to cooperate with you and honor their marriage vows, you have to consider options even if you would rather not. I agree with HD, get your ducks in a row first. I know that you have been struggling with these issues for years.
It is a very difficult decision that I did not want to make, but I was not given a choice either. My H would rather end the marriage, such as it was, rather than work on improving things between us. You are in the same boat as I have been in for years. They (LDS) don't understand that it's not just the sex, but all of the intimacy that is lacking between you.