I cannot believe he'd risk our family and future over THIS issue.
Personally, I wouldn't, but I'm not sure you really understand what he's thinking. This is not about getting laid. You've been rejecting him for how many years? He took you at your word/deed and took in the message that you don't love him, at least not "that way." TO HIM that makes complete sense. I realize it doesn't make sense to you--how can an entire loving relationship come down to sex?
But for him, it can, because sex was the difference between his love for you and his love for his mother, his sister, his daughter. He probably believes (I'm just going by my own brain here) that sex is the difference between YOUR love for him and your love for your father or a very good friend. If you don't want to do the one thing that sets it apart, then you must not see the difference--and for him, that's a depressing thought.
I'm very curious about this--what did he tell you over the years about how he felt? Was he clear, and you just missed it, or did he assume you must have known what you were doing?
