Absolutely it's scary, but what is worse is remaining with someone who does not actually want to be there. At first you might be elated just to have him there with you, but it won't be long before you'll feel cheated that he is not meeting your needs. You'll get tired of constantly being on guard not to irritate him because he might leave you again. He needs to want to be in the marriage with you for the right reasons, anything else will just cause you even more heartache in the future.