The problem is the usual one..he`s comparing a taboo, secret, sexual affair to a day-to-day partnership... the former always looks better in comparison...
You aren`t plan B, to him marriage is just painful and less exciting... The problem is affairs will die out or turn into day to day partnerships too.. and when they DO turn into partnerships they fail 99% of the time... mainly becuase the partners were immature enough to have an affair and think it could work in the first place...