I "leave" because I do not live there anymore and there has only been one occasion in which he asked me to stay. The night of the tractor ride last month. That night I declined because D12 was already having an attitude issue and she also had a friend over for the night. Under those conditions, I felt I was doing the best thing at that time. If it were to happen again, I might choose differently. I might not, though.
It is funny you ask me this because I was asking myself the very same thing on the way home from work tonight. Again though, he has only asked me to stay one time so I don't know why you refer to "each time". Each other time was simply the end of our night and since I don't live there, I'm supposed to come back to my own "home".