Sue, you are human, and you have needs like everybody else. Don't scold yourself for enjoying yourself for a little while. Your H isn't all bad, and it is good for the two of you to have a good relationship. You need to set the boundaries that you are comfortable with. There's no way that you will have the "fun" read that "decadent" life your H will. That's why you don't want to be at that apartment. You are in a tough position. It's not for us to judge you. We're here to try to help you think things through.