Won't he feel like I don't care and that I can live my life without him?
What you want is for him to feel like you DO care, but you CAN live your life without him, even though that's not what you wanted.
Does that make sense? If he knows he has you there, pining for him and tolerating his affair, and even willing to have sex with him . . . where's his incentive to end this destructive path he's on?
Have you attempted to set ANY boundaries with him since you found out about his affair?