It sounds to me that your sex drive may be tanking because, in the instance of you visiting him on a Thursday... HE didn't take it up a notch. I think it is fine you are sexually confident, and that you can go 'wherever,' but it doesn't sound to me like he is.... using that to his advantage, especially in a 'leading' manner.
Therefore, I think you might be feeling confused... because he is sending out confusing vibes, and you really aren't sure how to read it. I don't know that either of you is trying to confuse the other, but the fact that you both have baggage ain't helping the sitch... you are both acting reflexively from years of conditioning, and neither one of you may actually KNOW what the 'confusion' is all about.
So. You are just going to have to spit it out.
"I'm confused."
"Why?"
Explain. I think you will be able to muddle through from there.