I have to agree that his behavior was unacceptable. He knew your feelings, and even if he didn't, the behavior is inappropriate regardless. I am trying to picture what would happen if I were that woman "friend". For one, I would not go over to a married man's house when the wife wasn't there... period. And being in a separate closed off room from the rest of the house, nope.
I could certainly understand if you were there, and you all hung out together.
Have you brought it up with him at all yet? Is he receptive when you try to talk to him about it?