Step up and confront him. You aren't doing either of you any favors by not doing that. If he's pulling away when you do that and you back down...that's a behavior that's working for him.
I know what it feels like to not be ready to go head-to-head, but ask yourself this....how long are you willing to let this go? I guarantee, he's going to out wait you.
