Letting go doesn't mean you are letting it die. It can mean you are giving it air to breathe and to see if it can live.
However, I will tell you once again that you are an amazing, strong and impressive person. Stand up for yourself. Stop taking crap from him. Start by explaining the rules to him. Make him play the game the way you want, not the way he wants.
You make this too easy on him. Way too easy.
Maybe it's time for the 4x4
Be the person you know you can be. Make your daughter's proud of you.
