Because your H might bully you is not a good reason to not do another 180. You have a right to get what you want from life as well as him and his rights don't come above yours.
Stick to your guns and use the L to protect you.
On a different but related tack, Where are you with regard to clearly defined goals and plans for achieving them?
- Nobody has ever learnt anything important from happiness and success; problems make us grow - Consult your plan, not your feelings - If you haven't set goals, how can you expect to achieve anything?