Quote: I have set some boudaries but don't think I am doing it right or he just don't listen to anything. I have told him I didn't want him masturbating....
Those sound more like demands, which are controlling and violate H's boundaries. While it might be aggravating given the circumstances, I really think that it is none of your business if he masturbates (and the only reason that you care is that you're not getting the sexual attention you want). A boundary would be deciding what you will tolerate about something that is your business, and deciding how you will take care of yourself. Boundaries are about dealing with other people's behavior by controlling YOUR behavior. (For example: My goal is for H to be willing to have sex with me. I will try XXX and if it doesn't work, then I will try something else or seek other ideas. If I have put in all the effort I can and there are no positive results, then I will XXX.)
We would never expect a date to even hang out with us, let alone kiss us, if we were acting angry, critical, controlling, and demanding. How can we expect a spouse to want to ML to us when we're like that? You can jump up and down and scream all you want. That might work for getting him to take out the garbage, but I can't imagine it will ever work for getting any kind of intimacy.
What does H need in order to feel comfortable ML, or even comfortable in the R? What are his LLs? How are you filling his love tank? Have you considered that he may be feeling sucked dry with nothing left to give? Maybe not, but it's safest to check your own house before you start throwing stones.
As Michele points out, what we think "should" work is rarely what actually does work. I wish you success in getting past anger and "shoulds" to finding real solutions.