I always believe that a person should do what they feel is right for them. If you feel that it is better for your M not to go; by all means don't go.
I will say that I think if you do go and have a great time without involving him; it might show that you are getting on with or without his involvement. Plus, you always have the out "my BIL invited, so I came."
Just my thought. But do what you feel is best for YOU.
