The fact is a lot of people are ALWAYS going to think you're an abandoner and an a-hole for leaving your kids or refusing to work on your R. How can you or why should you protect him from people's opinions? Other men will think things like "that's what you get for getting married in the first place, sucker." You can't control other people's opinions from either side of the spectrum or what they say to your H about it. His guys at work saying all their exes are trying to drain them dry financially or whatever is just the flipside of the same coin. Detach from that mess either way.