Even a barely adequate parent is better for a child's development than no parental contact at all - unless that parent is abusing a child. I have a little bit of experience with child abuse issues, so here is a little advice that I hope helps...
If you have not already done so, look into getting a court appointed advocate for your child. In my area we have an organization called CASA. Accusations of child abuse can get thrown arround a lot in divorce court, os a child advocate's testimony will have more sway over a judge's decision than you will.
Don't be afraid to call child protective services for advice and recomendations. They really do try to keep families together and can help you make a plan to ensure your son's safety. If you can not get ahold of someone there, try calling a foster care shelter. Child protective services are often overwhelmed. You can't afford to get lost in the shuffle.
Document, document, document! Photos, anything in writing, written and signed eye witness accounts, all of this can help you make your case. Remember to stick to concrete facts though.
Remember that in cases where a child is in danger from a parent, the child custody arangement can change, even after a divorce is final.