I think moving is a VERY good idea. There has to be a way to get out of that town. Can you imagine raising your new baby in this environment of gossip might be like?!
And now you know another thing - talking to these other women will somehow come back to bite you in the a$$ - just as any other gossip eventually will.
25 is right - keep strong, ignore your ex and his nightmare of a life - and be eternally grateful that you got away from it!