I'm not in a good place of my own, either, so take with a grain of salt - but I think you did the right thing in setting boundaries and enforcing them. Don't let him walk all over you because you will feel worse and it may impact your children.