Heather, I'm glad that you and Louisa had a nice Easter. It's been a long time coming.
Quiet the fear you are feeling about your xh. I don't think he will disrupting your world as he's got enough on his plate right now. Put the fear aside in thinking that he will take Cali away from you. Cali is inviting him into her life all by herself and the drugs aren't helping w/that at all. Cali is the one that has reached out to him and the MIL from h@ll. She's 21 and is making choices that only she will have to face the consequences of her actions. You can't protect her any longer as she's grown woman. Yes, you can love her from afar, listen and sometimes validate what she tells you, but you already know that you can't trust her because she will toss you under the bus once again. You will need to utilize your db techniques when dealing w/her.
As for what has transpired in your life over the last couple of years, I wouldn't worry about what they think. You struggled and lived on a shoe string budget w/no help from Matt, his parent or Cali. From where I'm sitting, they have nothing to brag about in the way that they are living. At least you tried to keep a stable home life and certainly didn't drink or do drugs by any means. You are human and have been willing to own your mistakes and do better.
As for where you are now, you are happy and taking control over your life. You have a job that you like and you are sharing a home w/someone else which means less rent for you to pay. Don't sweat the small stuff.
Cali needs to fly solo, i.e., make mistakes and learn from them. Eventually all of them will see each other for who they are and that's when Cali may come to terms w/what her father did and continues to do...but she's going to have to hit bottom hard before that happens. Until then...love her from afar and be cautious in what you share w/her.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.