Beersha - Your situation is very similar to mine. My boys are 5, 3, 15 months. I am so sorry to see that you are in this spot. I am a bit further along in this journey than you, but still trying to hang in there. If you read back in my threads, you will find a lot of great advice that I have received early on from everyone here helping me to establish boundaries.
My H is living at his parents, does not take the kids overnight, etc. He used to come into the house, lay in our bed, shower, eat whenever he wanted. Back in November, I set up boundaries with him. He was not allowed to use the house like his own. We sent days for him to have the kids, so that he was no longer just swinging by for dinner whenever he felt like it. The boundaries are to protect you and the kids. They are not an ultimatum. They have nothing to do with timelines, etc. Ultimatums and timelines will not work. It make me feel like I had some control. H did not like it but he respected the boundaries.
This is going to be a marathon. Just focus on yourself and your kids. Set your boundaries and forget about what your H is doing.
I have to run, but if you post your schedule with the kids/work and we can help you come up with a proposal to talk with your H about. It really does help. It also helps with the kids so that they know when to except seeing their dad.