Keeping your distance from him is a great thing to do. Blaming you for his physical assults is just wrong and I am glad that you told him your boundries and are sticking to them.
Traveling 4 hours in seperate cars will be tough but well worth it if you do not trust him. You are giving him clear messages that you will no longer stand for his behavior and until he proves that he will not do that again I would keep contact to the minimum.
