Why DON'T you start going to the gym for four hours?
I'm serious - ever wonder what would happen if yoiu started mirroring his behavior? If he goes out - you go out too and be gone longer. Come home after him. Let him start to wonder and worry a little bit about where you are.
Go to the grocery store and come home late - without groceries. lol!