$100 is a fair bit of cash... does he usually keep that amount in his wallet?
You know, AFTER the weekend is over you could start a ledger. He complained you weren't managing money well so just start keeping track of EVERYTHING in an accounting ledger... yours AND HIS expenditures...
You can ask him to account for every dime all in the guise of wanting to manage household monies better
He can't complain since he's the one that WAS complaining in the first place...
He may give you the "well, I just meant you keep track of your money, i know what I buy"...
You can always reply back
"but I know what I buy, and you don't seem to be concerned about keeping ME informed.. where's YOUR ledger?"