You aren't doing anything wrong by asking him outright.. The financial thing was a courtesy.
In a safe, healthy marriage a spouse should feel safe to ask direct questions that might make their spouse uncomfortable... THAT is a GOOD MARRIAGE
Choices :
a. Don't ask, foster a climate of silent acceptance b. Ask directly and risk instigating a fight that you might avoid c. Ask diplomatically to keep the peace and your sanity at the same time