how I didn't "care" to enquire what his comment meant... I didn't really analyze it.... nor, did I really notice it when he first said it.
Good job.
Make one of personal goals as to not feel obligated to give him notice or an account of your private life. You should find it gives you a sense of power and detachment. Telling him your plans that are on your own time is old habits. Break free from it.
It is not about what you feel should work in your M. It is about doing the work that gets the right results. Do what works!