I think you should wear it as long as you are committed to the M. I wouldn't phrase it to S3 that way because it is somewhat disparaging to W. I would say something more like "It's daddy's and it's very special so I don't ever take it off". Same message but without the dig.