Maybell, I took my rings off after H stayed out all night and showed up at 6:30am the day before we left for a family cruise. I haven't worn them since and I really don't think he or anyone else has noticed. He hasn't worn his since before BD, that's how it all came out anyway, I asked him why he wasn't wearing them and to put them back on. That didn't happen.
At this point, you should do what feels right to you and don't worry about what impact it may or may not have on H.