You're not crazy but you might sabotage yourself by noticing his emotions too much. It will make you reactive and scared if you are not careful. Or at least it does me. Acknowledging the baby steps without getting in panic mode if they disappear seems key. Hope it is humanly possible!
