The baby steps build us up so much and the backwards slide kills all of that in a single instant
Yes, you're right that one backslide can wipe out a bunch of positives. I look at backslides as bringing one back to square one and resetting the clock to zero.
Getting built up by seeing baby steps suggests that the thing to do instead is to observe baby steps without attaching hopes or expectations to them, as well as learning how to control yourself so as not to get into backslides that discourage them.