Believe me the same thought has crossed my mind as well. I've even driven by his house a couple of times in moments of anger, but have talked my self down from doing something I'll regret later.
I too am guilty of this. I used to pack my MTN bike in my truck, and "ride past" to try and catch WW at OM's house. I don't know if they ever even hung out there together, because he's married too. But that didn't stop me from hitting the crazy train a couple times.