I agree that the adulterer betrays themself, their morals, their ethics, their honor.

Betray: to be disloyal or faithless to, to deceive, to forsake.

But I would definitely say they also betrayed their spouse, betrayed the promises, the covenant, the person.

Not to mention, I would say it's up to the person who's been betrayed to say whether they were or not. And oh, I definitely was.

Betrayal's in the eye of the beholder.