You can do whatever you want... that's the beautiful thing.
You see... anyone can get laid, have a one night stand, fire the big gun. And it's nice that you have so many friends who are willing to take one for the gipper. Is this a way of saying &%*&!!!%?
A respected poster's signature line is "What's best for my children is best for me." Your wife may have a date planned, been busy with a shmuck, living the life she wants. But guess what, Smile Guy.. you're the dad.. you're the emotional rock for your kids. You are home, you are there for your kids.
Consider exploring this with your therapist. After all, if having a dalliance is good idea today, it will still be next week.