As you know full well, your faith will get you through. Some people will understand and others will not. That will happen with pretty much anything you chose to make public. Right or wrong - people do judge.
Many people choose the path you have chosen and live a full and wonderful life with friends and activities. Are you socializing to satisfy a basic need for companionship. It doesn't have to be with women and maybe shouldn't be as you are staying true to your marriage but you should indeed be out living your life.
Of course you worry about your children. They are of you. You must continue to gently guide by example and to pray for them. Cling to Proverbs 22:6 Train up a child in the way he should go: and when he is old, he will not depart from it.
As far as who is counting - I think what is most important to consider is that you are. I am sure it is difficult not to do so. Should there be a counting up of the hours, the days, the years? Does counting help you to remain steadfast or does it cause your shoulders to droop and your soul to cry out? Just some things to consider as you continue your stand for what you believe in. {and please don't feel you need to answer them here out loud.}
Hang in there, man. The Lord is on your side. ~ swl
Look not mournfully into the past. It comes not back again. Wisely improve the present. It is thine. Go forth to meet the shadowy future, without fear. {Henry Wadsworth Longfellow}