There is a hidden message in what he is telling you and he may not know for sure what it is, but there is something that he is missing that he doesn't feel that you care.
There is definitely something that he is trying to say and something that he is trying to ask you for. There is also the issue of you working through your unforgiveness for this OW. You have every justifible right to be upset with her, but she is also someone that has been blinded by satan as well.
She does not have to become your best friend or anything like that, but you do need to work through the forgiveness part of it as well. She did not steal your husband away from you, he willingly went. That is something that can not be forgotten, because she would not have been with him if he would not have chose to go be with her. So be very careful about blaming her for all that has happened.
I know that it is not an easy thing to do. It is hard to remember that they are being held captive as well and need to be released so that they can find happiness and not be used like this again.
Although we don't want to admit, they get hurt as much as we do when all of this ends. They go through certain emotions that we went through because they put their heart out there as well. Am I justifying what they have done? No, but I am saying that they are people with feelings and with souls just like we are.
In a certain way, they were used and then left once they are done with them. They are now paying the consequences for their actions, as well as the prodigal does.
As for your husband, there maybe nothing that you can say or do. I would highly recommend that you pray to the Lord to enter into his heart and give him the peace and the love that he is so desparately looking for. He is not going to get that from you. You can try with all your might, but there was also a void in your life that only the Lord could fill and he filled that void while you were on your journey. Maybe that is what you should be sharing with your husband so that he can get the same comfort and peace that you have within you.
Then your marriage will be being built on the Lord, which will make it even stronger then what it was before. Just my thoughts.