Without going into details, let me say that I understand your situation all too well.
I think that part of what you are feeling stems from the loss of trust that occurs after an affair. How much do you trust H at this point in time? Has he given you reason to trust him again?
The bond of trust is, along with honesty, at the top of the list of marriage obligations for most couples. Having that trust bond broken can be a major psychological barrier to overcome.
Doing things like checking the monthly phone and cellphone bills to see if H has been in contact with OW after you've reconciled, or calling him at his friends house to see if that's really where he said he was spending the afternoon are some of the things people do in an effort to confirm that they can once again trust their spouse. It's very normal behavior for the recovering spouse and can last many months or even years. In short and rightly so, your spouse has to earn his respect and trust back from you. Is he doing that? Have you ever told him how he can do that?
The dreams and flashbacks you are having are also normal, but since it has been four years, I have to ask you this question: What has, and to this day, what is your H doing to help you cope with your feelings about the affair?
Do you still have unanswered questions for him regarding details of the affair? Have your really forgiven him for what he did? Have you remembered your role in the initial breakdown of your marriage?
This might be something you should discuss with your H. Communication is the only way. If you need for him to give you affirmation that everything is OK and that he is in love with you, then tell him that.
Angeli, us guys are not what you call real 'sensitive' or 'relationship perceptive'. We often need to be told exactly what, why, and how to do something. Like, for instance, how to show you on a daily basis that he loves you and will never cheat on you again. Talk to him about your feelings. Don't nag him or bust his balls, but explain your feelings to him. If he is truly back on track, then he will understand where you are coming from and give you the support that you require from him to move on and leave the affair in the past. I had many 'mini' talks with my W after we got back together. These talks were beneficial for both of us as it cued us as to what expectations we had for one another and how we could meet those expectations. We still have talks like that every week to find out how the other is doing and what we can do to resolve any issues bewtween us. Communication!
RE your sexual desire: Does your H spend enough time talking to you and getting to know you again? People change, and if they don't keep up with their partners changes they will find themselves out of touch not only emotionally, but physically. Perhaps you are having trouble forgiving him? Does he ever bring up the affair or tell you how sorry he is that it happened? I don't expect that he would say it that way after all this time, but his actions should tell you where he stands. I don't think you are going backwards, but maybe your H is not going forwards. Think about it.