I wonder why he thinks he will be a better father if he is out of the house. Is that because he thinks he will be a happier person? I think it is great also that he offered to take the baby at least. He seems to be the only little guy that your husband isn't spending much time with. That concerns me considering his comments about him being born. Maybe you could let your husband know that little guy is missing him. Just to help with a little bonding time for them.
Another thing that confuses me is he says he thinks he is not good enough for you even pre bomb. You would think if a person felt that way then they would be trying so hard and striving to be a great spouse. But he went and had an affair. Has he ever said why he strayed? Was it for emotional reasons or compliments.
Another thing you have stated is that you don't know if you ever loved him if I am correct. Do you think he senses that? Maybe it could attribute to him being out so much.
The one GREAT thing you have is a husband who will at least communicate with you. I am like him where I like to resolve things right away. Like me I have a husband who will not communicate. Then when he does communicate it is only attacking. So nothing ever gets resolved. So use it to your advantage that he does communicate. Maybe sit down with him and have a long talk about what you would like to happen in your marriage. Sometimes you have to start the pace of things in a marriage. Like if you would like him to buy you little gifts here and there. Maybe you could buy him a few and just put a note...was just thinking of you. Maybe you could arrange the first date this weekend just to talk about some things. He seems like he needs you to be really clear about what you would like.
There has to be something going on in that mind of his. He doesn't seem to have a problem either with having the kiddos overnight and staying home and doesn't gamble then. He is spending real quality time with them. Why can't he seem to do that with you. I would ask him that straight out. Maybe there is a resentment or hurt feelings that you are not aware of.