It's great that you're going away together. Maybe you can read up on some of the previews of the NFL season so you will have something to add if/when you discuss football. What else interests him? Is there something that you could learn to love also?
A suggestion for family time: make one night a week game night or family activity night. Do any of your kids play sports (or they may be too young, sorry I forgot their ages)? Going to the games and practices together is another good way to get together as a family on a regular basis.