Jen, I think maybe it could be an issue of trust. He may have felt that you withheld sex over the years to manipulate him and now you are "wanting" sex (or pretending to want it) in some kind of other attempt at manipulation. I think being available for him without being "too available" for him could be the key. Will he talk about it at all? Trust yourself and him that you will eventually find the right balance.
The most important thing a father can do for his children is to love their mother.