I think it can absolutely be casual. This man has 2 little ones, and so do I. His ex was a lying cheat (according to him) that he stayed with to try and keep family together until it wasn't doable anymore. So I doubt he is in any hurry to get serious and neither am I.
Once upon a time, a date was just a date. Not a lifetime commitment! It seems like in the US more and more a date is either a hookup/booty call or the first step to a serious relationship. I don't want either. I just want the fun of hanging out for a couple of hours with someone who makes you laugh and smile instead of fume or cry...