Only you can decide if you are doing the right thing. I will say this...if this is a guy who at 35 has decided to drop everything and pursue a career as a poker player, my opinion would be that he has some serious growing up to do. The lack of maturity is comfirmed then by his unwillingness to commit to anything. Wants to spend time with you but not be tied down. Sounds like something out of the mouth of a 16yo.
Do you have kids involved? If not, what kind of relationship do YOU want and do you honestly think he could offer it?
Stand up for what is right, even if you are standing alone.