You say you’re married and not looking for anything serious. A few dates and you give him full access to the puinani and life for him is good. Now you start pursuing and he is trying to keep you at arms length so he lets know why he’s not willing to give you more. You fix that it will certainly be something else. He probably is also not happy because you are not a woman of your word.
