Well, you had your one and three year old when your H wasn't living with you, right? And you said you were mysterious then and it caught his eye. Correct? So you can do it, even with a 1 & 3 year old since you did it once before.
Boundaries are the name of the game here. I know it is very scary to set them but you have to or else nothing will change for YOU!