Umm...when is your down time? As coworkers you had better set a schedule. This is such a big part of your life I think you may need a boundary or two. If you were working elsewhere ask yourself if you would be good with this. If so, great, but if not then you have to figure it out.

By your remarks, it seems you are not.