. . . While we have had a liberal relationship this broke our rules . . .
Um, you kinda glossed over what I'm sensing is a pretty important part of your backstory, Sad36. Were you guys swingers or something? Were you a cuckold?
Ordinarily, I almost always advice a strong, immediate, "I will not live in an open marriage" stance but obviously if you ALREADY had some sort of an open arrangement, that flies out the window.
I think we need some more background on this if we are to help you. In the meantime, YES move back into your home and YES move back into your marital bed.