Yes.. no partying with the enemy or the aliens, eh ? I dont know all the specifics.. but it is close to an ethical issue for the L to suggest.. or invite you ? Just tell your counsel about it and he/she can decide what to do ?
As for a conflict of interest issue.. If that L has represented you before on other personal matters which were and/or should not be disclosed to your spouse.. maybe ?
I am not licensed to practice law in your state.. so .. I cannot advise you.. just one guys opinion. Take care.. go hang out with some bingo folks or something tommorrow ? Something ?