Matilda, I assume you suspect a possible EA (numerous phone calls on the cell phone) with the cute lady with the clean house and boat. I could see why you would be uncomfortable meeting her for dinner, as part of a group.
It sounds like you have two options. You can accept your H's invitation next time to see if it would improve your R with him (the primary goal). You can respect that being in a group setting with the the lady he's having a possible EA, is too uncomfortable at this time. You will need to continue to assert your wishes, and accept that he won't understand, and be nonsupportive of your decision.
CL
CL 53 W 54 M 20 yrs. 03'-09' Separation + Old Patterns + GAL 10-14' Piecing
"The Master allows things to happen. She shapes events as they come."