I think your analysis of your part in perhaps making FF's More enticing (your panic, withdrawal etc) is RIGHT on the mark.
What an excellent insight, and the 180's it will breed are bound to change that entire dynamic.
I think back to the recent incident where your H shared his conversation with FS (female student). Your behaviour there told him loud and clear that he can share such things with YOU. FF's are far less likely to become "confidants" if YOU are the confidant instead!!!