I know that it can seem in the moment that a boundary is not working. Very rarely are they instantaneouly respected, especially in the heat of an argument.
That doesnt mean you relinquish. You will see the positive response, the implicied (rarely verbal) acceptance after some time. Especially when you keep you cool while doing it. Your boundary will never be respected or acknowledged if you retract it or apologize for it.
The problem I see with there being no official diagnosis, the danger of self diagnosing is that it becomes self defeating. Its easier to chalk your behaviors up to some apparent patholgy however slight, then taking responsibility for self discipline and taking action. Your W grew up with no boundaries so its understandable.
Again Im not saying she doesnt have ABC,XYZ. If you can give yourself hope, then you empower yourself to take the actions you can, and trying, then being resigned to failure.
Once you know, then its time for acceptance of what is. Youve had years of nonboundaries and little respect. Give her and you some hope that things could be better.