She'd "Deep Stealth", having had pretty close to a normal life since her mid 20's, so I can give no more details about her. Other than she's quite a gal.
It's about the nature of gender - and those transsexual people who for some reason or other never quite seem to look right in their acquired gender. Nothing you can put your finger on - but some just look exactly right, while others ring false.
Like you Jane, I'm convinced that the answer lies in the brain.
Look at the diagram - values of -4 to -1 we call "male", values 0.5 to 4 we call "female".
All sexed species have facilities for recognition of the opposite sex buried deep in their nervous system. The same goes for their sexual identity. That's necessary for species survival - though by some theories, having a trace amount of cross-gendered identification may be advantageous. Or maybe it's just in the nature of things that stuff goes wrong sometimes, and to make it absolutely 100% accurate would have penalties not worth the cost.
Now we know from autopsies, and most recently, dynamic MRI imaging that the human brain is very sexually dimorphic - at least as dimorphic as the genitalia. But it's also a very complex organ. It might be best described as a vector with not dozens, but thousands of values. Put into plain english, a brain might be FFFfFfffFmFFFfFFmfF - indicating strongly feminine (F) in some aspects, normally feminine (F) in others, weakly feminine (f) in others, and even a bit more masculine than feminine (m) in a few.
Really though, "feminine" could be anywhere from 0.5 to 4 as shown in the diagram, and I'm simplifying hugely.
Now my own studies, and research by other workers in Artificial and Natural Intelligence, indicate that we think consciously a lot less than we think we do, and think subconsciously a lot more. Moreover, we make use of stereotypes a lot to fill in the gaps in information that we don't have.
For example "it has 4 wheels and is on a road" leads us to the belief that "it" is a car, an automobile, and we're usually right in that belief. We don't have to examine the chemical composition of the paintwork, and other such details.
The brain's "programmable firmware" give us our instincts, our talents, the way we move and react, the way we carry ourselves. A recent study showed that gender recognition was based more on movement than anything else. You, Jane, could be wearing a boiler suit, and just the way that you walk, or even your stance when standing still, would clue anyone that you're a woman. That's because the base part of your brain causes your posture to be that way. Typically feminine, maybe 3 on the graph above. There are other sexual cues too, body shape, face, and there's additional cultural programming on top of that. In our society, men generally don't wear floral prints, skirts, lipstick, and these are additional cues, but the basics are still instinctive, we "think" without knowing that we think.
Now when things go wrong - and for some of us, that means 47xxy chromosomes, for others 46xy but a glitch in masculinisation, etc etc we may end up not with FFFfFfffFmFFFfFFmfF but with FFfMFfmfFmFfFfFFmfF. Still recognisably female, but with some bits under-feminised.
Though again, it's a bimodal distibution, numbers <0 are "typically male", >0 "typically female", so it's not quite right to say bits of the brain are "masculinised" or "feminised". They are what they are, typical or atypical for a given gender.
And that is why I was so emotionally affected by getting involved with the Women in Technology. I knew from my studies that my brain hadn't been feminised "properly", it couldn't have been, no matter how much I might wish it were so. But when I joined WIT, I found other women, natal women, who thought just like I did. I hadn't been damaged beyond repair, my femininity wasn't congenitally wrecked, I was no more masculine than any of the others there, and more feminine than a few. Yet they were all women. I wasn't "second class", "ersatz", or shoddy goods after all.
Anyway, that's why, despite there being a "spectrum", we still fit into one of two main categories. It's why 99.9% of us are recognisable no matter what we do, provided enough of the misgendering cues (such as build) aren't all consistently too far from the norm.
But if you are a woman, regardless of the genitalia or chromosomes you were born with, and the parts of your brain controlling how you move and carry yourself are misgendered, then you have a problem.
A minority of TS people have this. They must "learn" techniques to overcome their intincts and move in a way that will cause correct gender recognition. But others have the opposite problem, their intincts are so strong that they must "learn" techniques to overcome their instincts just so they will pass as the gender they're not. Those TS women with very masculine bodies (like me) are pretty much forced to do this, and the bar for success is lower than for those with androgenous/feminine bodies. Those, the andro/femme primary transitioners, have basically no hope of pretending to be guys successfully. Like you, Jane.
It does mean though that some TS women have to work at it, while the majority of us just have to relax, and get rid of as many somatically male cues ( voice, face, skin texture, clothing ) as we can.
The impassable ones are just as female as we are, and more so in some cases. They just have parts of their brain masculinised that we don't.