A key question concerns the extent to which sexual differentiation of human behavior is influenced by sex hormones present during sensitive periods of development (organizational effects), as occurs in other mammalian species. The most important sensitive period has been considered to be prenatal, but there is increasing attention to puberty as another organizational period, with the possibility of decreasing sensitivity to sex hormones across the pubertal transition. In this paper, we review evidence that sex hormones present during the prenatal and pubertal periods produce permanent changes to behavior.TLDR version - it's complicated. Here, let me give an example:
There is good evidence that exposure to high levels of androgens during prenatal development results in masculinization of activity and occupational interests, sexual orientation, and some spatial abilities; prenatal androgens have a smaller effect on gender identity, and there is insufficient information about androgen effects on sex-linked behavior problems. There is little good evidence regarding long-lasting behavioral effects of pubertal hormones, but there is some suggestion that they influence gender identity and perhaps some sex-linked forms of psychopathology, and there are many opportunities to study this issue.
Sex matters for human behavior as it does for behavior in other species. Human males and females differ in many ways, including their appearance, their social identity, their social partners, the activities that interest them, how they present themselves to others, their aspirations and values, the likelihood of experiencing psychological and physical health problems, and the specific form in which those problems are manifested (reviewed in ). A key question concerns the causes of those differences, particularly the ways in which they are shaped by genes, physiology, and socialization.So far, so good. Relatively simple. But when you get further into it...
The focus of this paper – in line with the other papers in this special issue of Frontiers of Neuroendocrinology – concerns the extent to which human behavioral sex differences are influenced by sex hormones present during sensitive periods of development acting to organize the brain.
More than 50 years ago, Phoenix et al.  provided an experimental demonstration in female guinea pigs that early exposure to androgens masculinized sexual behavior. This revolutionary work opened a new era in understanding sexual differentiation of behavior and led to thousands of studies in many species showing
unequivocally that sex hormones present early in development affect sexual differentiation of behavior as well as reproductive anatomy and function . These hormones are said to have ‘‘organizational’’ effects because they produce permanent changes to brain structures and the behaviors they subserve. They are contrasted with ‘‘activational’’ effects, that is, hormones acting later in life to produce temporary alterations to the brain and behavior (through ongoing changes to neural circuitry) as the hormones circulate in the body throughout adolescence and adulthood. The main distinctions between organizational and activational effects concern timing and permanence, although these distinctions are not absolute .
Organizational effects have generally been considered to occur early in life when the brain is undergoing rapid change, but there has always been consideration of potential other sensitive periods of brain development when sex hormones again act to induce permanent changes .
Recent work in nonhuman animals has focused attention on other particular periods when there are substantial – and relatively abrupt – changes in levels of sex hormones, and that might serve as additional opportunities for hormones to sculpt brain structure: puberty and pregnancy (e.g., [91,149]). In this paper, we consider how human psychological sex differences are influenced by organizational hormones during prenatal development and again during puberty. We consider the foundations of the work, the methods used to study the question, the evidence that prenatal and pubertal hormones produce longterm behavioral changes, and directions for future research.
 J.E.O. Blakemore, S.A. Berenbaum, L.S. Liben, Gender Development, Psychology Press/Taylor & Francis, New York, 2009.
 C.H. Kinsley, L. Madonia, G.W. Gifford, K. Tureski, G.R. Griffin, C. Lowry, J. Williams, J. Collins, H. McLearie, K.G. Lambert, Motherhood improves learning and memory: neural activity in rats is enhanced by pregnancy and the demands of rearing offspring, Nature 402 (1999) 137–138.
 C.H. Phoenix, R.W. Goy, A.A. Gerall, W.C. Young, Organizing action of prenatally administered testosterone propionate on the tissues mediating mating behavior in the female guinea pig, Endocrinology 65 (1959) 369–382.
 C.L. Sisk, J.L. Zehr, Pubertal hormones organize the adolescent brain and behavior, Front. Neuroendocrinol. 26 (2005) 163–174.
 K. Wallen, The organizational hypothesis: reflections on the 50th anniversary of the publication of Phoenix, Goy, Gerall, and Young (1959), Horm. Behav. 55
Data confirming the importance of pubertal hormones in gender identity comes from outcome studies of children with gender identity disorder, showing that the majority develop gender typical identity in adolescence and adulthood, although many continue to display sex-atypical characteristics. An early study of extremely feminine boys showed that most developed a homosexual orientation without gender dysphoria . Two recent studies confirm that most children with gender dysphoria do not remain dysphoric after puberty. In one follow-up study of girls with gender identity disorder in childhood, only 12% were found to have gender dysphoria in adulthood; most developed a heterosexual orientation without gender dysphoria, although there were elevated rates of nonheterosexual orientation . In another study of boys and girls with gender dysphoria in childhood, 27% of boys and 64% of girls were still gender dysphoric, and most had nonheterosexual orientation . Both studies showed evidence of a ‘‘dosage’’ effect: children with more childhood cross-sex behavior or gender dysphoria were more likely to be gender dysphoric at follow up.Paper  is flawed because the definition of "Gender Dysphoria" was far too relaxed; few met the actual criteria, and many were Intersex. Paper  is currently enmeshed in the controversy about the aversion therapy techniques used at UCLA - torturing children, basically - and the revelation that much of George Reckers work there was grossly flawed or even fraudulent. Green seems to have taken a second look at the subject though, to try to report on the actual situation rather than one seen through ultra-conservative fundamentalist eyes.
(This may explain why girls were more likely than boys in the second study to persist in dysphoria; they probably had to be more extreme to receive a diagnosis in the first place.) Additional evidence for the importance of puberty in gender identity comes from data showing reduced plasticity with age: individuals diagnosed with gender identity disorder in adolescence are more likely than those diagnosed in childhood to have persistent dysphoria into adulthood .
Nevertheless, testosterone at puberty is not essential for male gender identity. There are cases of male gender identity in individuals with male-typical chromosomes and prenatal androgen exposure who were castrated in early life and reared as girls because of genital defects (such as cloacal exstrophy or penile ablation [44,111,133]). Some typical girls develop gender dysphoria or male identity at puberty without any obvious exposure to testosterone .
 M. Diamond, H.K. Sigmundson, Sex reassignment at birth: long-term review and clinical implications, Arch. Pediatr. Adolesc. Med. 151 (1997) 298–304
 K.D. Drummond, S.J. Bradley, M. Peterson-Badali, K.J. Zucker, A follow-up study of girls with gender identity disorder, Dev. Psychol. 44 (2008) 34–45
 R. Green, The ‘‘Sissy Boy Syndrome’’ and the Development of Homosexuality, Yale University Press, New Haven, CT, 1987.
 H.F.L. Meyer-Bahlburg, Gender identity outcome in female-raised 46,XY persons with penile agenesis, cloacal exstrophy of the bladder, or penile ablation, Arch. Sex. Behav. 34 (2005) 423–438.
 W.G. Reiner, J.P. Gearhart, Discordant sexual identity in some genetic males with cloacal exstrophy assigned to female sex at birth, New Engl. J. Med. 350 (2004) 333–341.
 M.S. Wallien, P.T. Cohen-Kettenis, Psychosexual outcome of gender-dysphoric children, J. Am. Acad. Child Adolesc. Psychiat. 47 (2008) 1413–1423.
 K.J. Zucker, Gender identity development and issues, Child Adolesc. Psychiat. Clin. N. Am. 13 (2004) 551–568.
 K.J. Zucker, S.J. Bradley, Gender Identity Disorder and Psychosexual Problems in Children and Adolescents, Guilford, New York, 1995.
Doing good Science is hard enough without such complications, especially in an area so complex.
Summary: Post-natal hormones do have an effect on gender identity. Sometimes. Not always. And pre-natal hormones have an effect on gender identity. Sometimes. Possibly always. And it's dosage-dependant, and associated with other neural circuitry (usually) and sexual orientation (sometimes). I'd quote more, but I'm already close to the limits of "fair use" of a copyrighted article.
Yes, I'd like it to be nice and simple, a binary, boy brain or girl brain, always set before birth, not mutable in the slightest afterwards... but that's not what the evidence says.
I suppose this should have been obvious to me. Picking the name "Zoe" at age 10 (establishing gender identity well before puberty). Feeling my brain re-wire, the increased sense of smell, the acquisition of a sexual orientation etc in 2005-2006, some effects long before HRT, and due to a female hormone balance. The male-typical play patterns in my childhood, just to make things even more obscure, I was anything but a "sissy boy" - yet my emotional responses were female, even as a child.