There have even been attempts to enshrine this principle in law.
A case report of an XX male with complete masculinization but absence of the SrY gene
Ghalia Abou Alchamat, Marwan Alhlabi, Muhyiddin Issa , Middle East Fertility Society Journal January 2010, Vol.15(1):51–53,
34-year old man with complete masculinization and a history of several years of infertility was referred to us for genetic reviewing. His semen analysis showed azoospermia. Conventional chromosomal analysis indicates a 46,XX karyotype, molecular analyses excluded the presence of SRY (the sex-determining region of the Y chromosome) gene. This case is one of the rare cases reported in the literature in whom testicular differentiation and complete virilization were found in a 46,XX chromosomal constitution, with the absence of SRY gene. This finding suggests that other genes downstream from SRY play an important role in sex determination. Through reporting this rare case and reviewing previous literatures, the aim of this report is to highlight the value of genetically screening all males with azoospermia who present for evaluation of infertility, since the phenotype does not always correlate with the genotype.Tests for sex that rely on the presence of a Y "male" chromosome don't work. Some men don't have them. Tests for sex that rely on the presence of the SrY "male" gene somewhere on one of the other chromosomes also don't work. Some men don't have those either.
So why do we call these "male" genes or "male" chromosomes? Why do we, including those of us who know better, sometimes say someone with 46,XY chromosomes is "genetically male"? Because of laziness, basically. Imprecision. All but 1 in 300 men are 46,XY. That's most of them. Not all, and there are plenty of women who are 46,XY too, and some of those even give birth to 46,XY daughters.
This paper comes to the conclusion that there are other genes that may cause masculinisation. We know that to be true, DAX9 for example. But we also know that hormonal environment in the womb, absent anything unusual in the genome, can also cause the phenotype, the thing being built, to be uncorrelated with the genotype, the plan.
Trying to define anyone's sex purely from the genome is a philosophical or ideological issue, requiring much handwaving and dismissal of the existence of exceptions, or even outright denial that exceptions can exist, for philosophical reasons.
At best, we can say that DNA/Chromosomes/Genes determine sex.. except for the many cases where they don't. A good guide, usually true, but not completely reliable, so cannot possibly be used to "define" what sex anyone is.