APOE alleles are associated with sex-specific structural differences in brain regions affected in Alzheimer’s disease and related dementia. Here, APOE is linked to dementia.