APOE4 genotype, parental family history of AD, age, and sex are all associated with microstructural white matter differences in late middle-aged adults. Here, APOE is linked to Alzheimer disease.