APOE ɛ4, but not polygenic Alzheimer’s disease risk, is related to longitudinal decrease in hippocampal brain activity in non-demented individuals. Here, APOE is linked to Alzheimer disease.