In this cohort study of 1038 community-dwelling older adults, higher levels of neurodegeneration (total tau), axonal injuries (neurofilament light), and reactive astrocytes and neuroinflammation (glial fibrillary acidic proteins) were associated with accelerated cognitive decline in genetically susceptible APOE4 carriers without dementia compared with noncarriers. Here, APOE is linked to dementia.