APOE and Cognitive impairment: However, those with both the APOE e4 allele and loneliness were five times more likely to experience cognitive impairment.