Neuroinflammation is a central mechanism involved in neurodegeneration as observed in Alzheimer's disease (AD), the most prevalent form of neurodegenerative disease. Apolipoprotein E4 (APOE4), the strongest genetic risk factor for AD, directly influences disease onset and progression by interacting with the major pathological hallmarks of AD including amyloid-beta plaques, neurofibrillary tau tangles, as well as neuroinflammation. Microglia and astrocytes, the two major immune cells in the brain, exist in an immune-vigilant state providing immunological defense as well as housekeeping functions that promote neuronal well-being. It is becoming increasingly evident that under disease conditions, these immune cells become progressively dysfunctional in regulating metabolic and immunoregulatory pathways, thereby promoting chronic inflammation-induced neurodegeneration. Here, we review and discuss how APOE and specifically APOE4 directly influences amyloid-beta and tau pathology, and disrupts microglial as well as astroglial immunomodulating functions leading to chronic inflammation that contributes to neurodegeneration in AD.