Aging is a dynamic complex process involving social vulnerability over time. The social vulnerability index (SVI) was developed that predicted adverse health outcomes. This study examined effects between SVI status and two genotypes, apolipoprotein E (ApoE) and Serotonin transporter genotyping (5-HTTLPR), on all-cause mortality. Data from the Social Environment and Biomarkers of Aging Study (SEBAS) were obtained, and SVI was constructed using 32 self-reported items of social determinants. Data from 985 participants (age: 65.73 ± 9.47 years, 54.62% males) were obtained for analysis, and the median SVI was 0.35 (IQR 0.29–0.42) with a near normal distribution. Participants with a higher SVI were more likely to be women and have poor cognitive function, more depressive symptoms and poor physical function. Adjusted for age and sex, each incremental deficit in SVI was associated with a 12% increase in mortality risk (HR: 1.12, 95% CI: 1.04–1.20, p = 0.002). An interaction was found between ApoE and SVI but not 5-HTTLPR. The strata-specific hazard ratio confirmed that associations between SVI and mortality was only in non-ε4 carriers (HR: 1.15, 95% CI: 1.07–1.24, p < 0.001), and SVI did not significantly predict mortality among ε4 carriers (HR: 0.84, 95% CI: 0.65–1.10). Differential SVI effects on mortality among middle-age and older adults were identified. In conclusion, a higher SVI was associated with all-cause mortality among middle-aged and older adults, and the association was moderated by ApoE genotypes but not 5-HTTLPR. Further study is needed to evaluate the clinical efficacy of healthy aging intervention programs considering gene-environment interactions and social vulnerability.