Development, validation and clinical impact of a prediction model for 6-month mortality in older cancer patients: the GRADE

Patients and their baseline characteristics

Among the 959 consecutive older cancer in- and out-patients from the PF-EC cohort study, aged ≥ 65 years who were referred for GA up to September 30 2017, we excluded 356 inpatients. A total of 603 outpatients were included, 439 patients for the development cohort, and 164 for the validation cohort (Figure 1).

Figure 1. Consort diagram for the patient selection.

In the whole cohort, the mean age was 81.2 ± 6.1 years. The mean time lapse between the initial oncology consultation and the geriatric oncology consultation was 6.6 days ± 2.0. Most patients were West European (62.7%), women (53.5%), and had locally advanced (38%) or metastatic cancers (45%). As expected, colorectal, breast and lung cancers were the most common cancer types, whereas prostate cancer was uncommon because of the absence of urological departments in our two centers (3%). The Frailty phenotype concerned 58% of the overall cohort [14]. The geriatric domains impaired concerned 13.5% (BMI < 21 kg/m²) to 67.5% (grade 3 and/or 4 comorbidities) of the sample according to the measures and thresholds used (Table 1, Supplementary Table 1).

There was no difference between the development and validation subsets for cancer-related and demographic data. Significant differences were found for comorbidities, weight loss, and cognitive impairment. There was a larger proportion of patients presenting weight loss (≥ 5%), cognitive impairment and overall co-morbidities in the development cohort, while there were more severe comorbidities in the validation cohort (Table 1).

Development of a multivariate prediction model for 6-month mortality

At 6 months, the mortality rate was 17.5% (n=77/439). In univariate analysis, male gender, cancer site, cancer extension, exclusively supportive care, malnutrition (BMI < 21 kg/m² and weight loss ≥ 5%), and impaired mobility (gait speed < 0.8 m/s) were significantly associated with 6-month mortality. Age, comorbidities, dependency, depressed mood and cognitive impairment were not associated with 6-month mortality (Table 2). The final multivariate prediction model included two geriatric variables, malnutrition (unintentional weight loss) and impaired mobility (slow gait speed); two oncological variables (cancer site, cancer extension), and the decision whether or not to instate exclusively supportive care (Figure 2, Table 3). There was no interaction between cancer site, cancer extension, and exclusively supportive care. The final scores ranged from 2 to 19 with a median of 9 (6-11). Four groups at increasing risk were identified: 118 patients (25%) were at low risk (2 to 6), 196 (45%) at medium risk (7 to 10), 102 (23%) at high risk (11 to 14), and 23 (5%) at very-high risk (15 and over). Overall, the risk of early death ranged from 2% to 61%. The score was well calibrated (Figure 3A), and discrimination was good, with a Harrell’s C index of 0.75 (0.69-0.81). The Kaplan-Meier plot showed significant discrimination across the four risk groups. In particular, the 6-month risk of death was 2%, 14%, 32% and 61% for the low, medium, high and very high-risk groups respectively (Figure 3C).

Figure 2. Forest plot of the multivariate prediction model.

Figure 3. (A, B) Calibration curves in the development cohort (A), and in the validation cohort (B). The grey line indicates the ideal prediction; The black line indicates prediction with the GRADE; The dashed line indicates prediction with optimism correction. (C, D) Kaplan Meyer survival curves for 6-month mortality according to risk-groups in the development cohort (C), and in the validation cohort (D).

Internal validity and clinical impact of the final multivariate prediction model

In the validation cohort, the 6-month mortality rate was 18.9% (n=31/164). The score ranged from 0 to 18 with a median of 9 (6-11). 43 patients (26%) were at low risk, 68 (41%) at medium risk, 41 (25%) at high risk, and 12 (7%) at very high-risk. Overall, the risk of early death ranged from 0% to 58%. The score was also well calibrated (Figure 3B) and discrimination was good, with a Harrell’s C index of 0.76 (0.66-0.86), close to that for the development cohort. The Kaplan-Meier plot showed significant discrimination across the four risk groups (Figure 3D).

In stratified analyses, discrimination showed consistent results, with the best Harrell’s C index at 0.81 (0.63-0.98) in the 77- to 81-year-old subset (Table 4).

The decision curves provided better performances for our scoring system than for the ECOG-PS in accurately identifying the risk thresholds for purpose of therapeutic-decisions (Figure 4). In particular, at the 2% and 14% risk thresholds for early death our scoring system points the need for cancer treatment and is related to a favorable C/B ratio among older patients. At the 32% risk threshold for early death our scoring system yields a less favorable C/B ratio for cancer treatment among older patients and suggests the need for caution in therapeutic decisions (tailored case by). At 61% risk threshold for early death our scoring system discourages cancer treatment among older patients, since the C/B ratio is not in favor of these patients. One the basis of these considerations, 23/38 (60%) patients in the exclusively supportive care subset would be less likely to be undertreated. Conversely, 26/126 (21%) patients in the active treatment subset would be less likely to be overtreated. Overall, according to the GRADE, 49/164 (30%) patients would be misclassified for the final therapeutic decision.

Figure 4. The decision curves show estimates of the SNB (%) over a range of probability thresholds used to categorize observations as “high risk”. The curves help to evaluate a treatment policy that recommends treatment for patients who are considered to be “high risk” by comparing the population impact of a risk-based policy to “treat all” (cross line in grey) and “treat none” (baseline) intervention policies. A model for prediction of early death (curves in black) according to ECOG-PS, and the GRADE. At a given risk threshold for early death, the graph gives the expected SNB per patient for “treat none”, “treat all”, and to treatment according to the ECOG-PS, and the GRADE, in relation to the related Cost/Benefit (C/B) ratio.

Practical clinical application

For therapeutic decision purposes, the GRADE should be used in a two-step approach: the sum of the first four variables (i.e. weight, gait speed, cancer site and extension) gives a risk of death at baseline; then the addition of the last variable enables the final decision on whether or not it is exclusively supportive care that is required. This dynamic process is expected to help with weighing up therapeutic decisions, to avoid under- and over-treatment situations. We also created a nomogram using the five clinical variables (Figure 5) with a free website link to be used by clinicians in their daily practice: https://grade.shinyapps.io/dynnomapp/. In brief, the website gives the patient’s expected survival by way of click-check variables.

Figure 5. A nomogram is a graphical calculating device based on the results of a multivariate Cox regression. It is a quick way to interpret the 6-month mortality risk predicted by the multivariate Cox model. A numerical scale was created allocating scores for each predictor: Unintentional weight loss (“Yes”: 37; “No”: 28); Slow gait speed (“Yes”: 60; “No”: 28); Site (Hematological: “0”; Breast: “28”; Skin: “46”; Liver: “51”; CR: “55”; GU: “62”; Dig: “70”; Lung: “71”; Prostate: “74”; Other: “100”); Extension (Local: “28”; Locally-Advanced: “60”; Metastatic: “81”); and Exclusively supportive cares (“Yes”: 55; “No”: 28). The total score derived from all the covariates ranges from 112 to 333 points and indicates the probabilities (Pr) of dying in the 6-month follow-up. This device is available on the website: https://grade.shinyapps.io/dynnomapp/ Skin: skin with melanoma; CR: colorectal; GU: genito-urinary; Dig: digestive non-colorectal.

Reporting

We followed the Transparent Reporting of a multivariate prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines [18].

Study design and population

Patients were recruited from the Physical Frailty in Elderly Cancer patients (PF-EC) cohort. This prospective, observational two-center cohort study started in November 2013, and is described in detail elsewhere [9]. Briefly, all consecutive older in- and out-patients referred for a geriatric oncology assessment were prospectively included in a registry when a diagnosis of cancer was established and when a frailty was suspected, before any cancer treatment decision.

For the present study, we analyzed all outpatients who presented up to September 30 2017, regardless of tumor site or stage. We excluded inpatients because of the impracticality of conducting mobility tests (e.g. infusions limiting walking measures). The inclusion date was the date of the first geriatric oncology visit.

Informed consent was obtained from the patients before inclusion. The study was approved by the local ethics committee (CLEA-2015-019, Avicenne Hospital, Bobigny, France).

Cancer-related and demographic data

Demographic data (age, sex, region of origin classified as follows: West Europeans, East Europeans, Latin Europeans, North Africans, Sub-Saharan Africans, and Asians), tumor characteristics (site, extension: local, locally advanced i.e. unresectable tumor with no distant metastases, or metastatic disease stricto sensu) and Eastern Cooperative Oncology Group Performance Status (ECOG-PS) were obtained at the first geriatric oncology visit as part of the GA. Cancer treatment modes were categorized as exclusively supportive care or not, and were prospectively collected in the 6-month follow-up according to the treatment finally administered (i.e. supportive care only vs. at least one of the following treatment modes: surgery, chemotherapy, targeted therapy, hormonotherapy, radiotherapy, percutaneous treatment, and intra-arterial treatment). All treatment regimens, including exclusively supportive care, were validated in weekly institutional multidisciplinary meetings.

The geriatric assessment (GA)

The GA was performed at the first geriatric oncology visit and included seven domains (social environment, comorbidities, functional status, mobility, nutrition, mood, and cognition) [8, 14, 26–31] (Table 6).

Outcome

Overall 6-month mortality following the GA was recorded to assess predictors of early death. Vital status was determined from medical records or by telephoning patients or their families.

Statistical analyses

We allocated patients included up to July 30 2016 to the development cohort, and the remaining patients up to September 30 2017 to the validation cohort.

Categorical data are expressed as numbers and proportions, and continuous data as means and standard deviation (SD) or medians and quartiles (25^th-75^th).

Comparisons of baseline characteristics (cancer, demographic data, and GA-components) between the development and validation subsets were performed using the chi-square test or Fisher’s exact test for categorical variables, as appropriate.

We performed a correlation assessment using the Spearman’s rho test as appropriate for categorical variables. Multicollinearity between variables was defined as a rho test value ≥ 0.50.

Development of the onco-Geriatric scoRe of eArly DEath (GRADE)

A comparison of baseline characteristics (cancer, demographic data, GA-components) between six-month survivors and non-survivors in the development subset was performed using the log rank test. Univariate predictors of six-month non-survival were expressed using non-adjusted Hazard Ratios (HR) and 95% confidence intervals (95%CI). Variables yielding P values under 0.20 in the univariate analysis were considered for inclusion in the multivariate analysis. Because of multicollinearity between the GS and the IADL, the ADL and the ECOG-PS, we choose GS as a covariate of functional status. We therefore created a Cox multivariate proportional-hazard regression model to assess multivariate factors associated with 6-month mortality. Multivariate predictors were expressed by the beta-coefficient with standard error, and by adjusted HR (aHR) with 95%CI. Graphically, multivariate predictors were presented in a forest-plot. Model assumptions were verified. To maximize likelihood while minimizing losses in prediction, a final multivariate prediction model was created with a backward selection procedure according to the lowest Akaïke Information Criterion (AIC) [32]. A scoring of each predictor was performed using Schneeweiss’s beta-coefficient point-based scoring system [32]. This scoring system weights by 1 unit more or less with each 0.3 increase or decrease in the beta-coefficient. We categorized this score into four groups of increasing risk on the basis of the lowest AIC value in Cox regression. Graphically, we assessed the calibration slope in the final multivariate prediction model. Discrimination by the final multivariate prediction model was assessed using Harrell’s C index with 95%CI. Given the current lack of consensus on the best threshold to consider for the quality of discrimination for survival prediction models, we used a previously described categorization of Harrell’s C index discrimination as well: 0.5-0.59 (poor), 0.6-0.69 (moderate), 0.7-0.79 (good), 0.8-0.89 (very good), and ≥ 0.90 (excellent) [33]. Survival curves were plotted according to the Kaplan-Meier method with the final score divided into four groups.

Internal validation

we assessed the calibration and discrimination of the final score in the validation subset with the same methods as described above. Stratified discrimination analyses were performed in age quartiles, for an ECOG-PS ≤ 2 vs > 2, and for non-metastatic vs metastatic cancers. To assess the clinical impact of our score, we used the standardized net benefit (SNB) derived from decision curves [34, 35] (Supplementary Methods).

All tests were two-sided, and the threshold for statistical significance was set at P<0.05. The data was analyzed using R statistical software (version 3.4.3, R Foundation for Statistical Computing, Vienna, Austria; http://www.r-project.org). A multivariate imputation by chained equations was used to handle missing data for weight loss (n=9), and GS (n=3), via the MICE package in R.