Development of a machine learning-based multimode diagnosis system for lung cancer

Statistical analysis of epidemiological characteristics and clinical symptoms from 842 cases in the first-layer subsystem

The comparisons of the 14 features describing the epidemiological characteristics and clinical symptoms (between the 372 lung cancer and the 470 lung benign diseases) were shown in Table 1. Statistical analysis showed that there were significant differences between the two groups (P<0.05) for the characteristics of age by groups, age, gender, smoking status, drinking status, history of lung infection, expectoration, bloody sputum, fever or sweating, cough and hemoptysis. And, there were no significant differences between lung cancer and lung benign groups (P>0.05) for chest tightness or chest pain, family history of tumor and lung cancer.

Demographic characteristics and serum levels of ProGRP, VEGF, CEA, CYFRA21-1, and NSE for the study subjects in the second-layer subsystem

Demographic characteristics of lung cancer and lung benign disease patients in the second-layer subsystem were presented in Table 2. There were significant differences between the two groups (P<0.05) for the characteristics of age by groups, smoking status, history of lung infection, expectoration, bloody sputum, fever or sweating, hemoptysis and family history of lung cancer. In contrast, there were no significant differences between lung cancer and lung benign patients (P>0.05) for age, gender, drinking status, chest tightness or chest pain, cough and family history of tumor. As shown in Table 3, the levels of ProGRP, VEGF, CEA, and CYFRA21-1 in the lung cancer group were higher than those in the lung benign disease group (P<0.05). However, there was no statistical difference in the level of NSE between the two groups (P>0.05).

Statistical analysis of the 22 radiomic features extracted from lung CT nodules in the third-layer subsystem

The demographic characteristics of the subjects in the third-layer subsystem were shown in Supplementary Table 1. 22 lung CT nodule-based radiomic features were extracted from 123 lung CT nodules, which contained 64 lung benign nodules and 59 lung cancer nodules. However, the extracted lobulation grade f13 and spiculation grade f14 were 0 in both groups, which couldn't be further statistically analyzed. As shown in Table 4, statistical analysis indicated that there were significant differences between the two groups (P<0.05) for the radiomic features of gray mean f1, gray variance f2, gray histogram entropy f3, seven order invariant distance f4, calcification area f11, calcification area/nodule area f12, cavity number f15, contrast f18, correlation f19, energy f20, homogeneity f21 and entropy f22. However, there were no significant differences between lung CT benign and malignant nodules (P>0.05) for the seven order invariant distance f5, f6, f7, f8, f9, f10, cavity area f16 and cavity area/nodules area f17.

Development of machine learning models

As shown in Table 5, machine learning models were constructed to distinguish lung cancer from lung benign diseases. 14 epidemiological characteristics and clinical symptoms of 638 samples, including 296 lung cancer and 342 lung benign diseases, were used as input features to develop the models of C5.0-1, ANN-1, and SVM-1 in the training set. The accuracies of C5.0-1, ANN-1, and SVM-1 models in the training set were 79.78%, 73.04%, and 77.27%, respectively. 204 samples, including 76 cases with lung cancer and 128 lung benign diseases, were used as the testing set to verify the effect of the three models. The accuracies of the C5.0-1, ANN-1, and SVM-1 models in the testing set were 69.12%, 71.57%, and 65.20%, respectively. The 14 features mentioned above and the 5 serum tumor markers levels including ProGRP, VEGF, CEA, CYFRA21-1 and NSE from 208 patients were employed as the input variables to develop the C5.0-2, ANN-2 and SVM-2 models in the training set, which included 97 lung cancer and 111 lung benign disease patients. The accuracies of C5.0-2, ANN-2, and SVM-2 models in the training set were 97.60%, 85.58%, and 98.08%, respectively. 78 samples, including 32 lung cancer and 46 lung benign diseases, were employed to test the effect of C5.0-2, ANN-2, and SVM-2 models. The accuracies of models in the testing set were 80.77%, 89.74%, and 83.33%, respectively. 22 radiomic features were extracted from 90 lung CT nodules and adopted to train the C5.0-3, ANN-3, and SVM-3 models, which included 42 lung cancer nodules and 48 lung benign nodules. The accuracies of C5.0-3, ANN-3, and SVM-3 models in the training set were 100%, 93.33%, and 100%, respectively. 33 samples, including 17 lung cancer nodules and 16 lung benign nodules, were used to test the effect of the models. The accuracies of C5.0-3, ANN-3, and SVM-3 models in the testing set were 90.91%, 90.91%, and 84.85%, respectively.

Effect evaluation of machine learning models

As presented in Table 6, the testing effect of the model was evaluated by sensitivity, specificity, accuracy, PPV, NPV, and AUC. The sensitivities of C5.0-1, ANN-1, and SVM-1 models were 61.84%, 81.58%, and 59.21%, respectively. The specificities were 73.44%, 65.63%, and 68.75%, respectively. The AUCs were 0.676 (95% confidence interval [CI] 0.608 to 0.740), 0.736 (95%CI 0.670 to 0.795) and 0.640 (95%CI 0.570 to 0.706), respectively. The sensitivities of C5.0-2, ANN-2, and SVM-2 models were 78.13%, 84.38%, and 78.13%, respectively. The specificities were 82.61%, 93.48%, and 86.96%, respectively. The AUCs were 0.804 (95%CI 0.698 to 0.885), 0.889 (95%CI 0.798 to 0.949) and 0.825 (95%CI 0.732 to 0.902), respectively. The sensitivities of C5.0-3, ANN-3, and SVM-3 models were 94.12%, 88.24%, and 82.35%, respectively. The specificities were 87.50%, 93.75%, and 87.50%, respectively. The AUCs were 0.908 (95%CI 0.755 to 0.980), 0.910 (95%CI 0.758 to 0.981) and 0.849 (95%CI 0.682 to 0.949), respectively. To optimize the diagnostic model, the efficiency of different models was compared using the AUC in the testing set (Supplementary Table 2). Results showed that the efficiency of the ANN-1 model was higher than C5.0-1 (Z=1.981, P=0.048) and SVM-1 (Z=3.283, P=0.001). ANN-2 model was better than C5.0-2 (Z=2.021, P=0.043), and there was no difference between ANN-2 and SVM-2 by AUC comparison (P>0.05). But, the sensitivity of ANN-2 (84.38%) was higher than SVM-2 (78.13%). Although there were no statistical differences by AUC comparison among ANN-3, SVM-3, and C5.0-3 (P>0.05), C5.0-3 had the highest sensitivity of 94.12% in the three models.

Collection of clinical samples

Epidemiological characteristics and clinical symptoms of 372 lung cancer and 470 lung benign patients were collected from the First Affiliated Hospital of Zhengzhou University. All the subjects were surveyed through a questionnaire made up of 14 epidemiological characteristics and clinical symptoms, which included age, age grouping, gender, smoking history, drinking history, history of lung infection, family history of tumor and lung cancer, chest tightness or chest pain, expectoration, bloody sputum, cough, hemoptysis, fever or sweating. Smokers were defined as people who smoked one or more cigarettes per day for more than six months. The alcohol-drinkers were defined as drinking alcohol at least 12 times a year. A total of 129 patients with lung cancer and 157 patients with benign alterations of the 842 subjects donated the serum samples, among which the pulmonary CT images of 59 patients with lung cancer and 64 patients with benign alterations were simultaneously collected from the Radiology Department of the First Affiliated Hospital of Zhengzhou University. All patients with lung cancer were included according to the following inclusion criteria: (1) Patients were confirmed by the clinical diagnosis of pathology; (2) without undergoing surgical resection, chemotherapy, or radiotherapy; (3) without previous other organ tumors. Patients were excluded with significant organ function failure, pregnant, or lactating. Patients with histologically confirmed lung cancer included lung squamous cell carcinoma, lung adenocarcinoma, small cell carcinoma, and so on. Lung benign diseases included pneumonia, chronic obstructive pulmonary disease, pulmonary fibrosis, tuberculosis, and so on. The study protocol was approved by the Ethics Committee at the University of Zhengzhou. Permission for data and sample collection was obtained from the patients or their relatives.

Measurement of 5 serum tumor markers

3 mL venous blood was collected from every fasting subject in the morning, and then the blood samples were stored at 37°C for 30 minutes, centrifuged for 10 minutes at 1500g. Finally, the serum was separated and stored at -80°C for follow-up analyses. Serum ProGRP and VEGF were determined by ELISA kits (Shanghai enzyme-linked biological technology company) according to the manufacturer’s instructions. Chemiluminescence detection kits (Beijing huaketai biotechnology company) were employed to detect serum CEA, CYFRA21-1, and NSE according to experimental procedures.

Extraction of 22 lung CT nodule-based radiomic features

Radiomic features of lung CT nodules were extracted by MATLAB tool [27]. Firstly, the lung nodules on CT images were marked by three experienced radiologists. Then, threshold segmentation of pulmonary CT nodules was applied for the extraction of region of interest (ROI). Finally, 22 radiomic features of lung CT nodules were extracted. Among them, gray features included gray mean (f1), gray variance (f2), and gray histogram entropy (f3). Morphological features were consisted of seven order invariant distance (f4, f5, f6, f7, f8, f9, f10), calcification area (f11), calcification area/nodule area (f12), lobulation grade (f13), spiculation grade (f14), cavity number (f15), cavity area (f16) and cavity area/nodules area (f17). Texture features were composed of contrast (f18), correlation (f19), energy (f20), homogeneity (f21) and entropy (f22).

Flow chart of proposed work

A machine learning-based three-layer diagnostic system for lung cancer was proposed in this study as shown in Figure 1. The first layer was a broad-spectrum screening subsystem, which screened out the high-risk group of lung cancer from pulmonary disease patients. And, the machine learning-based screening models were developed using the 14 features of epidemiological characteristics and clinical symptoms. The high-risk individuals screened by the first-layer subsystem were included in the second-layer subsystem. The second layer was a machine learning-based auxiliary diagnosis subsystem constructed with the 14 features of epidemiological characteristics and clinical symptoms, and the 5 serum tumor markers for identifying suspected lung cancer patients from the high-risk groups. The suspected patients of lung cancer evaluated by the second-layer subsystem were further introduced into the third-layer subsystem. The third layer was an intelligent diagnosis subsystem, which was developed based on the 22 lung CT nodule-based radiomic features using machine learning models for further confirming lung cancer patients.

Figure 1. A three-layer diagnosis system for lung cancer.

Establishment of machine learning models

Based on the random sampling function of machine learning models, the samples were randomly divided into training set and testing set according to the ratio of 3:1 using partition node. The training set was employed to develop the models and testing set was used for evaluating the performance of the models. In each of the three subsystems, the 14 epidemiological characteristics and clinical symptoms were applied as the input variables for C5.0-1, ANN-1, and SVM-1 in the first-layer subsystem; The 14 epidemiological characteristics and clinical symptoms were combined with 5 serum tumor markers as the input variables for C5.0-2, ANN-2, and SVM-2 in the second-layer subsystem; The 22 lung CT nodule-based radiomic features were presented as the input variables for C5.0-3, ANN-3, and SVM-3 in the third-layer subsystem; While the groups (0 for lung benign diseases, 1 for lung cancer) were set as the output variables. Parameters for the models were set as follows:

Configuration parameters of the C5.0 model

Use partitioned data: yes; Output type: Decision tree; Use boosting: yes; Number of trials: 9/25; Mode: Expert; Pruning severity: 75/25; Minimum records per child branch: 2; Use global pruning: yes; Use misclassification costs: yes; Model Evaluation: Calculate variable importance.

Configuration parameters of the ANN model

Use partitioned data: yes; Method: Prune; Sample %:75.0; Accuracy:90.0%; Optimize: Memory; Use binary set encoding: yes; Show feedback graph: yes; Model selection: use best network; Mode: Expert; Hidden layers: Two or three (Layer 1: The number of variables. Layer 2: The number of features/2. Layer 3: 2). Model Evaluation: Calculate variable importance.

The input data of ANN were required to range from 0 to 1, so the parameters that did not meet this requirement were normalized using linear function to range from 0 to 1. Below was the formula:

$Y=\left(X-Xmin\right)/\left(Xmax-Xmin\right)$

(X was the original value, Y was transformed by the above formula via X, Xmax and Xmin were the maximum and minimum among all original data, respectively).

Configuration parameters of the SVM model

Use partitioned data: yes; Mode: Sample/Expert; Stopping criteria: 1.0E-3; Regularization parameter (C): 9/1; Regression precision (epsilon): 0.1; Kernel type: Sigmoid/Polynomial; Bias: 0; gamma: 0.5; Model Evaluation: Calculate variable importance.

Statistical analysis

Statistical analyses were performed by SPSS 21.0 software. SPSS Clementine 21.0 software was used for classification analysis. The data were expressed by Median (P25-P75) and analyzed with the Mann-Whitney U. Chi-Square test was employed for each contingency table. P-value of 0.05 was considered as a statistical test level.

Six indexes including accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic curve (AUC) were used to evaluate the classification models.