A COVID-19 risk score combining chest CT radiomics and clinical characteristics to differentiate COVID-19 pneumonia from other viral pneumonias

Patient characteristics

The clinical characteristics of patient data used are shown in Table 1. The COVID-19 patients had significantly higher lesion numbers, CK-MB activity, LDH activity, and bilateral, peripheral, or mixed central and peripheral pulmonary distribution than the non-COVID-19 viral-induced pneumonia patients. Most of the symptoms, laboratory results, and CT manifestations had no significant differences between COVID-19 and non-COVID-19 patients (Table 1). Representative images of COVID-19 pneumonia, adenovirus pneumonia, cytomegalovirus pneumonia, and influenza virus pneumonia are shown in Figure 1.

Human diagnosis of COVID-19

Supplementary Figure 5 shows the geographic distribution of 130 radiologists from 10 provinces in China, including Hubei. The performance of the 130 radiologists based on the chest CT images only (without providing patients’ clinical information and laboratory results) was moderate due to the overlap of CT manifestations between COVID-19 lesions and non-COVID-19 viral pneumonia lesions using a supervised human learning format (Table 2). Notably, the radiologists from Hubei Province, China, the epicenter of the COVID-19 outbreak in China, had a better performance than the radiologists from outside of Hubei Province (P < 0.05).

Patient-based risk scores

The patient-based risk scores using radiomic features only, clinical factors only, and a combination of radiomic features and clinical factors are shown in Figure 2A–2C and Equations (2)–(4), respectively. The utility of the risk scores achieved area under the receiver operating characteristic curve (AUC) values of 0.791 (95% confidence interval [CI]: 0651–0.932), 0.813 (95% CI: 0.682–0.944), and 0.915 (95% CI: 0.841–0.991), respectively, in the validation set (Table 3 and Figure 3), suggesting a high performance of COVID-19 classification using the COVID-19 risk scores.

$\begin{array}{l}\text{T}\text{h}\text{e}\text{p}\text{a}\text{t}\text{i}\text{e}\text{n}\text{t}-\text{b}\text{a}\text{s}\text{e}\text{d}\text{r}\text{i}\text{s}\text{k}\text{s}\text{c}\text{o}\text{r}\text{e} \text{u}\text{s}\text{i}\text{n}\text{g}\text{r}\text{a}\text{d}\text{i}\text{o}\text{m}\text{i}\text{c}\text{f}\text{e}\text{a}\text{t}\text{u}\text{r}\text{e}\text{s}\text{o}\text{n}\text{l}\text{y}\\ =-3.785+19.563\times \text{G}\text{L}\text{R}\text{L}\text{M}\text{\_}\text{L}\text{R}\text{L}\text{G}\text{E}\text{\_}(25,90)\\ +0.002\times \text{I}\text{D}\text{\_}\text{G}\text{l}\text{o}\text{b}\text{a}\text{l}\text{\_}\text{M}\text{a}\text{x}\end{array}$ (2)

$\begin{array}{l}\text{T}\text{h}\text{e} \text{p}\text{a}\text{t}\text{i}\text{e}\text{n}\text{t}-\text{b}\text{a}\text{s}\text{e}\text{d} \text{r}\text{i}\text{s}\text{k} \text{s}\text{c}\text{o}\text{r}\text{e} \text{u}\text{s}\text{i}\text{n}\text{g} \text{c}\text{l}\text{i}\text{n}\text{i}\text{c}\text{a}\text{l} \text{f}\text{a}\text{c}\text{t}\text{o}\text{r}\text{s} \text{o}\text{n}\text{l}\text{y}\\ = -15.680+2.833\times \text{l}\text{e}\text{s}\text{i}\text{o}\text{n} \text{n}\text{u}\text{m}\text{b}\text{e}\text{r}+0.104\\ \times \text{l}\text{a}\text{c}\text{t}\text{a}\text{t}\text{e} \text{d}\text{e}\text{h}\text{y}\text{d}\text{r}\text{o}\text{g}\text{e}\text{n}\text{a}\text{s}\text{e}-1.674\\ \times \text{c}\text{r}\text{e}\text{a}\text{t}\text{i}\text{n}\text{e} \text{k}\text{i}\text{n}\text{a}\text{s}\text{e} \text{i}\text{s}\text{o}\text{e}\text{n}\text{z}\text{y}\text{m}\text{e}\text{s}\end{array}$ (3)

$\begin{array}{l}\text{T}\text{h}\text{e} \text{p}\text{a}\text{t}\text{i}\text{e}\text{n}\text{t}-\text{b}\text{a}\text{s}\text{e}\text{d} \text{r}\text{i}\text{s}\text{k} \text{s}\text{c}\text{o}\text{r}\text{e} \text{c}\text{o}\text{m}\text{b}\text{i}\text{n}\text{i}\text{n}\text{g} \text{r}\text{a}\text{d}\text{i}\text{o}\text{m}\text{i}\text{c}\text{s}\\ \text{a}\text{n}\text{d} \text{c}\text{l}\text{i}\text{n}\text{i}\text{c}\text{a}\text{l} \text{f}\text{e}\text{a}\text{t}\text{u}\text{r}\text{e}\text{s}\\ =-114.053+9.911\times \text{l}\text{e}\text{s}\text{i}\text{o}\text{n} \text{n}\text{u}\text{m}\text{b}\text{e}\text{r}+122.045\\ \times \text{G}\text{L}\text{R}\text{L}\text{M}\text{\_}\text{L}\text{R}\text{L}\text{G}\text{E}\text{\_}(25,90)+0.0196\\ \times \text{I}\text{D}\text{\_}\text{G}\text{l}\text{o}\text{b}\text{a}\text{l}\text{\_}\text{M}\text{a}\text{x}+0.334 \\ \times \text{l}\text{a}\text{c}\text{t}\text{a}\text{t}\text{e} \text{d}\text{e}\text{h}\text{y}\text{d}\text{r}\text{o}\text{g}\text{e}\text{n}\text{a}\text{s}\text{e}-7.593\\ \times \text{c}\text{r}\text{e}\text{a}\text{t}\text{i}\text{n}\text{e} \text{k}\text{i}\text{n}\text{a}\text{s}\text{e} \text{i}\text{s}\text{o}\text{e}\text{n}\text{z}\text{y}\text{m}\text{e}\text{s}\end{array}$ (4)

where GLRLM_LRLGE_(25, 90) represents the radiomic feature, long-run, low gray-level emphasis, which describes the distribution of the long homogeneous runs with low gray-levels within the image. The numbers in the brackets represent the parameters used to calculate that particular radiomic feature. ID_Global_max represents the radiomic feature intensity direct global max, which describes that the binary mask was preprocessed for the features derived directly from the image intensity. A detailed description of the parameters used is shown in Figure 2.

In contrast, the developed patient-based random forest models demonstrate comparable AUC values, precision, recall, specificity, F1, and accuracy as compared to the patient-based risk scores in the validation set (Table 3). The results of the decision curve analysis (DCA) to evaluate the clinical utility of the risk scores and the random forest models built in this study are shown in Figure 3E, 3F. The risk scores show a comparable clinical utility as compared to the random forest models.

Lesion-wise COVID-19 risk score with radiomic features only

To characterize different infectious lesions within the same patient, a lesion-based risk score using three radiomic features alone was also constructed (Figure 4 and Equation (5)). The utility of the risk score achieved an AUC value of 0.931 (95% CI: 0.898–0.956) (Table 4 and Figure 5A).

$\begin{array}{l}\text{T}\text{h}\text{e} \text{l}\text{e}\text{s}\text{i}\text{o}\text{n}\\ -\text{b}\text{a}\text{s}\text{e}\text{d} \text{r}\text{i}\text{s}\text{k} \text{s}\text{c}\text{o}\text{r}\text{e} \text{u}\text{s}\text{i}\text{n}\text{g} \text{r}\text{a}\text{d}\text{i}\text{o}\text{m}\text{i}\text{c}\text{s} \text{f}\text{e}\text{a}\text{t}\text{u}\text{r}\text{e}\text{s} \text{a}\text{l}\text{o}\text{n}\text{e}\\ = -55.389-6.769\times \text{G}\text{L}\text{C}\text{M}\_\text{C}\text{o}\text{r}\text{r}\text{e}\text{l}\text{a}\text{t}\text{i}\text{o}\text{n}\_(25,0,1)\\ +0.33\times \text{I}\text{D}\text{\_}\text{L}\text{o}\text{c}\text{a}\text{l}\text{\_}\text{R}\text{a}\text{n}\text{g}\text{e}\text{\_}\text{S}\text{t}\text{d}+0.136\\ \times \text{G}\text{O}\text{H}\text{\_}\text{P}\text{e}\text{r}\text{c}\text{e}\text{n}\text{t}\text{i}\text{l}\text{e}\_(15)\end{array}$ (5)

where GLRLM_Correlation_(25,0,1) represents the radiomic feature gray-level co-occurrence matrix with statistical measurement of correlation between a pixel and its neighbor over the whole image. The numbers in the brackets represent the parameters used to calculate that particular radiomic feature. ID_Local_Range_Std represents the radiomic feature of intensity direct in the neighborhood region, which describes the standard deviation among all the voxels. GOH_Percentile_(15) represents the radiomic feature gradient orient histogram, which describes the percentiles of the occurrence probability values in the histogram of the image. A detailed description of the parameters used is shown in Figure 4.

In contrast, the lesion-based weighted support vector machine (WSVM) model using the radiomic features only demonstrates a comparable AUC value, precision, recall, specificity, F1, and accuracy as compared to the lesion-based risk score (Table 4). The results of the DCA analysis to evaluate the clinical utility of the risk score and the WSVM model using radiomic features only are shown in Figure 5B.

Patients

This study was approved by the Hangzhou Xixi Hospital Institutional Review Board. As this is a retrospective study, the need for written informed consent from patients was waived. A total of 193 patients confirmed with COVID-19 or other types of viral pneumonia were enrolled in this study. Eight patients with negative chest CT imaging result were excluded. A total of 108 patients with COVID-19 confirmed by RT-PCR between December 2019 and March 2020 in the Hangzhou Xixi Hospital were retrospectively included into this study. Another group of 77 patients with influenza virus-induced, adenovirus-induced, syncytial virus-induced, and cytomegalovirus-induced pneumonias from Hangzhou First People’s Hospital (19 cases) and Hangzhou Xixi Hospital (58 cases) were used as controls.

The patients’ electronic medical data were retrieved from the Hospital Information System (HIS). The high-resolution CT images were retrieved from the picture archiving and communication system of the hospitals. The patients’ RT-PCR results were retrieved from the electronic medical records in the HIS. The patients with negative chest CT results or lacking both chest CT and RT-PCR examinations were excluded from this study. Figure 6 summarizes the study workflow and methods.

Baseline clinical data, including patient’s age, gender, lesion number, and five biochemical indicators recommended in the “Handbook of COVID-19 Prevention and Treatment,’’ [29] including white blood cell count (WBC), C-reactive protein (CRP) levels, creatine kinase isoenzyme (CK-MB) activity, lactate dehydrogenase (LDH) activities, and plasma D-dimer (DD) levels, were collected by reviewing the medical records and data of serial CT imaging, including baseline, mid-treatment, and post-treatment CT scans, were also recorded to monitor the disease progression. The patients’ daily basic status, daily examination results, and complications were also analyzed to check how the disease progressed. Based on the RT-PCR results for COVID-19 confirmation, the enrolled patients were divided into two groups, i.e., the COVID-19 group and the non-COVID-19 viral pneumonia group.

CT image acquisition

All patients included underwent chest CT imaging using a two multi-detector row CT system (GE Revolution Evo CT, Chicago, USA; Siemens SOMATOM Emotion 16, Erlangen, Germany). The acquisition parameters were as follows: 120/130 kV, 100/10–240 mA, 0.35- or 0.8-second rotation time, a layer spacing of 5 mm, an acquisition layer thickness of 5 mm, high-resolution reconstruction with a lung window layer thickness of 1.25/1.50 mm, a detector collimation of 16×0.5 mm or 64×0.625 mm, a field of view of 350×350 mm, and an image matrix of 512×512. The CT scans before onset of symptoms or CT scans done ≤1 week after symptom onset were used as baseline. The baseline CTs of highly suspected patients were used in this study. GGO and/or consolidation are the main manifestations in the CT images at this early stage. The other CT imaging patterns included linear opacity, mixed type and interstitial change patterns including septal thickening and fine reticular opacity, and other features including vascular thickening, crazy paving pattern, pleural thickening, and pleural effusion.

Human diagnosis of COVID-19 using a human supervised learning fashion

To compare the classification performance between the COVID-19 risk scores developed and radiologists, 147 radiologists were invited to differentiate COVID-19 from the virus-induced pneumonias based on the CT manifestations only. The diagnosis was performed using a human supervised learning fashion.

A total of five COVID-19 CT images and five influenza virus-induced, adenovirus-induced, syncytial virus-induced, and cytomegalovirus-induced pneumonia images were randomly drawn from the total patient cohort to form a learning sample set. These learning sample images along with the CT manifestations described in the China Clinical Consensus on Radiological Diagnosis on COVID-19 [29] were used to train 130 radiologists in China with thoracic CT diagnosis experiences ranging from assistant attending radiologists or associate attending radiologists to attending radiologists using a human supervised learning fashion. The radiologists were then given the remaining 176 CT images without the clinical and follow-up information provided. Based on the CT manifestations they learned from the 10 image samples provided, the radiologists diagnosed whether these 176 CT images were COVID-19 or influenza virus-/adenovirus-/syncytial virus-/cytomegalovirus-induced pneumonias. The accuracy and the average time for diagnosis per CT image were used for statistical analysis. To rule out the random diagnosis, two equal CT images were mixed within the 176 CT images, and if the radiologist’s answers were not consistent for these two images, his or her answers were excluded from the statistical analysis. A total of 130 radiologists’ diagnoses were eligible for statistical analysis (40 radiologists from Hubei Province, the epicenter of the COVIA-19 outbreak in China, and 90 radiologists from outside of Hubei Province).

Radiomic feature extraction

Before extracting all chest CT radiomic features, 3D adaptive histogram equalization enhancement (AHEE-3D) and edge preserve smooth 3D (EPS-3D) methods were used to remove random noise in the images. The lesions of pneumonias on CT images were reviewed and manually delineated by two experienced attending radiologists who were blind to the clinical and follow-up information. The final contour for each lesion was agreed upon by both radiologists. The patient-based and lesion-based analyses were performed. The lesion region of interest (ROI) was segmented on the CT image as the only input for radiomics analysis of pneumonia.

A total of 1766 radiomic features were extracted from each ROI delineated using the image biomarker explorer (IBEX) public platform developed by the University of Texas MD Anderson Cancer Center for feature extraction and classification of radiomic features [30, 31]. The radiomic features extracted include seven categories: shape, intensity direct, intensity histogram, gray-level co-occurrence matrix (2.5D and 3D), neighbor intensity difference (2.5D and 3D), gray-level run length matrix (2.5D), and intensity histogram Gauss fit.

It is believed that some of the radiomic features are sensitive to each step of the data processing procedure, including image acquisition settings, image reconstruction algorithm, and the digital image preprocessing procedure, so that the repeatability and reproducibility of the extraction of these radiomic features are easily compromised [32]. To account for the potential impact of the accuracy of radiomic feature extraction, the radiomic feature extraction procedure was repeated twice and Lin’s concordance correlation coefficient (CCC) tests were performed to assess the feature reproducibility in repeated feature extraction [33, 34]. Only the 1237 radiomic features showing high CCC values (CCC > 0.99) were used. With 1237 radiomic features selected, 510 features with null value were eliminated and the remaining 727 radiomic features plus 9 clinical variables (lesion number, age, gender, WBC, LC, CRP level, LDH activity, CK-MB activity, and plasma DD level) were used for further analysis.

Patient-based risk scores

Of the patient data, 25% was randomly selected as an independent validation set (n = 46) and the remaining 75% of the patient data were used for the training set (n = 139). The ratio of COVID-19 to non-COVID-19 patients was about 1.41:1 in the training and validation sets (in the training set, COVID-19:non-COVID-19 = 81:58 patients; in the validation set, COVID-19:non-COVID-19 = 27:19 patients).

For our patient-based analysis, the same features extracted from multiple lesions within one single patient were combined using a weighted power mean method [35]. Briefly, all lesions of the patient were delineated and the radiomic features were extracted. A weighting calculation was performed to combine the same feature from different lesions within the same patient as described in the following equations (Equation 6):

$\begin{array}{l}\text{F}(\text{j})={\displaystyle \sum _{\text{i}=1}^{\text{n}}\frac{{\text{V}}_{\text{i}}}{{\text{V}}_{\text{T}}}\cdot {\text{f}}_{\text{j}}(\text{i})}\\ {\text{V}}_{\text{T}}={\displaystyle \sum _{\text{i}=1}^{\text{n}}{\text{V}}_{\text{i}}},\end{array}$ (6)

where F(j) represents the value of the j-th radiomic feature of the patient, i represents the i-th lesion of the patient, V(i) represents the volume of the i-th lesion, n represents the number of lung lesions in the patient, V_T represents the total volume of all lesions in the patient, and f_j(i) represents the value of feature j in the i-th lesion. The weight assigned was based on the volume of the lesion. The larger the lesion volume, the greater the weight value of the features extracted from that lesion. Thus the contribution of the features extracted from this lesion to the patient’s radiomic feature was also greater.

Owing to the imbalanced sample distribution between COVID-19 and non-COVID-19 patients (the number of non-COVID-19 patient is lower than the number of COVID-19 patients), synthetic minority over-sampling technology [36–40] was used to generate synthetic non-COVID-19 patient samples in the training set so that a synthetically class-balanced training set could be achieved prior to training the models in this study. Briefly, for each minority sample “a” in the non-COVID-19 patient group, the synthesis strategy was applied to randomly select a minority sample “b” from its nearest neighbors. And then one point was randomly selected as the newly synthesized non-COVID-19 patient sample on the line between “a” and “b,” so that the ratio of COVID-19 and non-COVID-19 patients in the training set was close to 1:1.

Three signatures, including a risk score using radiomic features only, a risk score using clinical factors only, and a risk score combining radiomic features and clinical variables, were built in this study (Figure 7). For the construction of the risk score using radiomic features only and the risk score combining radiomic features and clinical variables, principal component analysis (PCA), the Mann–Whitney U test, and least absolute shrinkage and selection operator (LASSO) regression with a four fold cross-validation method and a 100 times iterative selection process were successively applied to eliminate redundant features and irrelevant variables to establish the COVID-19 risk scores. A multivariate logistic regression method was used to build these two risk scores. For the construction of the risk score using clinical factors only, because the number of clinical factors is much lower than the number of radiomic features, a multivariate logistic regression method was directly applied to build the clinical signature. The model performance of the three risk scores was evaluated in the training and independent validation sets.

For the construction of the risk score using radiomic features only and the risk score combining radiomic features and clinical factors, PCA was used to reduce the feature dimensionality and select the radiomic features and radiomic features plus clinical variables that accounted for 90% of the significant feature subset variability to increase the discriminative ability. After PCA analysis, the feature dimensionality was reduced from 727 radiomic features to 32 features for the risk score using radiomic features only (Supplementary Table 1), and from 727 radiomic features plus 9 clinical variables to 26 features for the risk score combining radiomic features and clinical factors (20 radiomic features plus 6 clinical variables including lesion number, gender, WBC, CRP level, LDH activity, CK-MB activity, and plasma DD level) (Supplementary Table 2).

The Mann–Whitney U test was used to further explore the potential association of the features/variables selected from PCA with COVID-19 and further reduce the feature dimensions. For the risk score using radiomic features only (Supplementary Table 3), the feature dimensions were reduced from 32 to 20 radiomic features. For the risk score combining radiomic features and clinical factors, the feature dimensions were reduced from 26 to 17 features (11 radiomic features plus 6 clinical variables including lesion number, gender, WBC, CRP level, LDH activity, and CK-MB activity) (Supplementary Table 4).

To select the most suitable features for classification of COVID-19, LASSO regression with a four fold cross-validation method and a 100 times iterative selection process was used to continually choose non-redundant and the most robust radiomic features and radiomic features and clinical variables, respectively [41, 42]. The coefficient of each variable was controlled by the parameter λ in the LASSO method and only the features with non-zero coefficients were selected. The misclassification error was calculated to minimize the binary classification error and maintain a balance of optimal classification performance and the optimal number of radiomic features needed for binary classification (COVID-19 vs. non-COVID-19).

As such, for the risk score using radiomic features only, only those 16 features with non-zero coefficients were selected via the LASSO process (Supplementary Table 5 and Supplementary Figure 1). For the risk score combining radiomic features and clinical variables, only those five features with non-zero coefficients were selected via the LASSO process (two radiomic features, GLRLM_LRLGE_(25,90) and ID_Global_Max, plus three clinical variables, lesion number, LDH activity, and CK-MB activity) (Supplementary Table 6 and Supplementary Figure 2).

After the feature dimensionality was reduced, a multivariable logistic regression analysis was employed and only the features with P < 0.001 in this process were selected to build the COVID-19 risk scores. For the risk score using radiomic features only, two radiomic features, GLRLM_LRLGE_(25,90) and ID_Global_Max, were finally preserved (Supplementary Table 7). For the risk score combining radiomic features and clinical variables, seven features were further reduced to five features (two radiomic features, GLRLM_LRLGE_(25,90) and ID_Global_Max, plus three clinical variables, lesion number, LDH activity, and CK-MB activity) (Supplementary Table 8). The COVID-19 risk scores using radiomic features only and using radiomics and clinical variables were built as the final classifiers by summing these features multiplied with their respective coefficients.

The COVID-19 risk scores developed were also represented by nomograms. The threshold of using the risk score using radiomic features only to classify COVID-19 is 0.2. The threshold of using the risk score with combined radiomic and clinical variables to classify COVID-19 is 0.5. DCA was applied to evaluate the clinical decision utility of the COVID-19 risk scores developed [34, 35]. The definition of net benefit in the DCA is described in Supplementary Appendix 1.

For the risk score using clinical factors only, the multivariable logistic regression analysis was directly employed and only the variables with P < 0.001 in this process were selected to build the COVID-19 risk score, including lesion number, LDH activity, and CK-MB activity (Supplementary Table 9). The COVID-19 risk score using clinical factors only was also represented by a nomogram. The threshold using a nomogram to classify COVID-19 is 0.5.

Patient-based random forest models

As a comparison to the patient-based COVID-19 risk scores developed, three random forest classifiers using radiomic features only, clinical factors only, and a combination of radiomic features and clinical factors were also constructed using grid search with fourfold cross-validation with the following parameters: the number of trees in the forest (ntree) = 500 and the maximum depth of the tree (mtry) = 3.

Lesion-wise COVID-19 risk score with radiomic features only

A lesion-based COVID-19 risk score using radiomic features alone was also built so that potentially different infectious lesions could be characterized individually. In total, 772 COVID-19 lesions were extracted from COVID-19 patients and 83 non-COVID-19 lesions were extracted from related viral pneumonia patients in this study.

The feature dimensionality reduction was conducted to select the optimal radiomic features. Briefly, a total of 1766 radiomic features were extracted from each lesion individually. After eliminating the radiomic features with null values and employing PCA, 32 radiomic features were used for the Mann–Whitney U test (Supplementary Table 10). The Mann–Whitney U test further reduced the feature dimensions from 32 to 20 features (Supplementary Table 11). The LASSO regression further selected the 10 non-redundant and most robust radiomic features (Supplementary Table 12 and Supplementary Figure 3).

After the feature dimensionality was reduced, the multivariable logistic regression analysis was employed to choose the radiomic features with P < 0.001 to build the lesion-based COVID-19 risk score with radiomic features alone so that 10 features were further reduced to 3 features: GLCM_Correlation_(25,0,1), ID_Local_Range_Std, and GOH_Percentile_(15) (Supplementary Table 13). The lesion-based COVID-19 risk score based on three features only was also represented by a nomogram. The threshold of using the risk score to classify COVID-19 is 0.5. DCA was also employed to evaluate the clinical decision utility of the nomogram developed [43, 44].

Lesion-based weighted support vector machine analysis

As a comparison to the lesion-based COVID-19 risk score using radiomic features alone, a lesion-based WSVM analysis was also conducted using the 10 radiomic features (Supplementary Table 12) selected by the LASSO. The data distribution between the COVID-19 and non-COVID-19 lesions (approximately 9.3:1) was extremely imbalanced. To ensure the models with predictive power were equally balanced between COVID-19 and non-COVID-19, a previously described strategy was used to adjust the distribution imbalance between COVID-19 and non-COVID-19 lesions and construct the WSVM [45].

Briefly, the strategy is to separate the major class (i.e., the COVID-19 lesion group in this study) into small subset groups size-comparable to the minor class (i.e., the non-COVID viral pneumonia lesion group in this study) to achieve a balanced distribution between the major class and the minor class; the COVID-19 lesion groups was randomly decomposed into nine partitions, and all the non-COVID-19 lesions were combined with each partition of COVID-19 lesions to form an individual subset so that the ratio of the COVID-19 and non-COVID-19 lesions was nearly 1:1 in each individual subset. In each individual subset, the total lesions were randomly separated into the training set (70%) and the validation set (30%).

The support vector machine (SVM) was trained independently with 10 radiomic features selected by the LASSO process within the training set of each subset. The weight for the SVM was determined via the recall value of the prediction using the validation set to reduce the false negative rate. In each individual subset, the SVM generated was validated with the validation set of each subset (i.e., the balanced data of each subset) as well as the validation set of the entire data to evaluate the classification performances (Supplementary Table 14 and Supplementary Figure 4).

Finally, all constituent SVMs were combined by summing constituent SVMs multiplied by weights determined, divided by the sum of the weights. The classical (metric) multidimensional scaling matrix (CMDScale) was used to demonstrate the correlation of features and COVID-19 for each constituent SVM.

Performance evaluation and statistical analysis

The AUC between the risk scores and the random forest models and the WSVM model was compared using the Delong test. Six metrics, including precision, recall (sensitivity), specificity, F1, accuracy, and AUC, were calculated from the receiver operating characteristic (ROC) curve with the model output.

The classification performances of COVID-19 by the developed risk scores were assessed by ROC analysis. For numeric variables, mean and standard deviation were calculated and the differences between COVID-19 and non-COVID-19 patient groups were compared using rank-sum tests. A two-tailed P value less than 0.05 was regarded as statistically significant.

Data sharing

The datasets analyzed in this study will be available from the corresponding author (Xiadong Li, email: lixiadong2019@outlook.com) at the time of publication. Per institutional policy, the datasets are designated limited access. Upon receiving access, the investigator may only use them for the purposes outlined in the request to the data provider, and redistribution of the data is prohibited.