Development and Testing of a Machine Learning Model Using ¹⁸F-Fluorodeoxyglucose PET/CT-Derived Metabolic Parameters to Classify Human Papillomavirus Status in Oropharyngeal Squamous Carcinoma

The incidence of oropharyngeal squamous cell carcinoma (OPSCC), including OPSCC associated with human papillomavirus (HPV) infection, has increased in recent decades [1, 2]. Patients with HPV have a more favorable prognosis and longer disease-free- and overall survival than OPSCC patients without HPV infection [3, 4]. Accordingly, the American Joint Committee on Cancer has adjusted its staging manuals and classified HPV-associated OPSCC as a distinct entity [5]. HPV status in patients with oropharyngeal cancer is determined by histopathological, immunohistochemical, and molecular diagnoses [4, 6]. However, these diagnoses are only feasible through biopsy, performed using laryngoscopes, or during surgical excision [7]. Predicting HPV status preoperatively based on imaging may aid in patient counseling and planning treatment strategies [8]. Additionally, imaging characteristics may provide further insights on the underlying pathophysiology of the distinct clinical course of OPSCC. Furthermore, if the number of specimens is not sufficient for diagnosis or the patient cannot undergo surgery, the HPV infection status of patients with OPSCC cannot be determined. Therefore, other non-invasive diagnostic methods may be helpful in treatment planning.

Many efforts have been made to predict HPV status in patients with OPSCC using machine-learning approaches based on imaging studies [9]. For example, radiomics-based investigations have used structural MRI, with and without diffusion-weighted imaging [10, 11, 12]. Few studies have used radiomics from ¹⁸F-fluorodeoxyglucose (¹⁸F-FDG) PET to determine the HPV status [13]. However, the radiomics approach is vulnerable to issues related to reproducibility and is time-consuming because of the multiple preprocessing steps and feature extraction processes [14]. Recently, machine learning has become indispensable in the development of predictive models. The performance of machine-learning models varies depending on the combination of various feature transforms, feature selection, oversampling, and machine-learning algorithms. Thus, there is a need for machine-learning approaches with good performance that can be easily applied in real clinical situations.

This study aimed to develop and validate a diagnostic model for HPV status using ¹⁸F-FDG PET/CT-derived parameters with an appropriate combination of feature selection, oversampling, and machine learning methods in OPSCC.

Among the 130 enrolled institutional patients, four were excluded (one without CT or MRI data, two with suboptimal PET images, and one with a damaged DICOM file). Finally, 126 patients were included in the analysis. In this internal cohort, 118 were men and eight were women (mean age, 60 years), and 103 (81.7%) and 23 (18.3%) were HPV-positive and HPV-negative, respectively. These patients were divided into training (n = 88) and validation (n = 38) sets in a 7:3 ratio. The external cohort was enrolled from the following two institutions: 7 from the Korea University ANAM Hospital and 13 from the Korea University GURO Hospital. One patient with suboptimal PET images was excluded from the Korea University ANAM Hospital. Of the 19 included patients (6 from Korea University ANAM Hospital and 13 from Korea University GURO Hospital; 16 male and 3 female; mean age, 65.3 years), 15 (78.9%) and 4 (21.1%) were HPV-positive and HPV-negative, respectively. The 19 patients were allocated to the external test set (Fig. 1). The proportions of each HPV status did not differ significantly between the training and internal validation sets or between the training and external test sets. Age, sex, and smoking history did not significantly differ between the training and external test sets. Patient demographics in the training, internal validation, and external test sets are presented in Table 1. The smoking history of seven patients from the external test set was unavailable in the EMR system; therefore, they were considered missing values.

Table 1
Patient Characteristics in the Training Set, Internal Validation Set, and External Test Set

• Click for larger image
• Click for full table
• Download as Excel file

A comparison of PET-derived parameters between HPV-positive and HPV-negative OPSCC patients showed a significant difference in the SUV_max in the internal dataset (all patients, n = 126). However, there was no statistical difference in the SUV_max in the external test set (Supplementary Table 1).

Various models were trained by combining various machine learning preprocessing methods and classifiers, and their performances in the validation set were calculated. Candidate models were drafted using the AUC in the internal validation set from each group using PET-derived parameters, clinical features, and both. ROC analysis was performed to identify the best models with the highest AUC in each group (Fig. 3). Regarding the AUC, the best model was Model 3, which used both PET-derived parameters and clinical features using the ExtraTreesClassifier, MinMaxScaler transformation, mutual information-based feature selection, and ADASYN oversampling. This model used five features: two PET-derived parameters (SUV_max and SUV_max-TLR) and three clinical features (age, smoking history, and tumor subsite). In the external test set, the area under the curve of the model was 0.78 in the external test set. The AUC of Model 3 was significantly higher than that of Model 1 (AUC = 0.48, p = 0.047). The AUC of Model 3 was also higher than that of Model 2; however, the difference was not statistically significant (AUC = 0.52, p = 0.142). The performance of the models for each group is summarized in Table 2. The sensitivity, specificity, accuracy, and F1 score of Model 3 were 100.0%, 50.0%, 83.3%, and 88.9%, respectively.

Fig. 3
ROC curves for the best Models 1, 2, and 3 in the external test set.
The values in the box in the lower right corner are the area under the ROC curve [95% confidence interval]. ROC = receiver operating characteristic

• Click for larger image
• Download as PowerPoint slide

Table 2
Performance of Models 1, 2, and 3 in the Internal Validation Set and External Test Set

• Click for larger image
• Click for full table
• Download as Excel file

Using the SHAP algorithm, the importance of the five parameters was visualized using a bar plot (Fig. 4). Smoking history had the most significant effect on prediction, followed by tumor subsite, age, SUV_max-TLR, and SUV_max. Analysis of the dot SHAP summary plots showed that a low PY for smoking history, tonsil location, low age, low SUV_max-TLR, and low SUV_max predicted HPV-positive OPSCC (Fig. 5).

Fig. 4
Bar summary plot of mean absolute Shapley values for five parameters of the best Model 3.
PY = pack-year, SHAP = Shapley additive explanations, SUV_max = maximum standard uptake value, SUV_max-TLR = tumor SUV_max to liver SUV_mean ratio

• Click for larger image
• Download as PowerPoint slide

Fig. 5
Dot summary plots by SHAP in training set from the best Model 3.
PY = pack-year, SHAP = Shapley additive explanations, SUV_max = maximum standard uptake value, SUV_max-TLR = tumor SUV_max to liver SUV_mean ratio

• Click for larger image
• Download as PowerPoint slide

In this study, an extratree-based model was developed to predict the HPV status of OPSCC using ¹⁸F-FDG PET/CT-derived metabolic parameters by combining the MinMaxScaler feature transform and ADASYN oversampling. Our model predicted HPV status by incorporating clinical features and easily obtainable PET/CT parameters, and its performance in an external test set was comparable to that of prior research employing radiomic approaches [13].

Age, smoking history, and tumor site were included as clinical predictors of HPV status in our model. Smoking is a well-known risk factor for OPSCC. However, smoking is not a cofactor for HPV-mediated oropharyngeal carcinogenesis [27]. HPV-positive OPSCC is more common than HPV-negative OPSCC among non-smokers [28]. The prevalence of HPV-positive OPSCC decreases with age [29]. Most cases involve individuals aged < 65 years [30, 31]. In a recent systematic review and meta-analysis, Haeggblom et al. [32] found considerable variation in HPV prevalence across the anatomic locations of the oropharynx. In tonsillar regions, the presence of a highly specialized crypt lymphoepithelium has been established as a permissive environment for HPV-driven carcinogenesis.

Various studies have been conducted using CT or MRI to determine HPV status. ¹⁸F-FDG PET/CT is superior to MRI and CT for evaluating locoregional neck node metastasis and distant metastasis. The National Comprehensive Cancer Network guidelines also highlight the usefulness of ¹⁸F-FDG PET/CT for head and neck cancer images [33, 34, 35]. ¹⁸F-FDG PET/CT is distinct from CT and MRI in that it allows for direct imaging of metabolism. Therefore, it is worthwhile to investigate the broader applications of PET/CT, such as HPV prediction. ¹⁸F-FDG PET/CT, in contrast, has some drawbacks, including non-uniformity in processing, difficulties in quality monitoring of equipment, and a higher cost than CT.

When comparing PET/CT-derived parameters, HPV-negative tumors showed a significantly higher SUV_max than HPV-positive tumors in our internal dataset, which is consistent with the results of previous literature [36]. The different histological characteristics of HPV-positive and HPV-negative OPSCC might account for the disparate SUV_max values, possibly because of the different distributions of hypoxic zones within the tumor [37, 38]. Studies on the relationship between FDG parameters and glucose transporter (GLUT) in malignant tumors are ongoing [39, 40, 41]. Although it is expected that the HPV state, expression level of GLUT, and FDG avidity are related, many unknown factors remain, demonstrating the complexity of OPSCC pathophysiology [42, 43]. SUV_max and SUV_max-TLR were the PET parameters used in the final model. SUV_max-TLR is a standardized method for semi-quantitative measurements to compensate for the weakness of the SUV values. This method may provide reliable and reproducible data as well as improve the PET characterization of tumors [44]. Few studies have attempted to classify HPV status using the radiomic approach of PET imaging [13]. Radiomics is time consuming because of the need for region-of-interest mapping, complex preprocessing, and feature extraction. This time-consuming and labor intensive process is a major disadvantage for clinical applications. Therefore, we only included parameters that were simple and generally easy-to-measure. In our model, which is faster and easier to apply, Only SUV_max and SUV_max-TLR were required.

A biopsy of the primary oropharyngeal lesion is required to diagnose OPSCC because many cases are treated surgically. Even after diagnosis, the process of obtaining tissue is inextricably included in the diagnosis and treatment. Therefore, the necessity of determining HPV status using ¹⁸F-FDG PET/CT before biopsy and treatment and its clinical implications may be questioned. Nevertheless, as with many previous studies that used image-based gene prediction, this study began with exploratory objectives. Additionally, while an acceptable predictive performance to replace biopsy was not reached in this investigation, the AUC might be improved in our ongoing study, which combines MRI and PET. Moreover, the ability to double-check for possible biopsy sampling errors might be advantageous. Finally, if there is a discrepancy between the HPV status determined by actual biopsy and the predicted HPV status using images, we believe that future investigations should focus on determining the cause of the discrepancy.

Numerous image-based HPV classification studies have recently been conducted using machine learning, which incorporates a diverse set of pre-processing methods and classification models. The performance of the model is determined by the classifier utilized, as well as the feature transformation, selection, and oversampling approaches. However, there is a risk of overfitting without sufficient external validation. Developing a model that learns and makes predictions using non-uniformly distributed data is a challenging task. The incidence of HPV-positive cancer has recently increased in OPSCC and now accounts for a significantly greater proportion than HPV-negative OPSCC [45, 46]. When dealing with unbalanced data, proper oversampling should be used to avoid overfitting in directions with higher frequencies. In our investigation, we discovered that when the oversampling method was utilized, the performance was superior to when no oversampling was employed. We selected a tree-based model for the machine learning classifier. This finding is consistent with an earlier study utilizing PET/CT radiomics, which frequently uses XGBoost [13]. The tree-based classifier appeared to be appropriate for PET-based HPV predictions. This conclusion will be useful in future large-scale studies.

This study had some limitations. First, the model’s performance for determining HPV status was insufficient to substitute tissue biopsy with a molecular diagnosis. Second, although our external test set was enrolled from two hospitals, the small number of patients, particularly in the external test set, is a major limitation of this investigation. In particular, the number of HPV-negative cases in the test set is small. This is well demonstrated by the absence of a statistical difference between Models 2 and 3, as well as an SUV_max difference between HPV-positive and HPV-negative cases in the external set. In future research, a larger test set size would be desirable to validate the robustness of the model. Images were obtained using the same reconstruction algorithm with the same settings whenever possible; however, using different scanners in different institutions was also a limitation. Finally, we used p16 as a surrogate marker for HPV positivity. However, p16 and HPV statuses can be mismatched [47, 48].

In conclusion, using appropriate preprocessing and a classifier, we developed and tested an HPV status classifier model based on metabolic parameters derived from ¹⁸F-FDG PET/CT and clinical parameters in OPSCC patients. It exhibited higher performance than models using either PET or clinical parameters alone.

The Supplement is available with this article at https://doi.org/10.3348/kjr.2022.0397.

Click here to view.^{(29K, pdf)}