A machine learning model was subjected to external validation to predict six-month mortality in patients with advanced solid tumors.
Disease prognosis based on machine learning or machine learning Logical machinery (LM) has proven to be a promising tool for facilitating conversations about serious illnesses between physicians and patients. To improve conversations between healthcare professionals and patients with advanced cancer at treatment decision points (TDPs), a team previously developed and internally validated an LM model that classifies patients as having a low or high probability of surviving the next six months if they start a new line of therapy.
The model takes into account 45 characteristics derived from electronic health record data using the Fast Healthcare Interoperability Resources (FHIR) standard, which guarantees interpretability and interoperability. This model seeks to be implemented in a tool to communicate and explain the disease prognosis based on machine learning in a transparent way.
In this sense, the external validation of a machine learning model is crucial to ensure its reliability before its clinical implementation; the researchers' objective was to externally validate the model taking into account recent patient data.
This study performed external validation using electronic health record data extracted from the University of Utah Health data warehouse corresponding to October 12, 2022. However, the ML model was originally trained on TDP from June 1, 2014 to June 1, 2020, and the researchers evaluated newly identified TDP between June 2, 2020 and April 12, 2022.
The analysis assessed the model's performance using the area under the curve and determined the positive predictive value, negative predictive value, sensitivity, and specificity at the predetermined risk threshold of 0.3. It also calculated quality metrics, such as referrals for palliative care or hospitalization, for patients classified as having a low probability of survival.
The results showed that a total of 1822 patients underwent 2613 TDPs. The development (4192 patients) and validation (1822 patients) datasets were similar, except that the patients in the validation set were younger and had different proportions of cancer types. Thus, there was no significant difference in mortality rates six months after the TDPs.
The model performed well in the validation data with an AUC of 0.80, similar to the results from the development phase. Furthermore, it identified a low probability of survival in the 8.7% group of patients with low survival rates. Among the patients with low survival rates, the authors made annotations for palliative care, hospice, and hospitalization.
The results of this study support the need for a tool to facilitate conversations about serious illnesses as patients and healthcare professionals consider new cancer therapies. This machine learning model demonstrated good predictive performance in recent data, similar to the development phase. However, as with most clinical studies, the results are limited by single-source data and a lack of racial and ethnic diversity among participants. Therefore, the authors suggest further evaluations across multiple healthcare systems to more accurately determine its effectiveness.
