SCALED Choices for Clinical Outcome Prediction

Machine learning (ML) has demonstrated a remarkable ability to identify complex patterns in healthcare data. Nonetheless, high predictive performance does not necessarily translate into clinically useful decision support. A disconnect often exists between how prediction models are designed and how they must operate in practice. Substantial methodological variation across studies makes it difficult to determine which ML design choices contribute to clinically useful and operationally realistic behavior. This uncertainty is important because prediction errors directly affect patient care and hospital resource allocation, and imbalances between false positives and false negatives can undermine decision-making.

This study evaluated key ML design choices across two prediction tasks derived from the MIMIC-IV database: emergency department (ED) disposition prediction and intensive care unit (ICU) readmission prediction. We evaluated the effects of data preprocessing, feature engineering, class imbalance handling, and decision threshold selection, comparing commonly used literature-based methodologies with alternative approaches designed to better preserve real-world clinical data characteristics. Model behavior was highly sensitive to these choices, with similar aggregate performance metrics often masking meaningful differences in prediction behavior, precision-recall tradeoffs, and clinical utility. Decision threshold tuning emerged as a critical mechanism for balancing false positives and false negatives, enabling flexible operating points for different clinical and operational priorities.

ML design choices can significantly influence model behavior and clinical utility. Clinically useful and operationally realistic prediction models should balance false positives and false negatives while remaining interpretable and robust to real-world data conditions. This work demonstrates the importance of carefully evaluating ML methodologies when developing clinical prediction systems.