Learnings from Silent Live Testing in a Hospital Environment
Knowledge database Technology Data management & digitalisation Human C.2: Reduction of idle times of MRI systems in radiology departments through the use of AI-based scheduling software - MRIdleBefore using predictive models in a hospital setting, it is essential to understand how they might function in practice. For this, we performed a “silent live test” (SLT) which allowed us to simulate the execution of daily predictions, without affecting anything hospital processes or patient behaviours. This test validated first and foremost the accuracy of the model predictions, but also helped test the technical readiness of the full pipeline, from data extraction through to prediction generation and sharing of these predictions with the nursing staff
Problem description, research question and relevance
Model validation using retrospective data alone does not ensure that a predictive model will function reliably in real-world settings. Many models fail to deliver impact due to issues beyond model accuracy: data pipeline failures, operational constraints, and communication breakdowns. A lack of real-world rehearsal often leaves researchers unaware of these risks until deployment.
Methods and procedures in the project
In this project, we implemented an SLT to simulate real-time use of a model predicting no-show risk for MRI appointments. Daily predictions were generated from upcoming schedules, but not shown to staff. These predictions were logged and later compared to actual attendance, without influencing patient care. This gave us a further round of model validation, as well as allowing us to test the full pipeline which would be used in future intervention studies.
Results and findings
The SLT surfaced critical technical and operational insights:
- Data drift: Some variables changed in distribution over time, degrading model accuracy.
- System issues: Initially some daily jobs failed due to various reasons, all of which we could solve in this test-environment, leaving us more confident in our pipeline for the intervention.
Recommendations for practice
- An SLT, or any prospective validation of a model, offers an essential further round of model validation, vital to ensure accuracy of the model in practice
- It enables early detection of data drift, allowing for retraining or model recalibration before performance degrades.
- System fragilities—especially around automation and scheduling—can be addressed while still in this test phase
A SLT acts as an important bridge between model development and real-world integration—without risking patient safety or a break down in operations.