Virtual reality and conversational AI to supplement patient education in chronic disease management
Knowledge database Structures & processes Technology Data management & digitalisation Human Training & digital expertise Patient-centred approach C.1: Improved self-management with virtual reality companions
Problem description, research question and relevance
Chronically ill patients, such as people with chronic kidney disease (CKD), have a great need for effective education and counselling. Targeted patient education can have a positive impact on the course of the disease and slow down the loss of kidney function. Embodied conversational agents (ECAs) in virtual reality (VR) offer an innovative way to effectively utilise both verbal and non-verbal elements of interpersonal communication.
The aim of the research was to develop a VR-based system called "RealCo" and to evaluate its acceptance by patients. RealCo combines immersive VR technologies with advanced language models and is intended to enable socially present, virtual counselling for CKD patients.
Methods and procedures in the project
The project centred on the use case of educating CKD patients about a new group of drugs, the so-called sodium-glucose co-transporter 2 (SGLT-2) inhibitors.
The methodological approach included:
- Analysing consultations with nephrologists to understand typical conversation patterns.
- Iterative development and testing of the VR system and chatbot in close collaboration with medical experts
- User experience study: Conducting three VR counselling sessions with a total of 15 CKD patients in a controlled, outpatient hospital environment over a period of seven months.
Technically, a multilinear storyboard for the consultations was developed in Twine and imported into the VR development environment Unity; a dynamic dialogue system based on a finite state machine controls conversations with the consulting avatar, whose utterances are generated in real time using GPT (via Microsoft Azure OpenAI) and converted from text to speech. Non-verbal communication is realised using Salsa and Unity Animator. User feedback and session data are systematically recorded in a back-end database.
Results and findings
- By combining GPT and the Langchain framework, the strengths of large language models can be effectively utilised without losing control of sensitive medical content.
- Successful prompt engineering requires a deep understanding of the goals and phases of the conversation in order to engage patients in dialogues in a targeted and committed manner.
Recommendations for practice
- Integration of VR and Conversational AI: Immersive VR environments combined with intelligent language models offer great potential for patient education and should be integrated into future training programmes.
- Careful design of dialogues: Targeted prompt engineering and structured dialogue management are crucial to successfully engage patients* and build trust in virtual consultants.
- Interdisciplinary development teams: Close collaboration between technologists and healthcare professionals is essential to develop valid and practical solutions for the healthcare sector.
Literature and other sources
Brucker-Kley, E., Michot, J., Keller, T., Scherer, C., Segerer, S. (2024). Virtual Reality and Conversational AI for Complementing Patient Education in Chronic Disease Management. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2024. Lecture Notes in Computer Science, vol 15028. Springer, Cham. https://doi.org/10.1007/978-3-031-71704-8_27