RealCo - Virtual Reality Companion for patients with chronic renal failure
Knowledge database Technology Data management & digitalisation Human Patient-centred approach C.1: Improved self-management with virtual reality companionsMore than 30 years after science fiction author Neil Stephenson [1] coined the term "metaverse", the vision of shifting our social interactions into a virtual reality is realisable. Avatars in combination with powerful chatbots make it possible to simulate interpersonal communication with its verbal and non-verbal elements in virtual reality (VR). But can we imagine seeking advice from an intelligent machine embodied by an avatar?
Problem description, research question and relevance
Chronically ill patients have a high need for counselling, as they require continuous medical support and education in order to understand and effectively influence the course of their disease [2]. This also applies to patients with chronic renal insufficiency (CRF), which is one of the ten leading causes of death worldwide [3]. In Switzerland, CKD affects around one in ten adults, particularly older people and patients with diabetes and high blood pressure [4]. As CKD cannot be cured, the focus is on slowing the progression of the disease and delaying kidney failure in order to reduce treatment costs and improve patients' quality of life [5]. Research has shown that patient education and training can have a positive impact on disease progression and, in the case of CKD, delay the onset of dialysis and prolong survival [2], [6].
In view of the increasing number of CKD patients and the shortage of healthcare professionals, the use of digital solutions for patient education and training is obvious. Mobile apps or chatbots designed to increase health literacy or trigger behavioural change have been shown in studies to be effective interventions and supplements to traditional patient education [7], [8]. With the commercial availability of all-in-one VR goggles, the visual and interactive possibilities of virtual reality are used to visualise complex medical issues or to train skills, such as performing peritoneal dialysis at home [9], [10]. Avatars in VR offer additional possibilities for such digital interventions to enhance the qualities of a personal conversation. A conversation with an avatar that combines the verbal skills of a chatbot with gestures, facial expressions and posture gives patients the feeling of social co-presence in a three-dimensional virtual space. It is known that avatars increase the effectiveness of conversation-based interventions for patients with chronic illnesses [11]. If the verbal skills of these avatars are not realised with classic rule-based Q&A chatbots, but with modern language models and transformers such as GPT, they can create a natural conversational experience. Counselling avatars in VR can thus become affective "companions" that adapt to the individual situations and sensitivities of their human counterparts [12]. But are patients ready to engage with this completely new form of human-machine interaction? And how must such a consultation with a VR companion be designed so that it is successful, i.e. that patients perceive the VR companion as usable and useful and medical professionals see it as a relief in everyday clinical counselling?
Methods and procedures in the project
In order to evaluate the acceptance of a VR Companion for chronically ill patients, a prototype was developed iteratively as part of the project and tested with medical experts from Aarau Cantonal Hospital (KSA).
The use case is patient education on the sodium-glucose co-transporter 2 (SGLT-2) inhibitor drug group. Clinical studies have only shown since 2019 that the SGLT-2 inhibitor drug group, which was originally developed for the treatment of diabetes, slows down the loss of kidney function [13]. SGLT-2 inhibitors subsequently became relevant for a large number of CKD patients. As patient information is primarily focused on the treatment of diabetes, nephrologists are required to effectively communicate this knowledge about SGLT-2 inhibitors to a heterogeneous patient group during personal consultations [14]. This is where the VR companion "RealCo" comes in. RealCo enables CNI patients to understand how SGLT-2 inhibitors work and their side effects.
Figure 1: Communication model from social robot research
The user experience, i.e. the effect of Realco on CNI patients, will be evaluated in a qualitative laboratory experiment with 15 test subjects from the nephrology department of Aarau Cantonal Hospital, who will each undergo 3 sessions.
The research framework is based on a communication model from social robot research, which was extended to include the visual aspects of VR [15], [16] (see Figure 1).
Results and findings
The prototype is a VR app (see Figure 2) that was developed in Unity and is used with Meta Quest 3 VR glasses. The architecture comprises the following components:
- The consultation dialogue comprises 4-5 phases defined by the multilinear script (Twine) (1.greeting/warm-up, 2.basic information on SGLT-2 inhibitors, 3.free dialogue, 4.question on skill gain, 5.farewell).
- Free dialogue modelled as a state machine (finite state machine) that controls the interaction between patient and Realco by defining the states of the dialogue (idle, listen, speak, think) and the transitions between these states.
- Language Model API: GPT 4.0 via Microsoft Azure OpenAI (Switzerland)
- Vector database: LangChain manages the controlled SGLT2 knowledge corpus and extracts relevant information from it when a question is asked, which is then integrated into the prompts for GPT, which provides contextually appropriate and natural language answers enriched with this input.
- Speech-to-text and text-to-speech via Microsoft Azure.
- Interactive user feedback on the user experience is integrated into the VR app.
Recommendations for practice
- The development and validation of a knowledge corpus with doctors who regularly conduct patient consultations on the topic ensures that the dialogue system has the most up-to-date and well-founded information and uses technically correct and patient-appropriate vocabulary. In this way, the strengths of language models and transformers such as GPT are utilised without losing control of the content.
- Effective prompt engineering requires that the objectives and phases of the specific type of conversation are understood and defined. This is the only way to get the dialogue with the avatar going and keep it going. Patients do not ask questions simply because a dialogue system in the form of an avatar is available to them. They want to be greeted, guided through the dialogue, encouraged to ask questions and bid farewell.
- A pragmatic operating model for a VR companion starts in the first phase in the context of a hospital. Patients use VR glasses in the hospital, a day clinic or doctor's surgery and can thus make good use of waiting times. The vision of a 24/7 location-independent consultation by the VR companion can become reality as soon as the continuity of the consultation is guaranteed. This requires patient information to be integrated into the dialogue and the system architecture. A hybrid model that enables escalation to real specialists within the VR application, for example provided by telemedicine service providers, would increase the flexibility and security of virtual counselling.
Literature and other sources
[1] N. Stephenson, Snow crash. in Bantam spectra book. New York: Bantam Books, 1992.
[2] A. S. Narva, J. M. Norton, and L. E. Boulware, "Educating Patients about CKD: The Path to Self-Management and Patient-Centered Care," Clin J Am Soc Nephrol, vol. 11, no. 4, pp. 694-703, Apr. 2016, doi: 10.2215/CJN.07680715.
[3] WHO, "The top 10 causes of death," Global Health Estimates. Accessed: Sep. 28, 2022 [Online]. Available: www.who.int/news-room/fact-sheets/detail/the-top-10-causes-of-death
[4] Forni Ogna, "Prevalence and determinants of chronic kidney disease in the Swiss population," Swiss Medical Weekly, no. 17, Apr. 2016, doi: 10.4414/smw.2016.14313.
[5] P.-Y. Martin, "Advances in nephrology: beneficial for patients and costs," Swiss Medical Journal, vol. 98, no. 45, pp. 1484-1486, Nov. 2017, doi: 10.4414/saez.2017.06160.
[6] J. A. Wright Nunes, K. A. Wallston, S. K. Eden, A. K. Shintani, T. Alp Ikizler, and K. L. Cavanaugh, "Associations among perceived and objective disease knowledge and satisfaction with physician communication in patients with chronic kidney disease," Kidney International, vol. 80, no. 12, pp. 1344-1351, Dec. 2011, doi: 10.1038/ki.2011.240.
[7] M. Milne-Ives, C. Lam, C. D. Cock, M. H. V. Velthoven, and E. Meinert, "Mobile Apps for Health Behaviour Change in Physical Activity, Diet, Drug and Alcohol Use, and Mental Health: Systematic Review," JMIR mHealth and uHealth, vol. 8, no. 3, p. e17046, Mar. 2020, doi: 10.2196/17046.
[8] L. Laranjo et al, "Conversational agents in healthcare: a systematic review," Journal of the American Medical Informatics Association, vol. 25, no. 9, pp. 1248-1258, Sep. 2018, doi: 10.1093/jamia/ocy072.
[9] D. M. Goldsmith, S. Flash, J. Holdnack, and P. F. Brennan, "Designing Immersive Virtual Reality Environments for Supporting Patients at Home: Translating Input From Home Care Nurse Experts to Design Requirements," CIN: Computers, Informatics, Nursing, vol. 41, no. 3, p. 142, Mar. 2023, doi: 10.1097/CIN.0000000000000895.
[10] Weltenmacher, "Dialysis Orientation," Weltenmacher. Accessed: May 28, 2024 [Online]. Available: weltenmacher.de/en/products/dialysis-orientation/
[11] P. Wonggom, C. Kourbelis, P. Newman, H. Du, and R. A. Clark, "Effectiveness of avatar-based technology in patient education for improving chronic disease knowledge and self-care behaviour: a systematic review," JBI Database of Systematic Reviews and Implementation Reports, vol. 17, no. 6, pp. 1101-1129, Jun. 2019, doi: 10.11124/JBISRIR-2017-003905.
[12] S. Biundo and A. Wendemuth, "From cognitive technical systems to companion systems," Künstl Intell, vol. 24, no. 4, pp. 335-339, Nov. 2010, doi: 10.1007/s13218-010-0056-9.
[13] H. J. L. Heerspink et al, "Dapagliflozin in Patients with Chronic Kidney Disease," New England Journal of Medicine, vol. 383, no. 15, pp. 1436-1446, Oct. 2020, doi: 10.1056/NEJMoa2024816.
[14] T. Maddox and C. Chmielewski, "Virtual Reality in Chronic Kidney Disease Education and Training...Maddox T, Chmielewski C. Virtual Reality in Chronic Kidney Disease Education and Training. Nephrology Nursing Journal. 2022;49(4):329-381," NEPHROL NURS J, vol. 49, no. 4, pp. 329-381, Aug. 2022, doi: 10.37526/1526-744X.2022.49.4.329.
[15] E. Broadbent, D. Johanson, and J. Shah, "A New Model to Enhance Robot-Patient Communication: Applying Insights from the Medical World," in Social Robotics, S. S. Ge, J.-J. Cabibihan, M. A. Salichs, E. Broadbent, H. He, A. R. Wagner, and Á. Castro-González, Eds, in Lecture Notes in Computer Science. Cham: Springer International Publishing, 2018, pp. 308-317. doi: 10.1007/978-3-030-05204-1_30.
[16] K. Loveys, C. Hiko, M. Sagar, X. Zhang, and E. Broadbent, "'I felt her company': A qualitative study on factors affecting closeness and emotional support seeking with an embodied conversational agent," International Journal of Human-Computer Studies, vol. 160, p. 102771, Apr. 2022, doi: 10.1016/j.ijhcs.2021.102771.