Value in Healthcare and Education: The Potential of Surgical Training Based on Immersive Virtual Reality

Danny P. Goel, Clinical Professor, UBC Department of Orthopedic Surgery and CEO, Founder, PrecisionOS, and Ryan Lohre, Massachusetts General Hospital and Harvard Medical School

Contact: danny@precisionostech.com

This paper is dedicated to my late friend, mentor and Professor, Dr. William Mitchell, who offered me the most precious commodity, his time. — Danny P. Goel

Value-based innovation which is disruptive should deliver better results at a lower cost. In healthcare economics, value is considered from the viewpoint of the customer or stakeholder. Thus, the stakeholder can be one or more entities including the patient, the physician, the insurer, the hospital, and/or the vendor. Porter and Teisberg (1) have described the value equation as the unit of cost expended to treat a medical condition. Increasing the relative quality of care at the same cost increases value. This can be following best practices, evidence-based approaches and equipment, and employing highly trained teams. A study by Warner and Higgins (2) examined the volume-outcome relationship of practicing orthopaedic surgeons and demonstrated a direct effect on outcomes and costs. More experienced surgeons demonstrated lower complications and cost savings over less experienced surgeons. It thus stands to reason that any method which can increase surgeon skill quickly may have downstream effects on value. In fact, Porter and Kaplan (3) have described this added value of expertise as the “virtuous circle of value”.

Unlike the endpoint of a surgical procedure, value definitions in medical education are less clearly defined. This is due to the lack of consensus in (1) design and reporting of educational studies (2) consensus of outcome measures (3) ethical limitations of study on patients with trainees (4) financial constraints to longitudinal study and (5) lack of economic evaluations. As a result, educational programs proport to achieve a competency-based approach to balancing clinical, operative, educational, and non-clinical duties but lack evidence of effectiveness. Given the current hourly work limitations for surgeons in training, more educators would agree however, that a training program optimizing skill learning outside of clinical experience would provide added value.

Providing effective training, demonstrated through measurable outcomes of technical and non-technical skill in a cost-conscious framework can be considered a means of pushing the productivity frontier in education. Figure 1 demonstrates the theoretic productivity frontier as a function of quality and cost, with addition of immersive virtual reality (IVR) technology shifting the curve as an example. Multiple studies using IVR in surgical education demonstrate efficient and effective skill acquisition in cost-conscious frameworks, thus reducing costs and increasing quality.(1,2)(3,4)(5)

Figure 1. Productivity frontier for educational cost with addition of IVR technology

In education, stakeholders include medical students, residents, fellows, attending staff, program directors, university training programs, private and public hospital structures, patients, and industry vendors. Focus on optimizing educational value is largely considered in the context of improving learning outcomes for trainee stakeholders. Value may thus be considered as providing attainable proficiency through time and cost-effective means which is more effective than alternative training schedules.

As many institutions are publicly funded, educational economics requires a  balance between competing clinical demands and time spent improving surgical skills. Determining high-value, low-cost training curricula is thus the downstream goal of educational research. With this in mind, there are limitations of competency (or even proficiency) based models as trainees can be sufficient in either but lack the collective skills to operate independently.(6) Practically, studies examining skill acquisition, retention, and transfer of training to real scenarios provide the best evidence of value-added training.

Global rating scales (GRS) of performance, task specific check lists, and technologically assisted motion analyses have been used as reproducible and reliable outcome measures for tracking skill and comparing skill to experts. These outcome measures including Objective Structured Assessment of Technical Skills (OSATS) transfer of training (ToT), transfer effectiveness ratios (TER), and incremental transfer effectiveness with repetitive use (ITER) of training apparatus’ provides an insight into the downstream effects of training. Figure 2 demonstrates a proposed value framework of quality and cost considerations in orthopaedic educational study based on Porter and Kaplan’s value equation. This framework may provide definitions of high-value, low-cost training solutions.

Training Models in Orthopaedics and Evolution to Immersive Virtual Reality Simulation

Current competency-based medical education (CBME) in practice can suffer from poorly defined endpoints of “competency.” Proficiency-based training has been described as a more appropriate means of training as outlined by Gallagher.(7) Proficiency-based education provides expert surgical benchmarks of performance that trainees would practice toward, and once reached, able to be replicated over periods of time. This model has been shown in studies examining proficiency of trainees in performing knee arthroscopy and shoulder instability surgery.(8)(9) Both training formats focus on training outcomes, and industry and academia have leveraged simulators as an adjunctive and complementary learning format. While progressive, evidence predominantly exists for partial task training benchmarks as ethical principles prohibit patient-related evaluations. If we consider the entire patient, no simulation model presents an opportunity to learn in whole task training.

Immersive VR is a novel technology that incorporates advanced hardware and software to produce realistic, simulated training environments. Hardware allows for high-quality audiovisuals in an immersive and 3-dimensional setting through use of a head-mounted display equivalent to consumer electronics. Position tracking controllers allow for controlled movement in space while providing tactile feedback through the concept of haptics. Software flexibility allows for very specific operative scenarios to be developed. The combination of hardware and software allows for immersion previously unseen in orthopaedic education. Figure 3 demonstrates examples of this technology through multiple operative scenarios. The technology is scalable to training level and allows for tracking of user performance with multiple users from across the world in the same scenario. The system also provides ample opportunity for research and development with the ability to track motion, performance through proficiency and task-specific outcome measures, and frequency and duration of use.

Currently, evidence towards both non-technical and technical skill acquisition using IVR has been demonstrated in single series studies from both acute and longitudinal use.(1,2)(4) Based on these studies, IVR has the highest evidence-based level of recommendation for use in surgical training. Further study, including replicative multi-center studies with various training populations are required to further the quality of evidence to drive the value equation of IVR and are currently being undertaken at multiple academic training institutions in North America.

Economic Evaluation of Surgical Training – Direct and Indirect Costs

To fundamentally prove the value of simulation, economic evaluations carefully describing opportunity costs and clear, measurable technical and allocative efficiency are required. Yet, true economic evaluations are exceedingly limited in medical education. Only 1.6% of studies examining simulation technology in medical education describe comparative costs.(10) Haines et al. provided technical definitions of economic evaluation for an audience of clinical educators, promoting structured cost analyses similar to clinical medicine.(11) Only through a clear understanding of value added through quantifiable quality and cost-effective training solutions as seen in Figure 2, can programs understand and incorporate new innovative technologies.

Figure 2. A representative value framework for orthopaedic education based on Porter and Kaplan’s principle of value in healthcare

Costs of surgical training are difficult to ascertain. Training costs can be considered both direct and indirect. Direct costs to training include trainee salary while indirect costs include a myriad of educational and clinical parameters. Variability in regional training, including differences of educational staff, facilities, clinical and operative duties, and salary, make concrete training cost tabulation difficult. Zendejas et al. demonstrate in a systematic review the paucity of literature on cost-effectiveness in medical education compared to clinical medicine, postulating that most policy makers lack understanding and the training necessary to measure this.(10) A study by Calhoon et al. clearly demonstrated this through analysis of perceived program costs of running six thoracic surgery programs by program directors.

These perceived costs were significantly higher than accounting costs, by on average USD 483,000 per resident per year. Total training costs ranged from USD 330,000 to USD 667,000 per resident per year.(12) A study involving six surgical subspecialty residency training programs at the University of Tennessee Medical Centre-Knoxville showed lost OR time equating 11 184 minutes per resident in a five-year model.(13) Considering a mean generalizable OR time of USD 37/min, and larger centers having upwards of 4 million minutes of OR time annually, this may amount to significant financial losses.(14) It seems that current training models are at odds with sound fiscal strategy.

Simulation endorsement hopes to shift training time away from costly operative scenarios. These simulation strategies have garnered investment in specialized, staffed simulation centers. Equipment in these facilities are costly to both purchase, use, and maintain and typically include cadaveric specimens, bench-top, and varying fidelity simulators such as laparoscopic trainers. Weinstock et al. reported producing a pediatric-specific, 436-sq-ft laboratory simulation center for a construction cost of USD 472,000. Operation of this facility cost USD 67,875 yearly and did not comment on staffing costs.(15) Calhoon et al. demonstrated that simulation costs were up to USD 80,000 per resident per year in some thoracic surgery programs, equaling roughly 6.5% of the total educational cost.(12)

Creators of the successful Israel Center for Medical Simulation (MSR) clearly describe reliance on private, governmental, and professional organization financial support to cover startup costs and ongoing maintenance fees.(16) Nousiainen et al. evaluated simulation costs associated before and after implementing CBME.(17) Their institution estimated a total simulation cost of USD 1,856 per resident per year which increased by 15 times following CBME to USD 27,850. The main costs were cadavers (43%), materials (27.4%), staff labor (18.8%), and simulation models (8.5%). Of note, the USD 200 per hour of simulation center use was not factored into the above calculations and would likely be significantly higher if added. Industry partners typically help offset costs of these simulation centers through equipment use and contracts.

Advantages of Immersive Virtual Reality in Surgical Education

Direct buy-in cost of IVR hardware is approximately USD 300-USD 500 depending on headset with variable licensing costs. Few IVR related studies present cost or value consideration. Hooper et al. describe a USD 4,000-USD 8,000 software licensing fee in their test-retest study of fourteen junior orthopaedic trainees. Though improvements were seen in technical skill of the IVR trained cohort, neither cost analysis nor effectiveness was discussed.

Lohre et al. recently demonstrated significant improvement of validated outcome metrics for the use of IVR in learning shoulder arthroplasty. By comparing improvements in measurable outcomes, time of operative completion and learning, and cost between learning modalities of IVR relative to control, numerical representations of skill and cost were produced. They demonstrated that IVR training could supplement for up to 47 minutes of comparative 60 minutes of real operating room training, account for up to 51 operative cases, thereby shifting early learning curves, and be 34x more cost effective than attending a traditional cadaver-based course. To accomplish this, they utilized transfer of training, transfer effectiveness, and cost effectiveness ratios popularized in modern military simulation and training literature. (18)

A recent case report elegantly highlighted an effective use of IVR simulation for surgical training and showed clear transferable skill to the real world. A senior trainee with limited specific procedural experience utilized the Precision OS IVR simulator multiple times outside of regular duty hours in preparation for the real-life case. Through sequential virtual improvements in completion time, technical accuracy, and reduced x-ray use, the resident was able to subsequently complete the procedure under supervision. The case is particularly interesting as an original treating surgeon failed an attempt of fixation and with IVR training, the resident was able to complete the procedure safely, with more robust fixation, and 7.3x less radiation exposure than the index. Though a single experiential case, the benefits of active and deliberate practice are clear. Virtual training in this case demonstrated the potential for cost-effectiveness by providing effective training and clearly superior outcomes.

Effective, evidence-based simulators that forego costly infrastructure investments such as IVR provide a contrasting philosophy to these institutional investments. Cadaver-based centers do not provide evidence of benefits in skill training despite widespread acknowledgement as a “gold standard” in simulation.(19) Large, stationary simulators such as minimally invasive arthroscopic models for knee or shoulder surgery are also available and are effective in skill acquisition and retention in longitudinal research.(20)

These units, however, suffer from infrastructure and maintenance requirement costs. One study provided acquisition costs of USD 137,000 for equipment, installation and warranty fees not-withstanding further maintenance and housing costs. An additional study demonstrated that the stationary simulator would have to be used for over 300 hours per year to be cost-effective when compared to learning on cadavers. This extraneous time commitment may be unreasonable for trainees.

IVR simulators in contrast have minimal buy-in fees, are portable, scalable to multiple procedure types, and similarly effective. Furthermore, training is objectively efficient, with most module completion times in studies occurring on the order of minutes. The demonstrable improvement in skill and comparative cost effectiveness to cadavers or cumbersome stationary simulators provides significant value to educational institutions and is truly a disruptive innovation in this area.

Future directions include further cost analyses such as return on investment (ROI), break-even, cost-benefit, willingness to pay thresholds, discrete event simulation (DES), and time-driven activity-based costing (TDABC). As larger professional organizations and industry leverage IVR technology in their teaching portfolios, cost analyses such as willingness to pay and break-even analyses will further clarify the role compared to traditional offerings.

Medical device companies spend upwards of USD 14 million (median 5 million) per year predominantly on staffing, rental, and equipment costs.(21) These courses require travel and lost wages and largely focus on surgeons. There is an obvious need for cumulative OR team training including nurses and assistants as high-functioning teams produce the most consistent results.(22)

Using DES, costs of traditional training programs could be compared to those incorporating IVR using industry standard modeling software. This process has been used previously in health care optimization, but not regarding surgical educational structures or trainee performance in orthopaedics. Menendez et al. have shown TDABC useful in discerning cost containment strategies using shoulder arthroplasty data.(23) If educational programs treated their trainees as actionable investments, optimization using TDABC would be possible. This would require transparent educational costs and measurement of performance data using the myriad of available outcome measures in proficiency-based educational platforms. Regression modeling could then aid in determining cost-effective solutions to improving trainee performance.

Immersive VR may assist in this in the future, as the software is capable of tracking proficiency data of users over time. Studies currently are being undertaken to continue validating the use of these IVR proficiency scales to real user performance and may prove to be a valuable, cost-effective measurement of skill progression

Looking Forward

Surgical education follows a traditional structure. Though this educational approach has produced generations of excellent surgeons, current training models arise only from incremental innovation. In the preceding decade, publications in surgical education have focused on simulation training to establish transfer validity of attained skill to real operative scenarios.

Despite the quality of these studies, a key variable is the limited reporting around economic value. To promote adoption of innovative training technologies, programs with limited resources require concise value propositions. Value should be derived from standardized outcome measures in high-quality research, coupled with high-quality economic evaluations. Immersive VR has proven itself to be a disruptive innovator in surgical education. Trainees require realistic, effective, cost-conscious solutions to training. Immersive VR has the potential to provide a reliable solution to these needs.

References

1. Lohre R, Bois A, Athwal GS, Goel DP, (CSES) TCS and ES. Improved Complex Skill Acquisition by Immersive Virtual Reality Training: A Randomized Controlled Trial. J Bone Jt Surg. 2020;102(6):e26.
2. Lohre R, Bois A, Pollock J, Lapner P, McIlquham K, Athwal GS, et al. Effectiveness of Immersive Virtual Reality for Orthopaedic Surgical Skills and Knowledge Acquisition: A Randomized Controlled Trial. JAMA Netw Open. 2020;3(12):e2031217.
3. Logishetty K, Gofton W, Rudran B, Beaule P, Cobb J. Fully Immersive Virtual Reality for Total Hip Arthroplasty. J Bone Jt Surg. 2020;102(6):e27.
4. Logishetty K, Wade GT, Rudran B, Beaule PE, Gupte CM, Cobb JP. A Multicenter Randomized Controlled Trial Evaluating the Effectiveness of Cognitive Training for Anterior Approach Total Hip Arthroplasty. J Bone Jt Surg. 2020;102(2):pe7.
5. Hooper J, Tsiridis E, Feng JE, Poulsides L, Macaulay W, The NYU Virtual Reality Consortium. Virtual Reality Simulation Facilitates Resident Training in Total Hip Arthroplasty: A Randomized Controlled Trial. J Arthroplasty. 2019;34(10):2278–83.
6. Elfenbein D. Have We Created a Crisis in Confidence for General Surgery Residents? A Systematic Review and Qualitative Discourse Analysis. JAMA Surg. 2016;151(12):1166–75.
7. Gallagher AG, O’Sullivan GC. How to Develop Metrics from First Principles. In: Apell P, editor. Fundamentals of Surgical Simulation: Principles and Practices. Springer US; 2012. p. 133–40.
8. Angelo R, Ryu R, Pedowitz R, Beach W, Burns J, Dodds J, et al. A Proficiency-Based Progression Training Curriculum Coupled With A Model Simulator Results in the Acquisition of a Superior Arthroscopic Bankart Skill Set. Arthroscopy. 2015;31:1854–71.
9. Bhattacharyya R, Davidson D, Sugand K, Bartlett M, Bhattacharya R, Gupte C. Knee Arthroscopy Simulation: A Randomized Controlled Trial Evaluating the Effectiveness of the Imperial Knee Arthroscopy Cognitive Task Analysis (IKACTA) Tool. J Bone Jt Surg. 2017;99(19):e103.
10. Zendejas B, Wang AT, Brydges R. Cost : The missing outcome in simulation-based medical education research : A systematic review. Surgery [Internet]. 2011;153(2):160–76. Available from: http://dx.doi.org/10.1016/j.surg.2012.06.025
11. Haines T, Isles R, Jones A. Economic consequences in clinical education. Focus Heal Prof Educ A Multi-disciplinary J [Internet]. 2011;12(3):53–63. Available from: http://search.informit.com.au/documentSummary;dn=908656457525740;res=IELHEA
12. Calhoon JH, Baisden C, Holler B, Hicks GL, Bove EL, Wright CD, et al. Thoracic surgical resident education: A costly endeavor. Ann Thorac Surg [Internet]. 2014;98(6):2012–5. Available from: http://dx.doi.org/10.1016/j.athoracsur.2014.07.044
13. Bridges M, Diamond DL. The financial impact of teaching surgical residents in the operating room. Am J Surg. 1999 Jan;177(1):28–32.
14. Childers PC, Maggard-Gibbons M. Understanding Costs of Care in the Operating Room. JAMA Surg. 2018;90095.
15. Weinstock PH, Kappus LJ, Garden A, Burns JP. Simulation at the point of care: Reduced-cost, in situ training via a mobile cart. Pediatr Crit Care Med. 2009;10(2):176–81.
16. Ziv A, Erez D, Munz Y, Vardi A. The Israel Center for Medical Simulation : Acad Med. 2006;81(12):1091–7.
17. Nousiainen MT, McQueen SA, Ferguson P, Alman B, Kraemer W, Safir O, et al. Simulation for teaching orthopaedic residents in a competencybased curriculum: Do the benefits justify the increased costs? Clin Orthop Relat Res. 2016;474(4):935–44.
18. Korteling JEH, Oprins EAPBE, Kallen VL. Measurement of Effectiveness for Training Simulations. NATO RTO. 2009;SAS-095(2005):1–12.
19. James HK, Chapman AW, Pattison GTR, Griffin DR, Fisher JD. Systematic review of the current status of cadaveric simulation for surgical training. Br J Surg. 2019;106(13):1726–34.
20. Gomoll A, Pappas G, Forsythe B, Warner J. Individual skill progression on a virtual reality simulator for shoulder arthroscopy: a 3-year follow-up study. Am J Sport Med. 2008;36(6):1139–42.
21. Best Practices LLC Strategic Benchmarking Research. Professional Medical Education Excellence: Benchmarking Critical Program Trends Transforming the Medical Device and Biopharmaceutical Marketplace [Internet]. 2013. Available from: https://www.slideshare.net/bestpracticesllc/psm-286-a-professional-medical-education-budget-and-performance?next_slideshow=1
22. Ebadi A, Tighe PJ, Zhang L, Rashidi P. DisTeam: A decision support tool for surgical team selection. Artif Intell Med. 2017;76:16–26.
23. Menendez ME, Lawler SM, Shaker J, Bassoff NW, Warner JJP, Jawa A. Time-Driven Activity-Based Costing to Identify Patients Incurring High Inpatient Cost for Total Shoulder Arthroplasty. J Bone Joint Surg Am. 2018 Dec;100(23):2050–6.