doi: 10.56294/gr202327

REVIEW

Augmented reality as an accessory technology in surgery

Realidad aumentada como tecnología accesoria en cirugía

José Alejandro Rodríguez-Pérez¹ *

¹Hospital de Palamós. Palamós, Spain.

Cite as: Rodríguez-Pérez JA. Augmented reality as an accessory technology in surgery. Gamification and Augmented Reality. 2023;1:27. https://doi.org/10.56294/gr202327

Submitted: 03-09-2023                   Revised: 31-10-2023                   Accepted: 25-12-2023                 Published: 26-12-2023

Editor: Adrián Alejandro Vitón-Castillo

ABSTRACT

Introduction: Augmented reality technology uses 3D reconstruction, visualization, registration and tracking techniques to create images from MRI data. It has three basic characteristics: combination of real and virtual worlds, real-time interaction and precise three-dimensional recording of virtual and real objects. The objective was to characterize AR as an accessory technology in surgery.

Method: a total of 22 articles in Spanish and English were reviewed, from Pubmed, Scielo and Scopus; using as keywords: augmented reality, surgery, virtual reality, being more than 50 % of the last five years.

Result: Although primarily used for training, this technique can be used for planning and navigation in the operating room; since the precision and complexity of the three-dimensional reconstructed images are crucial to providing the correct data in surgery. Its main advantage is the integration of sensations and real-time interaction of the doctor while its limitations include technological ones and those associated with the way of use by the staff. Medical education has benefited from the popularization of virtual reality as it reduces ethical conflicts and promotes self-learning.

Conclusions: augmented reality systems combine the preoperative model with the intraoperative scenario to project images in real time, ensuring better results in terms of time, error rate and precision. Despite this, its use is not globalized and the available bibliography on its validity is insufficient.

Keywords: Augmented Reality; Virtual Reality; Surgery.

RESUMEN

Introducción: la tecnología de realidad aumentada utiliza técnicas de reconstrucción, visualización, registro y seguimiento 3D para crear imágenes a partir de datos de resonancia magnética. Posee tres características básicas: combinación de mundos reales y virtuales, interacción en tiempo real y grabación tridimensional precisa de objetos virtuales y reales. El objetivo fue caracterizar la RA como tecnología accesoria en la cirugía.

Método: se revisaron un total de 21 artículos en español e inglés, provenientes de Pubmed, Scielo y Scopus; utilizando como palabras clave: realidad aumentada, cirugía, realidad virtual, siendo más del 50 % de los últimos cinco años.

Resultado: si bien se utiliza principalmente para la formación, esta técnica se puede emplear para la planificación y navegación en el quirófano; pues la precisión y complejidad de las imágenes reconstruidas en tres dimensiones son cruciales para proporcionar los datos correctos en cirugía. Su principal ventaja constituye la integración de sensaciones e interacción en tiempo real del médico mientras que entre sus limitaciones se citan las tecnológicas y aquellas asociadas al modo de empleo por parte del personal. La educación médica se ha visto beneficiada tras la popularización de la realidad virtual pues disminuye conflictos éticos y promueve el autoaprendizaje.

Conclusiones: los sistemas de realidad aumentada combinan el modelo preoperatorio con el escenario intraoperatorio para proyectar imágenes en tiempo real, asegurando mejores resultados en cuanto a tiempo,  tasa de error y precisión. A pesar de esto su empleo no es globalizado y la bibliografía disponible sobre su validez es insuficiente.

Palabras clave: Realidad Aumentada; Realidad Virtual; Cirugía.

INTRODUCTION

The first experiments with medical imaging date back to 1895, when Röntgen discovered the existence of X-rays. The advent of visualization systems such as ultrasound, magnetic resonance imaging, and computed tomography made the use of two-dimensional images in diagnosing and treating surgical pathologies possible.⁽¹⁾

Augmented reality (AR), or three-dimensional (3D) technology, was first introduced for commercial purposes in the gaming and entertainment industry; it was subsequently applied to education, communications, and medicine. The U.S. Air Force conceived the first AR system in early 1992 to improve human performance during surgery.⁽²⁾

Carmigniani defines integrative as an interactive experience that uses the existing real-world environment on which computer-generated virtual information is superimposed to enhance the user experience. The system incorporates three essential features: a combination of natural and virtual worlds, real-time interaction, and accurate 3D recording of virtual and natural objects. The hardware components for AR are a processor, a display, sensors, and input devices.^(2,3)

Applied to medicine, it is known as a technology that works similarly to virtual reality, in which 3D reconstruction, visualization, registration, and tracking techniques are used to create images from MRI data. It thus allows the preoperative model to be combined with the intraoperative scenario, providing physicians with real-time surgical information and guidance.^(1,4) Its spread throughout the different branches of healthcare has been aimed mainly at supporting the organization, performance, and training in surgical procedures.⁽²⁾ The beauty of augmented reality is its potential to show simplified and easily recognizable anatomy during surgery.⁽¹⁾

The use of AR has become popular in the healthcare sector in the last decades, peculiarly in surgical specialties, one of the first being neurosurgery, five orthopedics, and spine, six laparoscopies, one hepatobiliary, seven maxillofacial, eight gynecology, nine surgical oncology, two plastic surgery, ten to cite a few examples. Despite the above, it is impossible to stratify its use since the degree of acceptance and application it has acquired in different countries is still being determined.

Given the above, the present research was carried out to characterize AR as an accessory technology in surgery.

METHODS

A bibliography review was carried out in which articles in Spanish and English from the Scopus, Pubmed, and Scielo databases, published in different contexts, were analyzed using the following keywords: augmented reality, surgery, and virtual reality. The information of interest was extracted from the 21 selected articles to synthesize and order for the preparation of this research.

DEVELOPMENT

Many areas of medical imaging, from simulation to training and therapy, benefit significantly from using mixed, augmented, and virtual reality technologies. While primarily used for training, AR can be used equally well for planning and navigation in the operating room.^(1,11)

Modern devices such as smartphones and tablets are suitable AR platforms. Various technologies are used to render AR, including optical projection systems, monitors, handheld devices, and display systems. AR displays can be rendered on devices that resemble glasses; versions include goggles that employ cameras to intercept the real-world view and display the augmented view through eyepieces and devices where the AR image is projected.⁽²⁾ 

Khor et al.⁽¹²⁾ assert that interest in AR followed the announcement of Google Glass, a technology that gained publicity until January 2015, when production ceased. This accessory benefited from a lightweight wearable overlay display screen and a high-resolution video camera, with features similar to those seen in a smartphone, such as wireless and cloud accessibility. The open-source development platform for the device enabled a creative surgical medical application, and similar devices specific to surgical enhancement were subsequently developed. 

The physical environment is a crucial difference between augmented reality and other navigation systems that use conventional visualization methods, which project the required information into the surgeon's field of view. It is considered a natural progression from well-established methods that mitigate errors associated with shifting attention by directly projecting the navigational guidance onto the surgical field; furthermore, projecting the information directly into the field of view mitigates errors associated with shifting attention.⁽⁹⁾

Currently, AR applications are limited by the essential requirement for preoperative 3D reconstructions of medical images. It is possible to create these reconstructions using commercial or proprietary software, enabling virtual exploration of target areas, planning a practical surgical approach, and improved orientation and navigation in the operative field. The quality of a reconstruction depends on the quality of the input data and the accuracy of the reconstruction system.⁽¹³⁾

On the other hand, the accuracy and complexity of the 3D reconstructed images are crucial to providing the correct data to the surgical team. Available studies show that optical systems have an accuracy of 5 mm, which is considered sufficient for clinical application, but the author considers it necessary to point out that the required accuracy differs significantly according to the procedure in question.⁽¹³⁾

Vávra et al.⁽¹³⁾ clarify that AR systems are most useful during surgery of organs with little movement and deformation, such as the brain and pancreas. At the same time, the intestines become significantly more complicated. 

Gomez Amarillo et al.⁽⁵⁾ indicate that in recent years, the usefulness of AR has been demonstrated for resecting brain tumors, especially of the skull base, where some cases may be more favored, such as repairs or those with anatomical variations. 

Neurosurgical processes have successfully employed AR systems, allowing a higher rate of accurate lesion localization and shorter operative time than traditional methods. In addition, neurosurgeons benefit from the precise localization of convolutions, blood vessels, and neural tracts and the ability to plan the operation corridor for tumor removal, epilepsy surgery, or neurovascular surgery.⁽¹³⁾

Incekara et al.^(9,14), in their clinical feasibility study of a portable mixed reality device in neurosurgery, evaluated the use of AR in the repair of the surgical plan of 25 patients and stated that there were benefits in terms of maintaining patient focus and attention, as well as a better understanding of the tumor-brain/skull relationship due to the direct three-dimensional holographic representation. 

In the particular case of orthopedics, Verhey et al. six state that the increasing costs of resident training combined with decreasing work hours and ethical concerns regarding patient care demand that trainees acquire skills outside the operating room. Cannon⁽¹⁵⁾, in his study on the use of the MIST-VR laparoscopic arthroplasty simulator, concluded that residents trained with AR performed surgery faster, while those who did not have access to AR were more likely to cause injury. 

Maxillofacial specialists such as Ayoub and Pulijala⁽⁸⁾ believe that AR has contributed to procedural planning and recognize the importance of the technology but rate the available literature on the impact on education and practice in the specialty as insufficient. 

Ghaednia et al.⁽¹⁶⁾ state that the prevalence and interest in minimally invasive spine surgery and the need for high-quality, easy-to-use navigation systems have increased. He considers that AR cannot be postponed because it displaces previous systems by reducing operating room time, exposing patients and the medical team to excessive radiation, and integrating advanced image quality and better results. 

The 2D or 3D projection of the different pelvic, breast, or liver neoplasms on the actual image of the lesion facilitates the dissection of the anatomical elements and the removal of the neoplasm in good conditions; in the particular case of the breast, AR images facilitate the discovery of the sentinel lymph node.⁽¹³⁾ 

Cheng et al.⁽¹⁰⁾, in their review of the literature on AR and plastic surgery, found that in several subspecialties, AR has demonstrated practicality and success in surgical planning and education. However, they rate its intraoperative use as minimal due to device-specific limitations or problems with physician usability. 

Importance 

The importance of AR lies in the way in which the components of the digital world overlap with a person's perception of the natural world, not as a simple visualization of data but by integrating sensations that are perceived as natural parts of the surrounding world.⁽²⁾

Access to real-time data is becoming increasingly important, as it can be used more quickly and efficiently in diagnosis and treatment. Real-time access to 2D images or 3D reconstructions during surgery is crucial in revealing images of tumors, their position at the parenchymal level of an organ, or their relationship to underlying anatomical elements such as blood vessels, nerves, and ducts. This ensures better control during a tumor resection and decreases the loss of healthy tissue from the procedure.⁽²⁾

A recent meta-analysis of randomized controlled trials showed reduced operative time, error rate, and accuracy when virtual reality training was used for new trainees with no previous experience or when supplemented with standard laparoscopic training.^(12,17)

The modern approach to surgical strategy in selected cases using augmented reality has made combining authentic intraoperative pelvic and abdominal images with those of 3D reconstruction a step forward in recognizing local anatomy, providing better orientation in the operative field, and increasing the chances of tumor resection. The ability to activate a device by voice commands is of great significance in the field of surgery, as it allows the device to be controlled without the need for manual assistance or compromising aseptic protocols.⁽²⁾ 

Verhey⁽⁶⁾, believes that the importance of AR in surgery lies in its availability for use without the need for an expert trainer to guide and supervise trainees. 

Virtual vascular endoscopy can generate endoluminal views of blood vessels that may be useful in preoperative planning, especially for patients with significant atherosclerosis or aberrant anatomy. Virtual reality devices would allow 360-degree viewing and the ability to view multiple images simultaneously.⁽¹²⁾

Limitations

The occurrence of blindness due to lack of attention, possibilities of distraction due to the amount of information the surgeon must handle, the weight of the accessories, and adverse effects such as nausea, vomiting, vertigo, and headache due to visual stimuli.^(13,18)

The authors point out among the main impediments of the technology precisely its high cost and, therefore, difficult access in third world countries. The digital support and the necessary computer field make its use impossible for a significant part of the population. 

Virtual environments 

Dassault Systèmes uses computational models in connection with anatomy and virtual reality, allowing clinical scientists to immerse themselves in the patient's anatomy to solve pathological problems. Proximie is another platform that aims to introduce AR technology to surgeons in the developing world, enabling them to visualize recorded or real-time operations performed by experts in other parts of the world and thus disseminate broad surgical expertise. Sony's AR headset is likely to be one of the biggest platforms in the virtual world due to its low price.⁽¹²⁾ 

The ProMIS is a simulator that trains laparoscopic procedures. It contains an instrument tracking system, which captures the movement of the instrument while providing realistic haptic feedback. Time, path length, and smoothness of movement can be objectively recorded and used as outcome parameters.⁽¹⁹⁾

The CAE VIMEDIXTM ultrasound simulator uses a transducer that provides position and orientation data to reconstruct images relative to a mannequin. The simulator has been used to train transthoracic echocardiography and transesophageal echocardiography.^(18,19)

The author considers that the development of AR-based virtual environments is not limiting, so the existence of such environments is distinguished both in the educational area and in medical practice. 

Academic application 

Critical situation simulation creates a promising opportunity for the education of medical professionals in a safe environment. Relevant scenarios can be practiced in environments where exploration and problem-solving are safe. AR systems can improve learning outcomes of different training procedures, improving patient safety and reducing costs and morbidity. Barsom⁽¹⁹⁾ concludes from his systematic review that despite the interest generated by AR blended learning applications, there needs to be more literature to demonstrate more excellent utility over traditional methods.

AR technology is being incorporated into simulation-based training. This technology will likely surpass the fidelity of current distance learning packages and two-dimensional videos by creating a near-real experience from the operator's point of view.⁽¹²⁾

Traditional anatomy teaching typically involves using an anatomical atlas and time spent in the dissection room. Augmented reality and virtual reality provide a better appreciation of virtual or real space structures to ease the transition from the learning environment to the clinical setting. AR can supplement anatomy learning by overlaying radiological images on a body and creating a direct view of spatial anatomy for the learner.⁽¹²⁾

It is recognized that medical students and residents must acquire skills and a broad range of knowledge over time to become competent practitioners. Anatomy, in particular, is one of the cornerstones of medical education. With a proper understanding of it, regardless of the area of healthcare, practitioners can perform tests efficiently. Although lessons taught on cadavers are sometimes the standard teaching goal, there are significant financial constraints and ethical conflicts regarding their use.⁽²⁰⁾

The application of AR to other areas, such as post-surgical intensive care, shows a number of advantages. This is noted by Bruno et al., revealing the popularity of these systems in pain management, stress reduction, support for abrupt lifestyle modifications, and rehabilitation, to name just a few examples.⁽²¹⁾

The authors recognize the need for systematic studies to assess the applicability and performance of AR in surgery in order to contrast its usefulness with other technological modalities despite the undeniable nature of its advantages.

CONCLUSIONS

AR systems use three-dimensional technology to project reconstructed images over the surgical field by combining the preoperative model with the intraoperative scenario, ensuring better time, error rate, and accuracy. Despite this, its use is not globalized, and the available literature on its validity needs to be revised.

REFERENCES

1. Malhotra S, Halabi O, Prasad Dakua S, Padhan J, Paul S, Palliyali W. Augmented Reality in Surgical Navigation: A Review of Evaluation and Validation Metrics. Appl Sci 2023;13:16–29.

2. Mihail Prunoiu V, Popa D, Stelian Æerbulea M, Brãtucu E, Simion L, Nicolae  Brãtucu M. Realidad aumentada en oncología quirúrgica. Una revisión de literatura. Cirugía 2022;117:517–25.

3. Carmigniani J, Furth B, Anisetti M, Ceravolo P, Damiani E, Ivkovic M. Augmented reality technologies. systems and applications. Multimedia Tools and Applications 2011;51:341–77.

4. Wang J, Suenaga H, Hoshi K, Yang L, Kobayashi E, Sakuma I, et al. Augmented reality navigation with automatic marker-free image regstration using 3D image overlay for dental surgery. Biomedical Engineering 2014;61:1295–304.

5. Gómez Amarillo D, Ordoñez Rubiano E, Ramírez Sanabria A, Figueredo L, Vargas Osorio MP, Ramón JF, et al. Augmented reality for intracranial meningioma resection: a mini-review. Front Neurol 2023;14:1269014.

6. Verhey J, Haglin J, Verhey E, Hartigan D. Virtual, augmented, and mixed reality applications in Orthopedic surgery. Int J Med Robot 2020;16:e2067.

7. Gavriilidis P, Bjorn E, Pelanis E, Hidalgo E, de Angelis N, Memeo R, et al. Navigated liver surgery: state of the art and future perspectives. Hepato Pancreat Diseases Int 2022;21.

8. Ayoub A, Pulijala Y. The application of virtual reality and augmented reality in Oral & Maxillofacial Surgery. Ayoub and Pulijala BMC Oral Health 2019;19:238.

9. Dennler C, Bauer D, Gita Scheibler A, Spirig J, Gotschi T, Furnstahl P, et al. Augmented reality in the operating room: a clinical feasibility study. BMC Musculoskeletal Disorders 2021;22:451.

10. Cheng L, Matinrazm S, Moroni E, Kass N. Augmented Reality in Plastic and Reconstructive Surgery: What It Is, How Far It’s Come, and the Limitations Impacting Further Adoption. PRS Global Open 2018;11.

11. Cartucho J, Shapira D, Ashrafian H, Giannarou S. Multimodal mixed reality visualisation for intraoperative surgical guidance. Int J Comput Assist Radiol Surg 2020;15:819–26.

12. Sim Khor W, Baker B, Amin K, Chan A, Patel K, Wong J. Augmented and virtual reality in surgery—the digital surgical environment: applications, limitations and legal pitfalls. Ann Transl Med 2016;4.

13. Vávra P, Roman J, Zonca P, Ihnát P, Nemec N, Kumar J, et al. Recent Development of Augmented Reality in Surgery: A Review. Journal of Healthcare Engineering 2019;2017:e4574.

14. Incekara F, Smits M, Driven C, Vincent A. Clinical feasibility of a wearable mixed-reality device in neurosurgery. World Nneurosurg 2018;118:e422–7.

15. Rahm S, Wieser K, Wicki I, Holenstein L, Fucentese S, Gerber C. Performance of medical students on a virtual reality simulator for knee artgoscopy: an analysis of learning curves and predictors of performance. BMC Surgery 2016;16:2–8.

16. Ghaedinia H, Fourman M, Lans A, Detels K, Dijkstra H, Lloyd S, et al. Augmented and virtual reality in spine surgery, current applications and future potentials. Spine J 2021;21:1–9.

17. Nagendran M, Selvan Gurusamy K, Aggarwal R, Loizidou M, Davidson B. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Library 2013.

18. Platts GD, Humphries J, Burstow DJ, Anderson B, Forshaw T, Scalia V reality training for surgical trainees in laparoscopic surgery. The use of computerised simulators for training of transthoracic and transoesophageal echocardiographic training. Heart Lung Circ 2012;21:267–74.

19. Barson E, Graafland M, Schijven M. Systematic review on the effectiveness of augmented reality applications in medical training. Surg Endosc 2016;30:4174–83.

20. Mihail Prunoiu V, Popa D, Stelian Æerbulea M, Brãtucu E, Simion L, Nicolae  Brãtucu M. Realidad aumentada en oncología quirúrgica. Una revisión de literatura. Cirugía 2022;117:517–25.

21. Romano Bruno R, Wolff G, Wernly B, Masyuk M, Piayda K, Leaver S, et al. Virtual and augmented reality in critical care medicine: the patient’s, clinician’s, and researcher’s perspective 2022;26:326–37.

FINANCING

No external financing.

CONFLICT OF INTEREST

The authors declare that there is no conflict of interest.

CONTRIBUTION OF THE AUTHORS

Conceptualization: José Alejandro Rodríguez-Pérez.

Research: José Alejandro Rodríguez-Pérez.

Methodology: José Alejandro Rodríguez-Pérez.

Writing - original draft: José Alejandro Rodríguez-Pérez.

Writing - proofreading and editing: José Alejandro Rodríguez-Pérez.