Microsoft’s HoloLens enables collaboration among doctors globally

Orthopaedic surgeons worldwide can operate upon patients equipped with sophisticated environmental mapping hardware and an eye-tracking camera

Way back in 2017, a patient in Jaraguá do Sul, Brazil was operated for a collarbone fracture using Microsoft’s HoloLens 2 mixed-reality headset. Joining orthopaedic surgeon Dr Bruno Gobbato were Professor Thomas Gregory from Paris and Dr John Erickson, based in New Jersey. The bone was repositioned by performing shoulder arthroscopy, which involved inserting a small camera into the joint to try to determine what was causing the man’s shoulder pain.

Dr Gregory, Chief of the department of orthopaedic and trauma surgery at Avicenne Hospital in Paris and a professor of orthopaedic surgery at the University Sorbonne-Paris-Nord and Dr Erickson were linked to Dr Gobbato’s headset via the Microsoft Dynamics 365 Remote Assist app and the field of view was shared on their computer screens through Microsoft Teams. The patient was visible from the holographic images Gobbato generated from a CT scan, one showing the patient’s damaged clavicle and another replicating his healthy clavicle. The three surgeons on three continents discussed how to approach the procedure, conferring on each step and sharing their respective approaches.

The procedure was part of a project aimed at showing how HoloLens 2 can benefit surgeons and enable collaboration between doctors worldwide. Conceived by Gregory, the initiative involved orthopaedic surgeons from 13 countries on five continents.

HoloLens is operated with hand gestures and voice commands, enabling surgeons to view three-dimensional holographic images of a patient’s anatomy created from X-rays or other scans. Surgeons can move those virtual images around to see them from different angles. HoloLens can be used to access patient data during surgery, call up videos or documents to help solve problems and contact other specialists for advice.

Between November and January, 12 doctors used HoloLens to perform a surgery — ranging from a knee procedure in the UAE to a shoulder replacement in South Africa — while the others, joined by Gregory, took turns sitting in to observe and offer input. 

Dr Gregory conveys the benefits of what he sees as “the smartphone for surgeons.” According to him, many people rely on smartphones to quickly access information in their daily work, but surgeons can’t use phones or computers in a sterile operating room setting. Procedures are planned and there is little access to patient data or other resources once surgery begins.

Dr Ashish Babhulkar never used a HoloLens before signing up for Gregory’s project. Dr Babhulkar, head of the shoulder and sports injuries department at Deenanath Mangeshkar Hospital & Research Center in Pune, India, did two reverse shoulder replacement surgeries — in which the ball and socket position of the shoulder joint is reversed — in collaboration with Gregory and an orthopaedic surgeon from the U.K.

With input from the two doctors, Babhulkar superimposed a holographic image of the socket implant on top of the patient before inserting it to get a perfect fit. Though Babhulkar didn’t need the manual for the procedure, he saw how useful HoloLens could be as a training tool.

In another case study, Dr John Sledge, an orthopaedic surgeon in Lafayette, Louisiana, performed one of the procedures that was part of the HoloLens project, a 10-hour spinal fusion surgery. As Gregory and two surgeons from the UK and the UAE looked on, Sledge pulled up X-rays and scans of the patient’s lower back to locate the pieces of hardware that needed to be removed and determine how to best position his instruments to access them. He had also loaded images taken shortly after the patient’s accident so he could explain the preop history to the other surgeons.

Says Dr Sledge, “With the HoloLens, I can pull up the images I want and make them bigger as needed. I’ve got all the images I need right in front of me, in whatever size and clarity that I need them to be.” Instead of printing out a 3D replica of a bone and having just one shot at practising an operation on it, Sledge says, he can test his approach on a hologram as many times as needed. For training, he can generate three-dimensional interactive models of any surgical procedure or even create simulated complications on a hologram — say, a fracture or unexpected bleeding — for doctors in training to solve.

The HoloLens and its precursor, the Kinect line of motion-sensing devices, were created by Microsoft Technical Fellow Alex Kipman, a visionary engineer who saw a need for a device that merged the physical and digital worlds in a meaningful, three-dimensional way.

Kipman sought to develop a device that could understand humans and their environments and transform the digital world from one of flat rectangles of glass — computer screens, phones — to a universe of holograms that can interact with or simulate the real world.

The first HoloLens was released in 2016, followed by HoloLens 2 in November 2019. The device is equipped with sophisticated environmental mapping hardware and an eye-tracking camera that enable it to understand the space it’s being used in and what the user is focussed on. The Microsoft Dynamics 365 Remote Assist application was designed to allow HoloLens to connect with Teams so users can collaborate with people worldwide.

The HoloLens 2 is now being used for a range of applications in health care, from virtual hospital rounds for medical students to treat patients with COVID-19, and a ‘HoloPatient’ used for simulations of various medical conditions to train doctors and standardise treatment.

The use of HoloLens in surgery is new, propelled by Gregory and other pioneering physicians, says Elena Bonfiglioli, Microsoft’s Managing Director and Regional Lead for healthcare and life sciences in Europe, the Middle East and Africa.

The power of HoloLens, Bonfiglioli says, is its ability to superimpose objects and data on the real world in a trusted digital environment and enable a natural means of learning.

“The future of medicine is empowered by holographic computing because holographic computing gives the new dimension of visual,” she says. “We have had the dimension of collaboration for a long time. We’ve had the dimension of data, powered by artificial intelligence. But we didn’t have something very typical to the human, which is visual enhancement. That’s why it’s so powerful. It’s a technology that in itself is a change agent, empowering those who can see the future.”

Microsoft partnered with Gregory in January 2020 and others to launch Institut Moveo, a research and teaching centre dedicated to artificial intelligence in surgery. The centre, located at University Sorbonne-Paris-Nord, brings together computer scientists, mathematicians and surgeons and provides training on using HoloLens for surgery.

There are plans to collect HoloLens data from surgeries and using it to develop algorithms that could improve surgical procedures or even identify, based on eye movements, when a surgeon is encountering difficulty.