Pharmaceutical organisations are discovering multiple applications for mixed reality solutions. They can remotely share their science to accelerate discovery and development, support clinical trials, and train workers, all while reducing procedural non-compliance and increasing yields.
Medical technology firms are using it to do audits and inspections remotely, make staff safer and more productive, and drive greater remote sales and service in times when travel has been restricted.
The technology has even trickled down to healthcare providers, who have used it treat patients during the pandemic, and train staff while coping with an increasing number of affected patients.
“Mixed reality puts the human at the centre in a new way for technology,” says Rody Senner, North America mixed reality sales lead at Microsoft. “We immerse ourselves in it by stepping through the screen where our digital world is now spatially placed around us, while we remain present in our physical world. This blending of the two is at the heart of mixed reality. And this immersion is now even more apparent, with intuitive ways to use voice, eye and hand tracking while being ‘heads up and hands free’ in our work, giving us the ability to empower those on the front line, who are critical to the healthcare sector.”
According to Senner, one of the major benefits of using mixed reality technologies in healthcare is being able to experience spatial concepts in 3D.
“Most technologies today are based in 2D, so there is a cognitive load for translating natively 3D concepts – such as the human body – from a 2D CT scan back into 3D,” she explains. “It introduces an opportunity for error. In addition, the 3D basis of mixed reality creates significant opportunities by overlaying relevant data onto real-world images. It ushers in a new wave of computing, one that has the potential to be the biggest wave we have seen yet.”
Research suggests that not only is the healthcare market aware of these new technologies and the benefits they deliver, it is adopting them at accelerating rates. According to the Global Mixed Reality in Healthcare Market report for 2020-2026, the global healthcare market is expected to grow by 10 times, and Grand View Research found that it is projected to be the top industry for growth in augmented reality, in its Augmented Reality Market Size, Share & Trends Analysis Report from February 2021.
Despite this, there are barriers to adoption.
“While costs are of course an important factor in healthcare, mixed reality tools like HoloLens can be deployed in very cost-efficient ways,” says Dr David Rhew, chief medical officer at Microsoft. “Not all users need to wear a headset, for example. In remote scenarios, participants can participate from their personal computers.”
Patient confidentiality and data privacy are also key considerations for healthcare organisations considering implementing mixed reality. “We have done our utmost to ensure the security of HoloLens,” says Rhew. “We have taken the device through a 10-point security checklist and it has been cleared as a zero-trust device. HoloLens also uses iris recognition for secure sign on, one of the most secure biometric frameworks that exists.”
Among those who have overcome these barriers, adoption of HoloLens and other mixed reality tools has been widespread. From design, manufacturing, sales and service through to the surgeon and specialist, there is significant opportunity for digital transformation with mixed reality, particularly in facilitating remote communication, education and observation.
“Whether you are working with a patient, in a pharmacy or on a manufacturing floor, the ability to get the assistance of an expert in real time has been a matter of business continuity and human life for many organisations over this past year,” says Senner. “Microsoft HoloLens enables frontline workers to connect with experts anywhere in the world in under a minute and give that expert the opportunity to see this person’s environment as though they were there, reach into the environment to digitally annotate or overlay holographic arrows for instruction, pull in other data for reference and more. Troubleshooting a problem in real time and eliminating travel also creates massive cost savings.”
The Imperial College Healthcare NHS Trust is one of the many healthcare providers that implemented HoloLens in this way.
After seeing 29 medical staff members working in close proximity during the Covid-19 pandemic, Dr James Kincross, a consultant surgeon at Imperial College, realised that this established way of working would not be sustainable over the following months. He designed a solution based on HoloLens 2 and Dynamics 365 Remote Assist that would enable medical staff to carry out ward rounds virtually.
“I’ve used HoloLens before in surgery and we quickly realised it had a unique role to play because we could take advantage of its hands-free telemedicine capabilities,” said Kincross. “It solved a major problem for us during a crisis, by allowing us to keep treating very ill patients while limiting our exposure to a deadly virus. Not only that, it reduced our personal protective equipment consumption and significantly improved the efficiency of our ward rounds.”
On top of facilitating remote communication and telemedicine and accessing the benefits they offer, mixed reality tools are also delivering opportunities for immersive training and workflows.
“Training is so fundamental to employee empowerment and performance,” says Senner. “With holographic guided instructions people can train faster and retain information more effectively than with 2D manuals. Healthcare organisations can also use holographic equipment so that users can train anywhere without taking physical equipment out of production and causing unnecessary downtime.”
For example, medicine and medical device manufacturer Novartis uses Taqtile’s Manifest augmented reality platform to develop training and administer certification for anaesthesia machines. Trainees are guided through instruction by job templates which feature 3D ink or shapes overlaid onto equipment to highlight an area of interest or concern. Manifest allows evidence to be captured during the execution of a task for auditing and certification.
The design, manufacturing and maintenance of medical equipment can also be improved with mixed reality.
“Imagine converting computer-aided design files to 3D digital twins to see a holographic version of a medical technology device to evaluate different variants of the design in minutes, collaborating in person and remotely,” says Senner.
This is precisely what Stryker did. The medical technology firm integrated HoloLens into the design process of its operating rooms.
“Different surgical disciplines use shared operating rooms, but these specialities have widely different needs in terms of configuration and set-up,” says Senner. “Equipment placement is critical as it effects ergonomics, efficiency and task load, all of which have the potential to burden staff and slow procedures.”
Using HoloLens, Stryker enabled a team of surgeons to collaborate and work with holograms in the physical operating room. This reduced the design time and errors, and shortened sales cycles of its operating rooms by enabling customers to interact with the product in 3D and understand its value.
But Microsoft is not resting on its laurels with the success of HoloLens and HoloLens 2. “We are investing heavily in cloud services to make it easier to develop collaborative and immersive mixed reality experiences,” says Senner. “We want to help our partners continue building industry-leading solutions and make it as easy possible for customers to use mixed reality to improve daily operations and patient experiences.”
Partner perspectives
We asked a selection of Microsoft partners how they are building on the HoloLens platform to help healthcare providers develop innovative solutions that empower employees to deliver better patient experiences. Below are extracts from their responses, which you can read in full from page 176 of the digital edition of the Autumn 2021 issue of Technology Record.
Sirko Pelzl, CEO and founder of apoQlar, said: “VSI HoloMedicine is a medically certified cloud-based platform that leverages the Microsoft HoloLens 2 hardware to transform medical images, clinical workflows and medical education into an interactive 3D mixed reality environment.”
Dr Till Bay, CEO of Incremed, said: “Incremed uses mixed reality and machine learning in combination with Microsoft HoloLens 2 and Azure Kinect to revolutionise diagnostics and therapeutic interventions across various disciplines of medicine.”
Phil Moore, digital innovation lead for the public sector at Insight, said: “The UK’s NHS trusts needed to rapidly implement Microsoft HoloLens but were struggling with both technical deployment and end user training. Insight developed the Fast Start service, which takes an organisation from unboxing HoloLens through to their first remote assist call, all within just one day.”
This article was originally published in the Autumn 2021 issue of Technology Record. To get future issues delivered directly to your inbox, sign up for a free subscription.