The Power of AR and AI in Medical Context

Medical imaging technologies are invaluable tools in the diagnosis and treatment of patients, allowing healthcare professionals to non-invasively see what lies deep inside the body, identify any disease and/or abnormalities, and determine appropriate treatments.

The next logical step is to use rapidly maturing augmented reality technologies to enable surgeons and other medical professionals to examine a three-dimensional visualisation of the patient’s anatomy and obtain a clearer understanding of their condition, and how to treat it.

But it can also go further and may ultimately be used in the operating room to guide physicians while carrying out a surgical procedure. The idea is to enable surgeons to view a three-dimensional rendering of medical scans during an operation, overlaid on the patient like a hologram, so the surgeons do not have to keep diverting their eyes and attention to a screen image to see what they are dealing with. This should result in a much-improved surgical outcome for the patient and deliver cost savings for medical institutions.

Bringing the pieces together

Medivis, a New York medical technology company established in 2016, aims to make this vision a reality with a fusion of augmented reality and artificial intelligence that links with standard medical image management systems to make real-time advanced surgical visualisation available in the operating room.

“In the surgical world, operations are still planned using two-dimensional images taken using CT scans and MRIs, but in reality, pathology is three dimensional and very complex. So we set out with Medivis to give surgeons a more complete view of the anatomy,” says Osamah Choudhry, a neurosurgeon who is CEO and co-founder of Medivis.

“The question is, can that data – currently being viewed as two dimensional – be visualised natively in true three-dimensional format using augmented reality, and then with the use of artificial intelligence, can that be viewed more intelligently? Meaning, for example, can a tumour be auto-segmented, or can certain vascular structures be automatically isolated and displayed? This really opens up a huge possibility for the power of AR and AI within surgery.”

To date, Medivis has built AnatomyX, which uses highly detailed 3D models of the human body for teaching clinical anatomy, and SurgicalAR, which is the platform intended for use in the operating room. The company has been granted clearance by the US FDA for use during surgical planning, and not yet during actual surgery.

Key components of the SurgicalAR solution are a powerful Dell Precision workstation with an NVIDIA GPU to generate the visualisations, and an augmented reality headset so the surgeon can see hands-free.

The operating room setup is also linked to the medical institution’s data centre, typically via an Ethernet network, from where it can access the medical images generated by CT scans, X-rays and MRI scans stored in the Picture Archiving and Communications System (PACS).

One of the first medical institutions to use SurgicalAR is VA Health Care, the provider of health services for US Armed Forces veterans, which has so far deployed it in five hospitals.

The Microsoft’s HoloLens 2 augmented reality headset connects to a Dell Precision workstation that serves up the image data, and the workstation connects to the PACS systems in the data centre via the network.

The workstation used in current deployments is a Precision 3640 with 32GB of DDR4 DRAM and 1TB NVMe M.2 SSDs to deliver the necessary data throughput. Graphics processing power comes from an Nvidia RTX 6000 or RTX 5000.

SurgicalAR uses the GPU to perform complex volumetric rendering of the image slices formed from the medical scans to produce a visualisation, creating 3D textures and using different rendering formulas, according to Choudhry.

“When you do that thousands of times a second for each eye, generating 60 frames per second, you end up with a virtual 3D representation of the scans,” he says. “In order to do that processing requires a high-end NVIDIA GPU that has all the latest advances to support AI.”

Dell Technologies is also testing out configurations using a more compact Precision 3240 workstation or a Precision 7750 mobile workstation using Nvidia’s Quadro RTX 3000 GPU, to see if these systems have enough power to drive the HoloLens part of the system.

Picking the right partners

Dr. Choudhry said that Medivis chose Dell as its technology partner for SurgicalAR because the firm has a range of workstations that meet the performance requirements, but also a compact footprint for the environment they will be deployed in. Dell Technologies also collaborates closely with NVIDIA to integrate the GPU capability, as well as Microsoft with its HoloLens 2 headset technology.

Another reason is that Dell has a wealth of experience in the healthcare industry, and works with many of the providers delivering the PACS medical imaging storage and archive solutions to medical institutions. Dell is able to furnish these providers with a broad hardware portfolio of infrastructure, including PowerEdge servers, networking, and storage systems such as Dell EMC Isilon scale-out network-attached storage arrays, which have features tailored especially for PACS environments.

Although SurgicalAR has so far been cleared only for use in planning operations, Choudhry believes it will be soon be more widely adopted and used even during surgery. The reason is that it will allow for greater precision, and fewer mistakes. Avoidable complications occur in around 10 percent of all invasive operations, and these can be very costly as well as impact the patient’s quality of life.

“We think that surgery will be using augmented reality hardware as a standard of care, probably in the next decade or so. Artificial intelligence, with its ability to auto segment medical images, is able to provide data that otherwise takes a lot of manual input to process, providing that sort of data immediately. I think that is a very exciting direction in which the field of holographic surgery is headed.”

Read the full Medivis case study here.