AI-driven imaging expands possibilities in surgery

"We aim to revolutionise surgery through the power of data science," says Prof. Dr. Lena Maier-Hein, explaining the mission of her Intelligent Medical Systems Division at the German Cancer Research Center (DKFZ) in Heidelberg. Bringing together experts in computer science, physics, mathematics and statistics, the team is developing cutting-edge machine learning and artificial intelligence (AI) techniques designed to make surgical interventions safer and more effective.

The surgeon’s visual perception is particularly limited in minimally invasive surgeries. This limitation arises partly because conventional endoscope cameras replicate the human eye by capturing light only within the RGB (red, green, blue) spectrum. As a result, it is extremely challenging to distinguish tissue types and assess organ function accurately when using standard imaging methods.

In close collaboration with partners at Heidelberg University Hospital, Maier-Hein's team has developed a novel endoscopic camera system that delivers significantly more detailed insights into tissue characteristics. Unlike conventional systems, this technology captures electromagnetic radiation not only across the visible spectrum but also ultraviolet and infrared wavelengths and processes the data in real time. Depending on the number of spectral bands captured, the technique is classed as multispectral imaging (covering up to several dozen bands) or hyperspectral imaging (offering high-resolution data that often has more than 100 spectral measurements per pixel).

"The reflection and absorption of irradiated light - both of which vary according to tissue type and wavelength - provide us with valuable information about tissue composition and physiological properties," explains Dr. Alexander Seitel, Deputy Head of Division. "Each structure and molecule possesses unique optical characteristics that enable us to infer its function. For instance, spectral imaging alone can be used to evaluate oxygen saturation, which reflects blood flow quality in specific tissues. This capability makes the technology especially promising for a wide range of clinical applications. We are currently exploring these in collaboration with our partners at the university hospitals in Heidelberg and Mannheim."

To prevent excessive bleeding during many surgical procedures, blood flow to the targeted tissue or organ must be temporarily stopped. Traditionally, whether the clamping has worked or not is verified by intravenously administering indocyanine green (ICG), a fluorescent dye. Once injected, ICG rapidly binds almost entirely to blood plasma proteins, enabling blood vessels to be clearly visualised. A lack of fluorescence indicates reduced or stopped blood flow. However, if the incorrect area is clamped or the flow is not fully stopped, the ICG test can only be repeated after a washout period of approximately 30 minutes. Serious adverse reactions to ICG can also occur, but this is rare.

In contrast, an MSI (multispectral imaging) system, developed by Maier-Hein's group in 2018, enables oxygen saturation to be analysed without relying on contrast agents. This innovative system utilises a multispectral camera attached to a standard endoscope to capture the distinct absorption spectra of oxygenated and deoxygenated haemoglobin. By leveraging advanced deep learning algorithms, the system processes image data at a rate of 25 images per second, enabling real-time analysis. To address the significant variability between individuals' organs, the researchers introduced a personalised approach. Rather than relying on data from a large cohort of patients, the algorithms use a short video sequence of the affected organ, recorded at the beginning of the operation, as a reference point. This tailored method helps the system detect reduced blood flow with greater accuracy.

In collaboration with the Karlsruhe Municipal Hospital (Prof. Dr. Dogu Teber), the innovative multispectral endoscope has already been successfully used in patients undergoing partial kidney resection. ¹⁾ Compared to the traditional ICG method, the MSI system provides equally effective visualisation of tissue perfusion. However, it has three significant advantages: it is non-invasive, reusable and enables continuous real-time monitoring.

Another project focuses on improving sepsis diagnostics. Since there are currently no definitive biomarkers for this life-threatening inflammatory condition, it is often only identified when organ failure has already begun. However, recent research has revealed that microcirculatory disorders occur in the early stages of sepsis, leading to small oedema in the tissue, among other symptoms. In a recently published study, researchers captured hyperspectral images (HSI) of the palm and ring finger of all patients in the intensive care unit at Heidelberg University Hospital.²⁾ Using advanced deep learning algorithms, the researchers analysed the image data to identify early indicators of sepsis, including oxygen saturation, blood flow, haemoglobin levels and water content. Their findings demonstrated that this method provides a more reliable diagnosis of sepsis and prediction of mortality than conventional tests. Hyperspectral imaging (HSI) analysis is a fast, non-invasive, mobile and cost-effective procedure. As such, it has significant potential as a pre-screening tool for identifying individuals who require further diagnostic investigations.

Since MSI and HSI are limited to capturing data from surface structures, the team led by medical computer scientist Seitel employs a different imaging technology: photoacoustic imaging. This technique involves sending short pulses of light into the tissue, where the light is absorbed by molecules and structures. The absorption causes localised heating and expansion, generating small shock waves that are detected using ultrasound sensors. As Seitel explains: "This approach allows us to generate multispectral three-dimensional images of tissue layers several centimeters deep. However, since the light is attenuated as it travels through the tissue, it becomes more challenging to derive precise conclusions on the concentration of the detected substances." This is the reason why the clinical application of photoacoustic imaging is still in its early stages. Nevertheless, the technology shows great potential, particularly for determining parameters such as oxygen saturation.

The Intelligent Medical Systems Division is also working on developing general approaches for applying artificial intelligence (AI) in surgery, such as analysing image data. In addition, they are creating foundation models that can be adapted to address a variety of challenges. "We aim to move away from the subjective expertise that only a few experienced surgeons acquire over time, towards collective knowledge generated by AI from millions of cases with known outcomes," explains Maier-Hein, who is also Managing Director of the National Centre for Tumour Diseases (NCT), referring to the motivation behind these research projects.

Another key focus is the validation of AI-based solutions. This requires sufficient data and appropriate metrics to ensure that the results reliably address the underlying problems. A large consortium of international experts was established to determine how the quality of algorithms in medical image processing can be assessed. Using a systematic, iterative survey, the core team in Heidelberg developed a set of recommendations for designing and evaluating AI-supported processes. These guidelines serve as a valuable resource for other researchers who wish to continue this work.

Overall, these efforts are focused on improving imaging techniques to make surgery safer. In 2024, Maier-Hein was awarded the German Cancer Prize for Translational Research and the Baden-Württemberg State Research Prize in recognition of her department's innovative work.