Jump to content
Intelligent Diagnostics

AI for added value in digital melanoma diagnostics

Intelligent Diagnostics is an interdisciplinary project that brings together the latest technologies and research institutes to better support doctors in diagnosing skin cancer through innovative imaging and artificial intelligence.

Is it a birthmark or perhaps a melanoma? The trained dermatologist's eye sees a lot, of course, but not everything. Studies show that even with dermatoscopes, i.e. simple mobile reflected-light microscopes, diagnostic sensitivity is currently only around 85 percent. Artificial intelligence (AI) is opening up new opportunities to offer doctors even better support. Five Baden-Württemberg research institutions have combined their expertise to jointly develop a novel diagnostic system based on AI-supported imaging. These institutions are: the FZI Research Centre for Information Technology, the ILM Institute for Laser Technologies in Medicine and Measurement Technology at the University of Ulm, the Hahn-Schickard Institutes in Villingen-Schwenningen and Stuttgart, and the NMI Natural and Medical Sciences Institute at the University of Tübingen. All five belong to the innBW research alliance (Innovationsallianz Baden-Württemberg) and have been networking for a long time.

Close-up of an arm underside with pigment spots, on which - as part of a multispectral image - the bluish, barcode-like projection pattern for one wavelength can be seen.
The section of a multispectral image shows the projection pattern for one wavelength. © FZI Research Centre for Information Technology

The joint concept was so convincing that it received funding from the Baden-Württemberg government from 2019 to 2021. The results of the first funding period were promising enough for the project partners to receive follow-up funding of two million euros for 18 months. The researchers want to use this money to further advance their project towards application. Essentially, their approach is to use sophisticated measurements to generate image-based data sets with additional information in order to train artificial intelligence models in such a way that they can distinguish benign from malignant skin changes. This approach is partly based on the knowledge that changes in skin tissue cause light directed at it to scatter and be absorbed differently and partly on structured irradiation and different penetration depths of varying wavelengths – even beyond visible light – to create a profile of skin changes and acquire data from deeper skin layers.

Creation of 3D topography of skin changes

Colour photo of Christoph Becker
Christoph Becker is a business informatics specialist who has worked at the FZI since December 2009. He heads the Intelligent Diagnostics project at the FZI. © FZI Research Centre for Information Technology

"The approach pursued in the project is based on multispectral imaging under structured irradiation. Although we work with cameras, the data sets generated are not comparable with conventional RGB images," says Christoph Becker, who heads the project for the FZI and coordinates the joint project. The idea behind this is that each wavelength investigated is projected onto the skin area to be examined with different fringe patterns. The reflections of the irradiation patterns are then recorded by a camera system. As the rays penetrate to different depths, it is possible to obtain information from skin layers at different depths. "The measuring system uses nine different wavelengths up to the near-infrared range and thus penetrates several millimetres deep into the skin. It can generate more than 40 images per picture," Becker specifies. With the help of structured irradiation, the image data can also be used to create a three-dimensional topography of the skin change under examination.

The optical measurement system was developed by the ILM and the required optics were improved in close cooperation with the Hahn-Schickard Institute in Stuttgart. At the moment, the project team is working with two cameras in order to eliminate interfering artefacts, such as those caused by surface scattering. In the second phase of the project, the process is expected to become more robust, which would also make it easier to build a compact device. "We may be able to do without the second camera if the effect of surface scattering turns out to be irrelevant. That is one of the issues we want to clarify in the second funding phase," says Becker.

Spreading tasks widely

The photo shows the demonstrator of the diagnostic system. On the left is the measuring system with tripod and camera, on the right computer and screen for operation (right).
A demonstrator of the diagnostic system was set up in the FZI’s House of Living Labs. It consisted of a measuring system (left) and a local computing unit for operation (right). © FZI Research Centre for Information Technology

An AI model is fed with images that have already been generated - more will be added in the next phase. This model then undertakes hundreds of learning cycles to determine which image data correspond to a benign or malignant tumour. As early as October 2020, the partners started training AI models with their data, and the accuracy is getting better and better. Results are constantly being verified with the help of laboratory analyses. "However, there is not yet sufficient data to conclusively evaluate the quality of the approach, so we are continuing to collect training data," says Becker, adding: "It is already quite challenging to collect training data, because very comprehensive data sets are needed for the neural network." This is where the expertise of the Hahn-Schickard Institute in Villingen-Schwenningen comes into play: the project partners there focus on aspects of distributed learning. "They work with measurement and simulated data," says Becker.

The ILM initially generated synthetic data, i.e. data based only on simulations. They subsequently worked with the Department of Dermatology at Tübingen University Hospital to carry out measurements on patients in order to obtain data from practical application. The goal now is to collect data at more hospitals, and possibly also doctors’ practices, and to have an AI model learn with the data on site at each of the hospitals and practices involved. The partners will then share their learned AI models, or merge them at a suitable point as well as providing feedback to the hospitals involved on what they have learned. "How exactly the communication will take place and how it can be automated constitute further development goals," says Becker.

Learning success of AI also depends on the quality of the basic data

Becker and his team specialise in business informatics and have an integrating function in the project: the data and AI management platform was set up at the FZI so the different systems used by the various partners work together and can be adapted when changes occur. The FZI's tasks also include the development of suitable visualisation for doctors.

The NMI Reutlingen has taken on another special task: its researchers are analysing which protein and mRNA-based molecular biomarkers enable underlying melanomas to be classified more precisely, for example, are able to indicate an aggressive disease course or a recurrence. The results are correlated with the image data and thus also find their way into the AI learning models.

The next phase of the project will also focus on the very practical aspects of manufacturing the devices. It is hoped to be able to manufacture the complex optics cheaply and flexibly. In addition, the optics need to be miniaturised, for example by using suitable micro-optical components. The ILM and the Hahn-Schickard-Institute in Stuttgart are working closely together on this. "The design and prototypes of the optics come from the ILM and are produced using two-photon lithography. The Hahn-Schickard researchers in Stuttgart are developing, among other things, a process to reproduce the optical prototypes easily and cost-effectively using injection moulding technology," Becker explains. All in all, he and his colleagues still have a long way to go before the system is ready for application: "At the moment, we are still fine-tuning the prerequisites for commercial product development. Once this is in place, we will be looking for suitable partners for the commercialisation of the product."

Die Bildschirmansicht zeigt mit mehreren Kacheln die Auswertung für den Arzt. Oben rechts ist das Ergebnis der KI-basierten Diagnose angegeben: in diesem Fall „Kein Melanom". Die Kachel darunter zeigt in einer Heatmap mit rotem Hintergrund, worauf die KI bei ihrer Klassifikation fokussiert hat (gelber Bildbereich).  Links daneben in der Kachel „Diagnoseergebnis“ ist ein Foto der Läsion und ihrer Umgebung zu sehen. Die untere Kachel „Punktwolke" zeigt ein 3D-Modell der Läsion.
The image on the left-hand side is the user interface for doctors that shows the AI-based diagnosis on the top right (melanoma). The image on the right-hand side shows the result of the AI-based diagnosis (no melanoma). The image below the AI diagnosis result is a heat map showing what the AI focused on when making a classification (yellow image area). This allows the doctor to check whether the AI is focusing on the correct lesion. On the left, in the ‘diagnosis result’ image, the doctor can override the AI system’s diagnosis. The ‘point cloud’ image at the bottom shows a 3D model of the lesion. © FZI Research Centre for Information Technology
Website address: https://www.gesundheitsindustrie-bw.de/en/article/news/ai-added-value-digital-melanoma-diagnostics