Project Overview
Work plan and project achievements
TRANSFORM LIVER will use AI for the first time to enable a systemic understanding of liver diseases based on multimodal routine clinical data.
Data analysis and preparation
The focus is on analyzing and pre-sorting our database. Our experts will work intensively on pathological images, going beyond the current technical standard based on Convolutional Neural Networks (CNNs).
Transformers for image analysis
The biggest technical challenge lies in the size of the image data in the gigapixel range. Conventional approaches fragment these images into smaller parts for analysis. Our goal is to overcome this limitation by developing specific transformer architectures. These transformers are trained directly on the entire histological tissue section, taking into account biologically relevant interactions between distant image regions.
Iterative training process and data expansion
Each area of the whole image is spatially barcoded to recognize its position in the original pathological complete image. By processing slightly different sections of the image in each iteration, we enable the Transformer to process the entire gigapixel image over hundreds of iterations. This approach is not only for data augmentation, but also for generating robust classifiers.
Clinical applications and benefits
Our mission goes beyond technical development; we ensure that our approach has a direct medical benefit. The transformers are specifically trained on diagnostic challenges, including the detection of rare liver diseases in a cohort of patients with fatty liver disease. Furthermore, we focus on the prognosis of liver diseases, especially hepatocellular carcinoma, and on prediction tasks to determine the response of individuals with liver cancer to specific therapies.
Transformer
Transformer for multimodality
Transformer technology, originally developed for processing natural language, can be used in a wide range of applications. Vision transformers treat image sections as tokens that are linearized and converted into one-dimensional vectors. Our innovative approach is not to limit these transformers to image data, but to extend them to multimodal data. Within the existing consortium, patient metadata such as age and sex are made available. The integration of these different types of data – images and clinical information – enables improved prediction. This technological and medical innovation paves the way for a transformer that integrates images and clinical data in the field of liver disease – a unique development to date.
Transformer for explainability
The ability of transformers to link different parts of the input signal through awareness is essential for explainable models. In our project we use multimodal transformers. This allows users to interactively explain how the model is making certain decisions. For example, a user can select a word or a section of an image, and the transformer visualizes which other parts of the text or image are related to this selection. These explainable approaches are new in the medical field and we will develop them and evaluate them extensively with medical experts. These explanatory mechanisms make it possible to understand links between different characteristics of a patient, such as age, comorbidities and laboratory values, and parts of a pathological image. Our user-friendly interface ensures that external experts can also interactively use the trained system to research liver diseases.
Transformer services in systems medicine
Our research project TRANSFORM LIVER goes beyond conventional AI approaches by utilizing transformers in systematic medicine for a comprehensive understanding of liver diseases. In contrast to non-transparent „black box“ systems, we aim to use our AI systems not only to generate clinically relevant results, but also to advance a mechanistic understanding of disease processes.
In our project, we train transformers with clinically relevant data from patients with liver disease. Through innovative explanatory methods, we enable human experts to understand the transformer’s decisions. Our approach integrates seamlessly with the philosophy of „systems medicine“, which aims to gain a mechanistic understanding of disease from large data sets.
We enable experts to interact with the trained transformer. For example, they can load histological specimens, mark certain areas and see which other areas in the image receive a high level of attention. Such interactive experiments lead to mechanistic hypotheses that can be tested experimentally. We also use multimodal transformers to visualize new data interactions, for example by tagging lab values together with histological images.