Image, ontology and process-supported assistance for minimally invasive endoscopic surgery

Minimally invasive surgical surgeries are characterized by their high technical complexity and their field of application, which is strongly determined by technology. In the context of demographic change, older patients in particular benefit from reduced access trauma, faster recovery and shorter times for inpatient stays and rehabilitation. Minimally invasive surgery involves access to the inside of the body through very small incisions. The special challenges for the surgeon are orientation and navigation without a direct view of the surgical area as well as instrument guidance with limited hand-eye coordination by looking at endoscopy images on a screen.

Target position

The aim of the BIOPASS project is to develop a new navigation approach and supporting measures to increase reliability on the basis of inherent information and data from endoscopic imaging and the course of the operation. An assistance system is to be developed which - in an intermediate step as a hybrid system (i.e. in combination with conventional tracking), later without the need for additional marker tracking cameras and conventional imaging (e.g. CT, MRI) - can only work on the basis of learned situations from the endoscopic imaging sequences. The intended technique should adapt to the surgeon's cognitive abilities and learn his individual preferences so that he is supported by the system in his actual clinical work. Young and inexperienced surgeons, for example, can be given more assistance in a given situation if the system detects uncertainties in the movement of the endoscope, whereas older users benefit from a reduction in the technical complexity and hardware of navigation systems. The use of the innovative assistance system reduces the external complexity of the surgeon's work environment by eliminating the components of conventional navigation systems such as tracking cameras and artificial markers and simplifying his work processes. Patients, as secondary users of BIOPASS navigation, benefit above all from a safe and less invasive procedure without external markers.

The aim is to develop a surgical assistance system for minimally invasive surgery based on a new navigation method. In this approach, the existing surgical navigation systems are to be supplemented or replaced by a marker-free navigation system that works with inherent information and data from endoscopic imaging and the surgical process. The assistance system should - in an intermediate stage as a hybrid system (i.e. in combination with conventional tracking), later without the need for additional markers, tracking cameras and conventional imaging (e.g. CT, MRI) - only be able to work on the basis of learned situations from the endoscopic imaging sequences.

Since not all users and scenarios can be determined in advance, the system must react to changing conditions in a self-learning way depending on the situation. In new situations or with special pathologies, the corresponding process and image data are transferred into the knowledge base, taking into account the ontology and by adapting the underlying process model. They are thus available as a reference for the following interventions. The system continuously expands itself through interaction with the user.


The processing and analysis of the endoscopic image data is the essential basis for a markerless navigation system. The image data provided by the project partners were analysed in order to identify essential features and problems. Based on this, methods were finally developed to automatically evaluate the image data received by the system during runtime with regard to their quality, to make minor image corrections and to make statements about their usability for further analysis. In addition, various methods were realized as separate modules that calculate basic image characteristics. These modules and data were exchanged with the project partners in order to give the partners the possibilities for data pre-filtering and evaluation. This concerns methods for the evaluation of the image sharpness, for the detection of the camera position in relation to the target area ("inside / outside recognition"), a recognition and correction of highlights in the images as well as an initial depth reconstruction on the basis of mono camera images.

The process chain established within the framework of the project provides for the methods and modules of the project partners to be able to access local image properties in order to include them in their own calculations, evaluations and decisions. In particular, statements about local color changes within an image as well as appearing/repeating patterns and textures were considered relevant and the endoscopy videos were examined. Finally, modules were implemented by Dornheim and made available to the project partners which determine the desired image attributes for given input image data in the individually required granularity.

In order to implement data exchange at module level and thus be able to develop a common software system, it was necessary to establish a common data format standard and a suitable message system. Based on positive previous experience, it was proposed to build the system on the basis of an Observer/Observable concept. The modules run independently and transmit your data to a central broker, who takes care of message distribution and notification of registered "subscribers" (client modules). In particular, this was realized using the "Message Queuing Telemetry Transport" protocol (MQTT), i.e. all realized image processing and image analysis modules are able to make their data available to a central MQTT broker or to receive input data from there for processing. This setup was successfully tested with all running modules during several live tests in the context of phantom data recordings. The messages exchanged via MQTT contain the data of the modules in JSON format. The only exceptions are the extensive image data streams, for which the MQTT messages were extended by a binary image format.

Stereoscopic image data in particular will be of interest in endoscopy due to the technical development of education. For this reason, parallel to the processing and analysis of ordinary monoscopic endoscopy data, the focus has also been placed on novel stereoscopic images. After a detailed analysis of the data, existing methods were further developed and integrated, which allow a depth estimation from the stereoscopic endoscopy data. On the basis of such initial depth estimates, methods for the reconstruction of surface data and their texturing were then implemented. After such a reconstruction process for individual image pairs could be successfully carried out for one time each from a series of images, it was important to integrate the temporal component. One of the main challenges then was to locate the images of the different points in time spatially and to bring the reconstruction data into a common space based on this. This finally made it possible to consolidate the reconstructed images into an overall model of the area optically captured by the stereo endoscope.

Especially for the processing of the stereoscopic image data and the depth estimation and surface reconstruction based on it for the recognition of spatial image characteristics, an evaluation of the realized methods is indispensable in order to be able to evaluate the achievable quality. Therefore, a workflow based on different tools was developed, which can be used to create clearly defined objects with known dimensions, texture them, visualize them three-dimensionally and finally export a camera movement as video animation. The artificially generated videos can then be used for reconstruction under controlled conditions. The resulting 3D data was then compared with the artificially generated reference data.

Project partners and research funding
  • LOCALITE GmbH, St. Augustin (LOCALITE)
  • Universitätsklinikum Leipzig, Klinik und Poliklinik für HNO-Heilkunde (HNO)
  • Klinikum rechts der Isar, Klinik für Viszeralchirurgie (VISZ)
  • Leipzig University, Innovation Center Computer Assisted Surgery (ICCAS)
  • TU Munchen, Klinikum rechts der Isar, Forschungsgruppe für Minimal-invasive Interdisziplinäre Therapeutische Intervention (MITI)
  • Zuse-Institut Berlin (ZIB)