The aim is to develop a surgical assistance system for minimally invasive surgery based on a new navigation method. In this approach, the existing surgical navigation systems are to be supplemented or replaced by a marker-free navigation system that works with inherent information and data from endoscopic imaging and the surgical process. The assistance system should - in an intermediate stage as a hybrid system (i.e. in combination with conventional tracking), later without the need for additional markers, tracking cameras and conventional imaging (e.g. CT, MRI) - only be able to work on the basis of learned situations from the endoscopic imaging sequences.

Since not all users and scenarios can be determined in advance, the system must react to changing conditions in a self-learning way depending on the situation. In new situations or with special pathologies, the corresponding process and image data are transferred into the knowledge base, taking into account the ontology and by adapting the underlying process model. They are thus available as a reference for the following interventions. The system continuously expands itself through interaction with the user.

Content

The processing and analysis of the endoscopic image data is the essential basis for a markerless navigation system. The image data provided by the project partners were analysed in order to identify essential features and problems. Based on this, methods were finally developed to automatically evaluate the image data received by the system during runtime with regard to their quality, to make minor image corrections and to make statements about their usability for further analysis. In addition, various methods were realized as separate modules that calculate basic image characteristics. These modules and data were exchanged with the project partners in order to give the partners the possibilities for data pre-filtering and evaluation. This concerns methods for the evaluation of the image sharpness, for the detection of the camera position in relation to the target area ("inside / outside recognition"), a recognition and correction of highlights in the images as well as an initial depth reconstruction on the basis of mono camera images.

The process chain established within the framework of the project provides for the methods and modules of the project partners to be able to access local image properties in order to include them in their own calculations, evaluations and decisions. In particular, statements about local color changes within an image as well as appearing/repeating patterns and textures were considered relevant and the endoscopy videos were examined. Finally, modules were implemented by Dornheim and made available to the project partners which determine the desired image attributes for given input image data in the individually required granularity.

In order to implement data exchange at module level and thus be able to develop a common software system, it was necessary to establish a common data format standard and a suitable message system. Based on positive previous experience, it was proposed to build the system on the basis of an Observer/Observable concept. The modules run independently and transmit your data to a central broker, who takes care of message distribution and notification of registered "subscribers" (client modules). In particular, this was realized using the "Message Queuing Telemetry Transport" protocol (MQTT), i.e. all realized image processing and image analysis modules are able to make their data available to a central MQTT broker or to receive input data from there for processing. This setup was successfully tested with all running modules during several live tests in the context of phantom data recordings. The messages exchanged via MQTT contain the data of the modules in JSON format. The only exceptions are the extensive image data streams, for which the MQTT messages were extended by a binary image format.

Stereoscopic image data in particular will be of interest in endoscopy due to the technical development of education. For this reason, parallel to the processing and analysis of ordinary monoscopic endoscopy data, the focus has also been placed on novel stereoscopic images. After a detailed analysis of the data, existing methods were further developed and integrated, which allow a depth estimation from the stereoscopic endoscopy data. On the basis of such initial depth estimates, methods for the reconstruction of surface data and their texturing were then implemented. After such a reconstruction process for individual image pairs could be successfully carried out for one time each from a series of images, it was important to integrate the temporal component. One of the main challenges then was to locate the images of the different points in time spatially and to bring the reconstruction data into a common space based on this. This finally made it possible to consolidate the reconstructed images into an overall model of the area optically captured by the stereo endoscope.

Especially for the processing of the stereoscopic image data and the depth estimation and surface reconstruction based on it for the recognition of spatial image characteristics, an evaluation of the realized methods is indispensable in order to be able to evaluate the achievable quality. Therefore, a workflow based on different tools was developed, which can be used to create clearly defined objects with known dimensions, texture them, visualize them three-dimensionally and finally export a camera movement as video animation. The artificially generated videos can then be used for reconstruction under controlled conditions. The resulting 3D data was then compared with the artificially generated reference data.