A marker-free assistance system
for minimally invasive surgery
Objective
The aim of the BIOPASS project is to develop a new navigation approach and supporting
measures for increasing operational reliability based on inherent information and data from
endoscopic imaging and from the course of the surgical process. The intention is to develop
an assistance system that – as an intermediate stage in the form of a hybrid system (i.e.,
in combination with conventional tracking), and later without the need for additional
markers, tracking cameras, and conventional imaging (such as CT or MRI) – can operate solely
on the basis of situations learned from endoscopic recording sequences. In doing so, the
intended technology is designed to adapt to the surgeon's cognitive abilities and to learn
their individual preferences, so that they are supported by the system in their actual
clinical work. Young and inexperienced surgeons can, for example, be given more situational
guidance when the system detects uncertainties in endoscope movement, whereas more
experienced users benefit from the reduction in technical complexity and the decrease in
navigation-system hardware. The use of this innovative assistance system leads to a
reduction in the external complexity of the surgeon's working environment, as the components
of conventional navigation systems – such as tracking cameras and artificial markers – are
eliminated and their workflows are simplified. Patients, as secondary users of BIOPASS
navigation, benefit above all from a safer and less invasive procedure without external
markers.
The goal is to develop a surgical assistance system for minimally invasive surgery based on
a new navigation method. In this approach, the existing surgical navigation systems are to
be complemented or replaced by a marker-free navigation system that works with inherent
information and data from endoscopic imaging and the surgical process. The assistance system
is intended – as an intermediate stage in the form of a hybrid system (i.e., in combination
with conventional tracking), and later without the need for additional markers, tracking
cameras, and conventional imaging (such as CT or MRI) – to operate solely on the basis of
situations learned from endoscopic recording sequences.
Since not all users and scenarios can be determined in advance, the system must respond in a
self-learning manner to changing conditions as they arise. To this end, in novel situations
or in the case of particular pathologies, the corresponding process and image data are
incorporated into the knowledge base – taking the ontology into account and by adapting the
underlying process model – and are then available as a reference for subsequent procedures.
Through interaction with the user, the system continuously expands itself.
Content
For a marker-free navigation system, the processing and analysis of endoscopic image data is
the essential foundation. The image data provided by the project partners was analyzed in
order to identify key features as well as potential problems. Building on this, methods were
ultimately developed to automatically assess the quality of the image data entering the
system at runtime, to perform minor image corrections, and to make statements about the
data's suitability for further analysis. In addition, various methods were implemented as
separate modules that compute fundamental image features. These modules and data were
exchanged with the project partners in order to give them the means to pre-filter and
evaluate the data. This concerns methods for assessing image sharpness, for detecting the
camera position relative to the target area ("inside/outside detection"), for detecting and
correcting specular highlights in the images, and for an initial depth reconstruction based
on single-camera (mono) images.
The processing pipeline established over the course of the project provides for the methods
and modules of the project partners to be able to access local image properties, so that
they can incorporate them into their own computations, evaluations, and decisions. In
particular, statements about local color changes within an image as well as
occurring/recurring patterns and textures were deemed relevant for this purpose, and the
available endoscopy videos were examined accordingly. Finally, Dornheim implemented modules
and made them available to the project partners that, for given input image data, determine
the respectively desired image attributes at the individually required level of granularity.
In order to enable data exchange at the module level and thus develop a shared software
system, it was necessary to establish a common data-format standard as well as a suitable
messaging system. Based on positive prior experience, it was proposed to build this on an
Observer/Observable concept. Here, the modules run independently of one another and transmit
their data to a central broker, which handles message distribution and the notification of
registered "subscribers" (client modules). Specifically, this was implemented using the
"Message Queuing Telemetry Transport" protocol (MQTT); that is, all implemented image-
processing and image-analysis modules are able to make their data available to a central
MQTT broker as well as to receive input data from it for processing. This setup was
successfully tested during several live tests as part of phantom data recordings with all
running modules. The messages exchanged via MQTT contain the modules' data in JSON format.
The only exception is the extensive image data streams, for which the MQTT messages were
extended with a binary image format.
Stereoscopic image data in particular will be of interest in endoscopy owing to ongoing
technical advances in imaging. For this reason, in parallel with the processing and analysis
of conventional monoscopic endoscopy data, the focus was also placed on novel stereoscopic
images. Following a thorough analysis of the data, existing methods were therefore refined
and integrated that enable depth estimation from stereoscopic endoscopy data. On the basis
of such initial depth estimates, methods for the reconstruction of surface data and for its
texturing were then implemented. After such a reconstruction process could be carried out
successfully for individual image pairs, each corresponding to one point in time within a
recording series, it was important to also integrate the temporal component. One of the key
challenges then lay in spatially registering the recordings from the different points in
time with one another and, on this basis, bringing the reconstruction data into a common
space. This finally made it possible to consolidate the reconstructed recordings into an
overall model of the region optically captured with the stereo endoscope.
Especially for the processing of the stereoscopic image data and the depth estimation and
surface reconstruction built upon it for detecting spatial image features, an evaluation of
the implemented methods is indispensable in order to be able to assess the achievable
quality. A workflow based on various tools was therefore developed, by means of which
clearly defined objects with known dimensions can first be created, textured, visualized in
three dimensions, and finally exported as a video animation of a camera fly-through. The
synthetically generated videos can subsequently be used for reconstruction under controlled
conditions. The 3D data generated in this way was then compared with the synthetically
generated reference data.
