Medical Applications of 3D Surface Measurement

Accurate, robust, and fast 3D surface measurements are crucial in modern medical imaging and can support a wide range of clinical applications from surgical guidance to telemedicine. For instance, providing surgeons with an accurate, real-time 3D view of the surgical field can be essential for complex procedures, e.g., in robotic-assisted or AR/VR-assisted surgeries. However, established techniques like structured light triangulation can face significant difficulties in surgical environments due to partially specular reflections from wet, bloody surfaces. Several of our developments aim to actively address these key challenges that have historically limited the use of advanced 3D surface measurement techniques in medical settings. Moreover, we are working on novel methods to enable high-quality 3D imaging using commodity devices like tablets and smartphones with the goal to make basic diagnostic tools, e.g., for the 3D analysis of skin lesions, more accessible. Additionally, our previous projects have focused on seamless co-registration of multimodal imaging data (MRI, EEG, MEG) and motion compensation during procedures like radiation therapy or CT imaging, leveraging real-time 3D sensing for improved results.

Besides imaging the surface of the human body, a significant part of our research portfolio is devoted to imaging through scattering media, which has potential applications in medical imaging to image deep inside the human tissue (see our Synthetic Wavelength Imaging research track for more info).

SkinScan

The “SkinScan” research track introduces a series of systems that only require commodity devices such as screens, (web-) cameras, low-end tablets or mobile phones to capture high-quality 3D data on human skin. For instance, one embodiment of the system exploits screen and front camera of mobile devices for photometric stereo-inspired 3d measurements. This can potentially enable several early diagnostics applications, including the 3D analysis of skin lesions. Due to their high sensitivity to high object frequencies, the systems can be also used to scan subtle skin features, like wrinkles or fingerprints.

Selected Publications

Skinscan: Low-cost 3d-scanning for dermatologic diagnosis and documentation.

Merlin A Nau, Florian Schiffers, Yunhao Li, Bingjie Xu, Andreas Maier, Jack Tumblin, Marc Walton, Aggelos K Katsaggelos, Florian Willomitzer, Oliver Cossairt

2021 IEEE International Conference on Image Processing (ICIP), 2021.

Skinscan: 3D Dermatologic Diagnosis and Documentation with Commodity Devices.

Merlin A Nau, Florian Schiffers, Yunhao Li, Bingjie Xu, Andreas Maier, Jack Tumblin, Marc Walton, Aggelos K Katsaggelos, Florian Willomitzer, Oliver Cossairt

2021 55th Asilomar Conference on Signals, Systems, and Computers, 2022.

Single-Shot Fringe Projection Triangulation on Wet and Shiny tissue

Fringe Projection Triangulation is widely regarded as a “gold standard” in macroscopic optical 3D imaging due to its excellent noise characteristics and the high accuracy it can deliver. However, the method is inherently designed for diffuse surfaces and encounters significant challenges when applied to wet, shiny, or bloody tissue surfaces, such as those found in open wounds. In this research track, we focus on developing novel computational and hardware-based techniques to suppress specular reflections and mitigate artifacts caused by these highly reflective, complex surfaces. Our goal is to enable motion-robust and accurate 3D measurements of tissue in surgical and medical contexts, by leveraging adapted single-shot Fringe Projection Triangulation methods. Additionally, we are exploring new calibration strategies tailored to the unique requirements of medical environments, further enhancing measurement accuracy and reliability for clinical applications.

Selected Publications

Radiometric calibration of active 3D imaging setups using superquadric fitting.

James Taylor, Jiazhang Wang, Florian Willomitzer

Interferometry and Structured Light 2024, 2024.

“On-the-Fly” Radiometric Calibration of Active 3D Imaging Setups using Superellipse Fitting.

James Taylor, Jiazhang Wang, Florian Willomitzer

Frontiers in Optics, 2023.

Hand-Guided Single-Shot Multi-Line Triangulation

Multi-line triangulation systems are highly effective for capturing precise 3D profiles of surfaces in single-shot measurements. However, physical and information-theoretical constraints limit the number of projected lines, resulting in relatively low data densities per single-shot for conventional techniques. This research track, aims for the development of novel concepts to push the data density of multi-line triangulation systems to their theoretical limits, enabling the creation of dense and accurate 3D models of the human body. Each model is captured in single-shot, which allows for “3D-videos” of fast-moving objects. In the related “Flying Triangulation” approach, a sparse line triangulation sensor projects approximately 10 narrow lines onto the surface while advanced real-time registration algorithms align the captured profiles. This method is particularly well-suited for free-hand scanning of the human body, or when the patient moves relative to the sensor. The developed techniques have demonstrated significant potential across various medical imaging applications, e.g., in EEG and MEG co-registration and source localization, or motion compensation during MEG and CT measurements and radiation therapy (see below).

Visit the Osmin3D YouTube channel for more Videos.