Open Semester Projects and Thesis

Temporally consistent depth reconstruction from a monocular video is an important task for 3D machine vision. Great progress has been made towards that goal with recent generative AI models. Still, existing methods are either limited to very short videos (e.g., DepthCrafter []), or produce results that are not completely smooth in time (e.g., RollingDepth [], developed at PRS). In the project, students will work towards improving RollingDepth by post-processing its outputs with either analytical and learning-based methods[3].

[1] Ke, B., Narnhofer, D., Huang, S., Ke, L., Peters, T., Fragkiadaki, K., Obukhov, A. and Schindler, K., 2024. Video Depth without Video Models. arXiv preprint arXiv:2411.19189.

[2] Hu, W., Gao, X., Li, X., Zhao, S., Cun, X., Zhang, Y., Quan, L. and Shan, Y., 2024. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095.

[3] Wang, Y., Shi, M., Li, J., Huang, Z., Cao, Z., Zhang, J., Xian, K. and Lin, G., 2023. Neural video depth stabilizer. ICCV.

The project is suitable for both a BSc Thesis and MSc Project.

Requirements

• Basic knowledge of deep machine learning

• Ideally, elementary understanding of denoising diffusion models

• Programming in Python/PyTorch.

The overal goal to achieve smooth video depth requires several steps:

1) Implement quantitative consistency evaluation and evaluate existing methods

2) Implement and test analytical smoothing schemes

3) Implement and test smoothing based on deep neural networks

4) Optionally, develop a new method with better performance

Dominik Narnhofer (dnarnhofer@ethz.ch) Bingxin Ke (bingke@ethz.ch) Anton Obukhov (anton.obukhov@gmail.com) Konrad Schindler (schindler@ethz.ch)

Thermal imaging lags behind regular photography due to the technological challenges of measuring radiant energy in the thermal infrared spectrum. The largest limitation is resolution: obtaining HD images is prohibitively expensive. Therefore, most conventional thermal cameras deliver low-res outputs, and the problem of increasing the perceived resolution of thermal sensors becomes important. The recent uptake in generative modeling [1] opens up new possibilities to solve such image enhancement tasks [2]. In particluar, Marigold-DC [3], developed in the PRS group, combines a pre-trained monocular depth prediction model with sparse measurements from a physical sensor [3]. In the thesis project, you will explore the possibility to apply generative vision foundation models in a similar manner in order to improve the spatial resolution of thermal images.

[1] https://arxiv.org/abs/2112.10752 Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models”

[2] https://arxiv.org/abs/2312.02145 Ke et al., “Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation”

[3] https://marigolddepthcompletion.github.io, Viola et al., “Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion”

Requirements

● Problem-solving mindset. ● Experience with Python and PyTorch ● Prior exposure to diffusers and the Hugging Face ecosystem is not mandatory, but a plus ● Willingness to learn about generative AI models for images (c.f. the references in the description above)

Location

On-site at the Photogrammetry and Remote Sensing Laboratory (Prof. Schindler, ETH Zürich)

The thesis aims to develop a pipeline for thermal image super-resolution. The inputs are a high-resolution RGB image and a corresponding low-resolution thermal image; the desired output is a high-resolution thermal image. Steps towards that goal are

(1) conduct a review of existing approaches to thermal super-resolution

(2) collect suitable datasets for evaluation, guidance, and possibly training

(3) set up the evaluation protocol

(4) apply the Marigold-DC approach, identify remaining challenges, and take steps to solve them

Julius Erbach (erbachj@student.ethz.ch) Anton Obukhov (Huawei Research Zürich, anton.obukhov@huawei.com) Konrad Schindler (schindler@ethz.ch)

Tree species maps are essential for better forest management, forest cover, biomass, and biodiversity assessment. The temporal and spatial location and identification of tree species is extremely important and necessary for forest management and conservation. The use of remote sensing products in forestry allows for time flexible and cost-effective assessment of forest characteristics. Deep learning methods enable high predictive accuracy and have the potential to revolutionize forestry understanding, data collection and enable the development of numerous applications. Tree species identification is essential for assessing biodiversity, understanding forest resilience to climate change, and developing forest management strategies. However, identifying tree species is challenging, and further research needs to focus on developing new models to address this issue.

This work aims to apply novel and powerful deep learning approaches to detect and identify individual trees and tree species in mixed forests. The work comprises the following milestones:

1. data preparation;

2. development of a deep learning model that can work with multiple input data;

3. evaluation of the model robustness across new areas.

Methods

The potential of convolutional neural networks (CNNs) or Transformers and high-resolution RGB imagery will be evaluated for mapping tree species in mixed forests. The analysis will be implemented using ArcGIS Pro and Python. A database of >80000 geolocated trees and 20000 tree canopy delineations are already available for the DL model training.

Wanted

Highly motivated student interested in modelling and who is willing to learn. The project has a flexible starting date.

You will get to

• Learn new and highly required theoretical and practical knowledge about Deep Learning object recognition that will be of great importance for your future work.

• Expand your network by discussing your work with experts from the intersecting fields of forest sciences, and computer sciences.

• Be a co-author on a publication resulting from this work.

• Be part of a motivated, fun, and energetic team of scientists.

The project has a flexible starting date.

Supervisors: Dr. M. Beloiu Schwenke from the Department of Environmental Systems Science and Prof. K. Schindler from the Department of Civil, Environmental and Geomatic Engineering.

If the idea of participating in cutting-edge interdisciplinary research excites you, please contact us.

Location On-site at the Photogrammetry and Remote Sensing Laboratory or Forest Resources Management(ETH Zurich)

Application We look forward to receiving your online application with the following documents: (1) your CV and (2) your transcript of records.