Open Semester Projects and Thesis

So far damage mapping is mostly done by manual inspection of very high-resolution (VHR) images. Current research efforts try to automate it with deep learning (DL) to enable rapid, large-scale assessments. An important asset in that context are training datasets. The main resource at present is the xBD dataset [1], which comprises VHR satellite images acquired before and after natural disasters, paired with detailed manual damage assessments. Humanitarian organizations require accurate and timely data on building damage, e.g. following natural disasters or during armed conflicts, to efficiently direct aid and resources. Satellite imagery has been an invaluable tool in this endeavor, especially in hard-to-reach or high-risk areas, where direct on-the-ground reports are not an option. So far damage mapping is mostly done by manual inspection of very high-resolution (VHR) images. Current research efforts try to automate it with deep learning (DL) to enable rapid, large-scale assessments. An important asset in that context are training datasets. The main resource at present is the xBD dataset [1], which comprises VHR satellite images acquired before and after natural disasters, paired with detailed manual damage assessments. However, xBD contains only expensive VHR images and is therefore not suitable for large-scale and/or long-term monitoring efforts, such as tracking damages during armed conflicts. Building on an earlier project at the PRS group, the thesis shall extend the xBD dataset with freely available imagery from Earth observation satellites such as Sentinel-1 and Sentinel-2. These sensors offer multi-spectral and SAR images that have lower resolution, but offer frequent and regular coverage anywhere on Earth for free. The hope is that, by including such imagery, the dataset will form a basis to build systems able to automatically monitor building damages during ongoing conflicts. **Requirements** - Programming in Python and Pytorch - Experience with GIS software and remote sensing data - Ideally, basic knowledge of deep learning

- Extend the xBD dataset with freely available satellite imagery. - Benchmark damage detection performance when using the extended dataset

Olivier Dietrich (odietrich@ethz.ch) Konrad Schindler (schindler@ethz.ch)

Historical architecture forms an important part of our cultural heritage. Both to document and digitally preserve historical buildings and to support their physical maintenance and preservation, there is a need to digitize them into structured 3D building models. In the project, we aim to adopt recent progress in machine learning and (visual) AI to analyze and reconstruct structured building models from scan data. The project is a collaboration project between Institute of Geodesy and Photogrammetry and WolkenVision AG (https://wolkenvision.com). Your contribution will support practical applications in heritage conservation, and will allow you to gain hands-on experience in a tech industry start-up. The project is suitable for an MSc project or MSc thesis. Group work (up to two people) is possible, and strongly encouraged for the MSc project. **Requirements:** - Proficiency in Python and familiarity with PyTorch. - A strong foundation in geomatics and/or computer vision, and preferrably experience with 3D point clouds. - Data science and machine learning knowledge.

- Develop and implement efficient methods to automatically segment point clouds into semantically meaningful units (like walls, windows, columns, etc.) - Building on the segmentation result, devise computational methods to automatically convert the point cloud into a structured model made of geometric entities (like, edges, planes, etc.) - Combine those 3D entities into parametric, CAD-like object models.

Konrad Schindler, schindler@ethz.ch Dominik Narnhofer, dnarnhofer@ethz.ch Boxiao Ma, boxiao.ma@wolkenvision.com

Marigold, a recent diffusion model and associated fine-tuning protocol shows impressive results for monocular depth estimation. Its core principle is to leverage the rich visual knowledge in Stable Diffusion, and fine-tune with a small amount of synthetic data. Although showing very great results, there are many exciting future directions to explore, e.g.: How to speed up? How to adapt it for other modalities? High-resolution? How to predict depth in metrics, etc. **Requirements** - Programming in Python and Pytorch - Good knowledge of deep learning - [Preferable] have experience in diffusion models **Reference** Marigold paper: https://arxiv.org/abs/2312.02145

Develop the next generation of the SOTA monocular predictor

Songyou Peng (songyou.peng@inf.ethz.ch) Anton Obukhov (anton.obukhov@gmail.com) Bingxin Ke (bingxin.ke@geod.baug.ethz.ch)

In previous work at the Photogrammetry and Remote Sensing Group, we have made the first step to synthesize realistic LiDAR scans by optimizing a neural field scene representation. This prototype [1] opens up many directions to be explored. For example, we would like to extend it to handle dynamic scenes [2] and adverse weather. In the student project, we look into ways to exploit existing 3D assets for LiDAR view synthesis. There are at least two way to use digital object models for our purposes: (1) in real LiDAR scans only those parts of the object are captured that face the scanner, so we are missing information when rendering them from a different viewpoint. The first part of the project will investigate how to construct an a-priori model of object shape from existing 3D assets to improve reconstruction quality. (2) 3D assets can also be used to improve the diversity of simulated scans, by inserting new, dynamic objects - for instance inserting additional cars or trucks to create more complex traffic scenarios. To do this in a realistic manner, one cannot simply cast rays and intersect them with the surfaces of the new object: it is also necessary to correctly simulate point cloud noise, ray drops due to mirror reflections, attenuation etc., and multi-returns on the object boundary. The second part of the project will explore ways to directly learn these patterns from large volumes of data and to transfer them to synthetic scans. Students should be motivated, have previous experience with PyTorch, and ideally also with point cloud processing. Knowledge of generative and 3D AI tools like NeRF, GANs and diffusion models is a plus. If successful the project may lead to a publication in a reputable robotics or computer vision conference. The project is suitable for MSc project or MSc thesis. Group work (up to two people) s possible, and strongly encouraged for the MSc project. [1] Huang, Shengyu, et al. "Neural LiDAR Fields for Novel View Synthesis." ICCV 2023. [2] Huang, Shengyu, et al. “Dynamic Neural LiDAR Fields for Novel View Synthesis." to appear. [3] Manivasagam, Sivabalan, et al. "Towards Zero Domain Gap: A Comprehensive Study of Realistic LiDAR Simulation for Autonomy Testing." ICCV 2023. Settings for applications: Experience with PyTorch required Practical experience with point clouds preferred Ways to stand out: knowledge of generative and 3D AI tools like NeRF, GANs and diffusion models

1) Explore learning methods to leverage the information from existing digital 3D assets and construct a shape prior for object reconstruction from LiDAR scans 2) Develop methods to insert 3D objects into synthetic LiDAR scans by transferring scanning patterns from real scans.

Shengyu Huang, shengyu.huang@geod.baug.ethz.ch