Continuous Aerial Path Planning for 3D Urban Scene Reconstruction

ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2021)


Han Zhang1    Yucong Yao1    Ke Xie1    Chi-Wing Fu2    Hao Zhang3    Hui Huang1*

1Shenzhen University    2The Chinese University of Hong Kong    3Simon Fraser University



Fig. 1. A 0.1km2 urban area (roughly 0.25km × 0.53km) is 3D-reconstructed from 1,423 images captured by a single-camera drone (DJI Phantom 4 RTK) flying along a continuously-planned path by our method: 18,491m in length and color-coded from blue to red with the two small flags indicating the start and end points. Our path planning algorithm optimizes scene coverage, scene capture efficiency, and path quality, resulting in fewer sharp turns, shorter path lengths, as well as higher 3D reconstruction quality (see the geometric and appearance details in the zoomed-in inset pairs above) at a reduced flight time.


Abstract

We introduce the first path-oriented drone trajectory planning algorithm, which performs continuous (i.e., dense) image acquisition along an aerial path and explicitly factors path quality into an optimization along with scene reconstruction quality. Specifically, our method takes as input a rough 3D scene proxy and produces a drone trajectory and image capturing setup, which efficiently yields a high-quality reconstruction of the 3D scene based on three optimization objectives: one to maximize the amount of 3D scene information that can be acquired along the entirety of the trajectory, another to optimize the scene capturing efficiency by maximizing the scene information that can be acquired per unit length along the aerial path, and the last one to minimize the total turning angles along the aerial path, so as to reduce the number of sharp turns. Our search scheme is based on the rapidly-exploring random tree framework, resulting in a final trajectory as a single path through the search tree. Unlike state-of-the-art works, our joint optimization for view selection and path planning is performed in a single step. We comprehensively evaluate our method not only on benchmark virtual datasets as in existing works but also on several large-scale real urban scenes. We demonstrate that the continuous paths optimized by our method can effectively reduce onsite acquisition cost using drones, while achieving high-fidelity 3D reconstruction, compared to existing planning methods and oblique photography, a mature and popular industry solution.



Fig. 3. Algorithm overview. Given a coarse proxy of the target scene (a), we first build a view information field (VIF) (b) to encode continuous view coverage information of the scene in the free space. Then, we start our path planning algorithm to iteratively sample viewpoints continuously in free space using the VIF and expand our random search tree (c) to explore the space, until we find a path (d) that adequately covers the scene. In (b), the red (high) to blue (low) colors indicate the amount of scene information. In (c), the red dot indicates the root of the tree and the black dots indicate the current best branch. In (d), the brown dots are waypoints and the blue dots are viewpoints in-between. In (e), some challenging regions of the final reconstructed scene are shown in blown-up views.



Fig. 7. A running example of our path planning algorithm. The bigger green dot near the middle marks the root node, whereas the red and yellow dots mark the internal and leaf nodes, respectively, in the random tree. Also, the yellow dots and the blue edges in-between mark the dominant (optimal) path at the moment. The colors on the scene proxy reveal the amount of scene reconstructability (red (low) to blue (high)) that remains under the dominant path.



Fig. 8. To adapt our method for single-camera drones, we extend the 3D view information field to become a 5D view information field to account for the view direction (S2) and position (R3) of viewpoints in the free space (left). Note that, aerial paths planned for single-camera drones are usually longer (and more complex) than paths planned for multi-camera drones (for the same scene) since the drone has to travel longer to capture different aspects of the scene.



Fig. 12. Visual comparison on ArtSci (left: 76,900m2) and Library (right: 69,471m2). In each scene, first column shows the result of our methods, whereas second column shows the result of [Zhou et al. 2020]. Please zoom in to look at the blown-up views to compare the details in the 3D reconstructions.



Fig. 13. The scene Residency is of 0.1km2. Top left: our planned aerial path (from blue to red). Top right: aerial path (in green, from blue dot to red dot) by oblique photography (OP). Bottom: “detail” comparison on this scene; 3D reconstruction results of our method (first row) and OP (second row).



Fig. 14. The scene High-rise is the largest one in size, 0.5km2 (0.71km × 0.87km), covering buildings and other structures with large variation in height. The tallest building is 140m high, so the aerial path for oblique photography (OP) is set at a height of 170m, while our path can direct the drone to adaptively fly between 80m and 170m elevations, to best capture the scene details. The top row shows the entire 3D scene reconstructions produced from images captured using our path (top left) and OP (top right). Due to the complexity of the scene, we show four sets of blown-up views at the bottom for a better visual comparison between the fine details reconstructed. Also, we show plain renderings without colors for better assessment of the reconstructed geometry.



Acknowledgements

We thank all reviewers for their valuable comments. This work was supported in parts by NSFC (U2001206), Guangdong Talent Program (2019JC05X328), Guangdong Science and Technology Program (2020A0505100064), DEGP Key Project (2018KZDXM058), Shenzhen Science and Technology Program (RCJC20200714114435012, JCYJ20210324120213036), HKSAR (CUHK14206320), NSERC (611370), National Engineering Laboratory for Big Data System Computing Technology, and Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ).


Bibtex

@article{DronePath21,

title={Continuous Aerial Path Planning for 3D Urban Scene Reconstruction},

author={Han Zhang and Yucong Yao and Ke Xie and Chi-Wing Fu and Hao Zhang and Hui Huang},

journal={ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA)},

volume={40},

number={6},

pages={225:1--225:15},

year={2021},

}



Downloads (faster for people in China)

Downloads (faster for people in other places)