Aerial Path Planning for Online Real-Time Exploration and Offline High-Quality Reconstruction of Large-Scale Urban Scenes

ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2021)

Yilin Liu1    Ruiqi Cui1    Ke Xie1    Minglun Gong2    Hui Huang1*

1Shenzhen University    2University of Guelph

Fig. 1. Given an unknown large-scale urban site with arbitrary boundary shape (red polygon shown on left), our algorithm designs an aerial flight trajectory in real-time, which guides a single-camera UAV to both explore the site (trajectory shown in yellow) and observe all buildings (trajectory shown in blue). 9,148 images were captured, which support high-quality reconstruction of this 1.35km2 area through available multi-view stereo matching techniques (right).


Existing approaches have shown that, through carefully planning flight trajectories, images captured by Unmanned Aerial Vehicles (UAVs) can be used to reconstruct high-quality 3D models for real environments. These approaches greatly simplify and cut the cost of large-scale urban scene reconstruction. However, to properly capture height discontinuities in urban scenes, all state-of-the-art methods require prior knowledge on scene geometry and hence, additional prepossessing steps are needed before performing the actual image acquisition flights. To address this limitation and to make urban modeling techniques even more accessible, we present a real-time explore-and-reconstruct planning algorithm that does not require any prior knowledge for the scenes. Using only captured 2D images, we estimate 3D bounding boxes for buildings on-the-fly and use them to guide online path planning for both scene exploration and building observation. Experimental results demonstrate that the aerial paths planned by our algorithm in real-time for unknown environments support reconstructing 3D models with comparable qualities and lead to shorter flight air time.

Fig. 2. Overview: Taking the live feed video from a UAV on an appointed site as input, our system detects buildings and estimates their bounding boxes (e.g., the first model in the top row), which guide the UAV to further explore the site (yellow trajectory) and observe buildings from different perspectives (blue trajectory). Additional observations further enhance our knowledge on this site, allowing more buildings being detected and modeled as boxes (remaining models in the top row). At the end of the flight, the captured image sequence (middle) is used to build a high-quality model for the whole scene (bottom).

Fig. 4. The proposed Region-Division method for global scene exploration. a): At the beginning, the UAV only knows the boundary of the whole area. A (green) region is defined using a default size ∈ for the UAV to explore. During the exploration, the UAV discovers target building 1, 2, 3 through the perception distance of two cells by default. It switches to reconstruction mode once it arrives at the cells of building 1, 2, but leaves building 3 for the latter processing since it locates outside the green region. b): The target building 3 will guide the formation of the next (blue) region. c): Since no new building is detected during the exploration of the blue region, the next (yellow) region to be explored is defined using size ∈ again. Buildings 4, 5 detected and reconstructed during the exploration of the yellow region. d): The final path generated by Region-Division divides the whole site into four non-overlapping polygon-shaped regions. It ensures all cells are explored and all buildings are observed. The total path length (22,114) is shorter than the two baselines (Greedy-Nearest: 23,852 and Two-Step: 25,170); see Supplementary for further details.

Fig. 5. Local path planning for target building reconstruction. A set of simple rules are used so that the planning can be done in real-time and adapt to new observations. The trajectories are initialized based on the shape of the building’s bounding box (left) and deformed when the bounding box changes or potential collusion is detected (right).

Fig. 8. We show on the left a real scene referred to as Academic Building. As illustrated, a single 3D orientated bounding box cannot fit the target building tightly and thus may lose many important geometry details especially in the concave areas. On the right, we show an ablation study that uses different trajectories to observe the buildings: a) 80% overlap trajectory with split strategy nicely covers all areas of the buildings and hence produces the best model; b) 80% overlap trajectory without split yields shortest flight path, but the samples are not enough for accurate reconstruction; and c) 60% overlap trajectory with split strategy has similar path length as b), but noticeable better reconstruction result.

Fig. 11. Comparison on flight trajectories and reconstruction results on Academic Building. Our approach achieve similar reconstruction quality as the offline optimization-based approach [Zhou et al. 2020], while using much smoother flight trajectories. Note that the trajectory of Oblique Photography covers a larger area to ensure the sufficient image overlap on buildings near the boundary. This is a default behavior in Dji-Terra.

Fig. 14. Visualizing the trajectories designed by the two baseline approaches and the proposed Region-Division method. Our approach decomposes the scene into non-overlapping regions, which can be handled independently.

Fig. 16. Visual comparison with Oblique Photography on a real large-scale Campus scene. Our approach decompose this large (1.35km2) and irregular-shaped site into multiple non-overlapping regions (Top-Left). The density of the flight trajectory highly depends on the locations of buildings. The finally reconstructed models are noticeably more detailed than those generated by Oblique Photography.


We thank all reviewers for their valuable comments. This work was supported in parts by NSFC (U2001206), Guangdong Talent Program (2019JC05X328), Guangdong Science and Technology Program (2020A0505100064), DEGP Key Project (2018KZDXM058), Shenzhen Science and Technology Program (RCJC20200714114435012, JCYJ20210324120213036), NSERC (293127), National Engineering Laboratory for Big Data System Computing Technology, and Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ).



title={Aerial Path Planning for Online Real-Time Exploration and Offline High-Quality Reconstruction of Large-Scale Urban Scenes},

author={Yilin Liu and Ruiqi Cui and Ke Xie and Minglun Gong and Hui Huang},

journal={ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA)},






Downloads (faster for people in China)

Downloads (faster for people in other places)