Autoscanning for Coupled Scene Reconstruction and Proactive Object Analysis

ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2015)

Kai Xu1,2        Hui Huang1*           Yifei Shi2          Hao Li3        Pinxin Long1        Jianong Caichen2,1         Wei Sun1        Baoquan Chen3*

1SIAT           2National University of Defense Technology            3Shandong University

Figure 1: Autonomous scene scanning and reconstruction with object analysis aided by robot pushing. Left: A PR-2 robot with one arm equipped with a depth camera interacts with a cluttered table-top scene. Middle: The reconstructed scene with extracted individual objects shown with distinct colors. Right: Zoomed-in views.

Detailed scanning of indoor scenes is tedious for humans. We propose autonomous scene scanning by a robot to relieve humans from such a laborious task. In an autonomous setting, detailed scene acquisition is inevitably coupled with scene analysis at the required level of detail. We develop a framework for object-level scene reconstruction coupled with object-centric scene analysis. As a result, the autoscanning and reconstruction will be object-aware, guided by the object analysis. The analysis is, in turn, gradually improved with progressively increased object-wise data fidelity. In realizing such a framework, we drive the robot to execute an iterative analyze-and-validate algorithm which interleaves between object analysis and guided validations. 

The object analysis incorporates online learning into a robust graphcut based segmentation framework, achieving a global update of object-level segmentation based on the knowledge gained from robot-operated local validation. Based on the current analysis, the robot performs proactive validation over the scene with physical push and scan refinement, aiming at reducing the uncertainty of both object-level segmentation and object-wise reconstruction. We propose a joint entropy to measure such uncertainty based on segmentation confidence and reconstruction quality, and formulate the selection of validation actions as a maximum information gain problem. The output of our system is a reconstructed scene with both object extraction and object-wise geometry fidelity.

Figure 2: An overview of the syste

Figure 3: Plot of push information gain for all points (c) scanned for the input scene (a), based on the segmentation in (b). In (d), physically infeasible pushing points are filtered out, by zeroing their information gain values. The Next Besh Pushes (NBPs) can be selected from the remaining non-zero points.

Figure 4: A gallery of object-aware reconstruction results for five real-life indoor scenes, including a meeting room (a), a kitchen (b), a computer lab (c), a cafe (d), and a three-room apartment (e). Our system extracts 18, 19, 41, 93 and 60 objects for these scenes, respectively, while achieving object-wise quality reconstruction. The reconstruction of large scale room structure (e.g., ground, walls and large furnitures) is incomplete, which is not the focus of this work. Note in the kitchen of the apartment (the second zoom-in view in (e)), the hanging kitchen wares are extracted by segmentation without physical validation.

Download and Reference  

We provide the C++ code of object-level segmentation and entropy estimation used in our system.

Source code (ZIP, 116MB)
[To reference our code or data in a publication, please include the bibtex below and a link to this website.] 


We thank all the reviewers for their valuable comments and constructive suggestions. We are especially grateful to Hao Zhang and Daniel Cohen-Or for their help and discussion. We would also like to acknowledge our research grants: NSFC (61232011, 61572507, 61202333, 61272327), National 973 Program (2015CB352500, 2015CB352501), Guangdong Science and Technology Program (2015A030312015, 2014B050502009, 2014TX01X033), Shenzhen VisuCA Key Lab (CXB201104220029A), The Fundamental Research Funds of Shandong University, and SIAT Innovation Program for Excellent Young Researchers (201402).


@article {xu_siga15,
    title = {Autoscanning for Coupled Scene Reconstruction and Proactive Object Analysis},
    author = {Kai Xu and Hui Huang and Yifei Shi and Hao Li and Pinxin Long and Jiannong Caichen and Wei Sun and Baoquan Chen},
    journal = {ACM Transactions on Graphics (Proc. of SIGGRAPH Asia 2015)},
    volume = {34},
    number = {6},
    pages = {177:1--177:14},
    year = {2015}

Downloads (faster for people in China)

Downloads (faster for people in other places)