SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image

ACM Symposium on User Interface Software and Technology (UIST 2017)

Yuwei Li1        Xi Luo1        Youyi Zheng2,1*         Pengfei Xu3        Hongbo Fu4

1ShanghaiTech Univ.         2State Key Lab of CAD&CG, Zhejiang Univ.         3Shenzhen Univ.         4City Univ. of Hong Kong

Figure 1:Our SweepCanvas system allows artists to quickly create 3D prototype models using a sketch-based interface on top of an RGB-D image as context. The left and right examples were created within 30 and 60 minutes, respectively, by a user with a short training period. Left and middle right are the original images overlaid with 3D models.


The creation of 3D contents still remains one of the most crucial problems for the emerging applications such as 3D printing and Augmented Reality. In Augmented Reality, how to create virtual contents that seamlessly overlay with the real environment is a key problem for human-computer interaction and many subsequent applications. In this paper, we present a sketch-based interactive tool, which we term Sweep-Canvas, for rapid exploratory 3D modeling on top of an RGBD image. Our aim is to offer end-users a simple yet efficient way to quickly create 3D models on an image. We develop a novel sketch-based modeling interface, which takes a pair of user strokes as input and instantly generates a curved 3D surface by sweeping one stroke along the other. A key enabler of our system is an optimization procedure that extracts pairs of spatial planes from the context to position and sweep the strokes. We demonstrate the effectiveness and power of our modeling system on various RGB-D data sets and validate the use cases via a pilot study.

Figure 2:Our user interface consists of two main panels: a sketching panel where the user sketches on top of an RGB-D image to create scanvases and a candidate panel where best-ranked candidate s-canvases are listed. The red and blue polygons are the supporting planes for the profile and trajectory strokes, respectively.

Figure 3:Various 3D conceptual models created using our system. Our system is capable of quickly anchoring desired s-canvases on top of the given contexts. Most of the examples were sketched within 20 minutes.

We thank all reviewers for their valuable comments. This work was supported in part by the National Natural Science Foundation of China NO. 61502306, NO. 61602310, the China Young 1000 Talents Program, Shenzhen Innovation Program NO. JCYJ20170302154106666, and the grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. CityU11300615, CityU11204014, CityU113513).


    title = {SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image},
    author = {Yuwei Li and Xi Luo and Youyi Zheng and Pengfei Xu and Hongbo Fu},
    journal = {ACM Symposium on User Interface Software and Technology (Proceedings of UIST 2017)},
    pages = {387-399},
    year = {2017},

Downloads (faster for people in China)

Downloads (faster for people in other places)