ACM SIGGRAPH / CHI
Yujia Wang, Wenguan Wang, Wei Liang, Lap-Fai Yu
Comic-Guided Speech Synthesis.
ACM Transactions on Graphics (Proceeding of SIGGRAPH Asia 2019)
Project Page, Paper, Video, Bibtex
Automatic narration of characters in comics, mangas, etc. Conditionally accepted, more details coming soon.
Rawan Alghofaili, Yasuhito Sawahata, Haikun Huang, Hsueh-Cheng Wang, Takaaki Shiratori, Lap-Fai Yu
Lost in Style: Gaze-driven Adaptive Aid for VR Navigation.
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2019)
Project Page, Paper, Video, Bibtex
Feel lost when navigating in a virtual environment (such as when playing a VR game)? This tool helps you get back on track by predicting when you need navigation help and popping up wayfinding aids.
*Yongqi Zhang, *Biao Xie, Haikun Huang, Elisa Ogawa, Tongjian You, Lap-Fai Yu
*Equal contributors
Pose-Guided Level Design.
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2019)
Best Paper Honorable Mention Award
Project Page, Paper, Video, Bibtex
Procedural game level generator for motion/exercise games (e.g., Just Dance, Speed of Light).
*Haikun Huang, *Michael S. Solah, Dingzeyu Li, Lap-Fai Yu
*Equal contributors
Audible Panorama: Automatic Spatial Audio Generation for Panorama Imagery.
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2019)
Project Page, Paper, Video, Bibtex
This work automatically synthesizes realistic spatial sounds for 360° panorama images for delivering immersive VR experiences
Noah Duncan, Lap-Fai Yu, Sai-Kit Yeung, Demetri Terzopoulos
Approximate Dissections.
ACM Transactions on Graphics (Proceeding of SIGGRAPH Asia 2017)
Project Page, Paper, Video, Bibtex
Inspired by tangrams, our approach can automatically generate pieces that can be reassembled into different interesting shapes.
Noah Duncan, Lap-Fai Yu, Sai-Kit Yeung
Interchangeable Components for Hands-On Assembly Based Modelling.
ACM Transactions on Graphics (Proceeding of SIGGRAPH Asia 2016)
Featured in SIGGRAPH Asia 2016 technical papers video trailer
Project Page, Paper (High Res), Paper (Low Res), Video, Fast Forward, Bibtex
This paper may be regarded as a co-fabrication approach for generating a set of seamlessly interchangeable components from 3D Models. It can be applied for toy and product design. From a small set of components generated by our approach, you can assemble thousands of different objects by hand. Have fun!
Tian Feng, Lap-Fai Yu, Sai-Kit Yeung, KangKang Yin, Kun Zhou
Crowd-driven Mid-scale Layout Design.
ACM Transactions on Graphics (Proceeding of SIGGRAPH 2016)
Project Page, Paper (High Res), Paper (Low Res), Video, Bibtex
This paper proposes to use crowd simulations to procedurally generate crowd-aware layout designs.
Noah Duncan, Lap-Fai Yu, Sai-Kit Yeung, Demetri Terzopoulos
Zoomorphic Design.
ACM Transactions on Graphics (Proceeding of SIGGRAPH 2015)
Project Page, Paper, Video, Bibtex
This paper describes a computational approach to generate some cool animal-inspired designs.
Lap-Fai Yu, Sai-Kit Yeung, Demetri Terzopoulos, Tony F. Chan
DressUp! Outfit Synthesis through Automatic Optimization.
ACM Transactions on Graphics (Proceeding of SIGGRAPH Asia 2012)
Featured in SIGGRAPH Asia 2012 technical papers video trailer
Project Page, Paper (High Res), Paper (Low Res), Bibtex
In addition to dressing someone up nicely, one can also apply the technique to avoid having repeated guys in video game scenes, e.g. GTA   
Lap-Fai Yu, Sai-Kit Yeung, Chi-Keung Tang, Demetri Terzopoulos,
Tony F. Chan, Stanley J. Osher
Make it Home: Automatic Optimization of Furniture Arrangement.
ACM Transactions on Graphics (Proceeding of SIGGRAPH 2011)
Project Page, Paper, Bibtex
This idea is rooted from the old days when I was tired of the monotonous interior layouts of role-playing game worlds. Coincidentally, our goal fitted perfectly well with SIGGRAPH 2011's slogan, Make it home.
Conferences
Yining Lang, Wei Liang, Yujia Wang, Lap-Fai Yu
3D Face Synthesis Driven by Personality Impression.
AAAI Conference on Artificial Intelligence (AAAI 2019)
Project Page, Paper, Video, Bibtex
Our approach automatically modifies a 3D face to give different personality impressions.
Yung-Shan Su, Shao-Huang Lu, Po Sheng Ser, Wei-Ting Hsu, Wei-Cheng Lai, Biao Xie, Hong-Ming Huang, Teng-Yok Lee, Hung-Wen Chen, Lap-Fai Yu, Hsueh-Cheng Wang
Pose-Aware Placing with Semantic Labels - Brandname-Based Affordance Prediction and Cooperative Dual-Arm Active Manipulation.
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019)
Paper, Project Page Video, Bibtex
Training a robotic arm to manipulate and place 3D objects with brand name labels. Conditionally accepted. More details coming soon.
Qianwen Chao, Xiaogang Jin, Hen-Wei Huang, Shaohui Foong, Lap-Fai Yu, Sai-Kit Yeung
Force-based Heterogeneous Traffic Simulation for Autonomous Vehicle Testing.
IEEE International Conference on Robotics and Automation (ICRA 2019)
Paper, Video, Bibtex
A force-based, heterogeneous traffic simulator that models the interactions between different road users in a simple and unified way.
Rawan Alghofaili, Michael S. Solah, Haikun Huang, Yasuhito Sawahata, Marc Pomplun, Lap-Fai Yu
Optimizing Visual Element Placement via Visual Attention Analysis.
IEEE Virtual Reality (VR 2019)
Project Page, Paper, Video, Bibtex
Populating visual elements (e.g., ads) in 3D virtual environments based on a visual attention predictor trained by eye-gaze data.
Yining Lang, Wei Liang, Lap-Fai Yu
Virtual Agent Positioning Driven by Scene Semantics in Mixed Reality.
IEEE Virtual Reality (VR 2019)
Project Page, Paper, Video, Bibtex
Positioning virtual agents in mixed reality by considering scene semantics and user interaction tasks.
Tian Feng, Quang-Trung Truong, Thanh Nguyen, Jing Yu Koh, Lap-Fai Yu, Alexander Binder, Sai-Kit Yeung
Urban Zoning Using Higher-Order Markov Random Fields on Multi-View Imagery Data.
European Conference on Computer Vision (ECCV 2018)
Paper, Bibtex
Automatic urban zoning using HO-MRF built on multi-view imagery data including street-view photos and top-view satellite images.
Tzu-Kuan Chuang, Ni-Ching Lin, Jih-Shi Chen, Chen-Hao Hung, Yi-Wei Huang, Chunchih Teng, Haikun Huang, Lap-Fai Yu, Laura Giarre, Hsueh-Cheng Wang
Deep Trail-Following Robotic Guide Dog in Pedestrian Environments for People who are Blind and Visually Impaired - Learning from Virtual and Real Worlds.
IEEE International Conference on Robotics and Automation (ICRA 2018)
Project Page, Paper, Video, Bibtex
Training a robotic guidedog that can help blind people to navigate in the real world
Yining Lang, Wei Liang, Fang Xu, Yibiao Zhao, Lap-Fai Yu
Synthesizing Personalized Training Programs for Improving Driving Habits via Virtual Reality.
IEEE Virtual Reality (VR 2018)
Project Page, Paper, Video, Bibtex
Synthesizing personalized driving training programs for VR-based training.
Hanqing Wang, Wei Liang, Lap-Fai Yu
Transferring Objects: Joint Inference of Container and Human Pose.
IEEE International Conference on Computer Vision (ICCV 2017)
Project Page, Paper, Video, Bibtex
Our approach jointly infers the human pose and the container for performing a transfer task.
Carla Aravena*, Mark Vo*, Tao Gao, Takaaki Shiratori, Lap-Fai Yu
*Equal contributors
Perception Meets Examination: Studying Deceptive Behaviors in VR.
Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci 2017)
Project Page, Paper, Video, Bibtex
An interesting study on human deceptive behavior based on the Theory of Mind inspired by the game Classroom Aquatic.
Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, Sai-Kit Yeung
SceneNN: a Scene Meshes Dataset with aNNotations.
IEEE International Conference on 3D Vision (3DV 2016)
Oral presentation
Best Paper Honorable Mention
Project Website, Paper, Supplementary Material, Bibtex
We provide 100+ scene meshes with detailed annotations. The dataset will keep updating and expanding. Stay tuned on our project website.
Chen Li, Lap-Fai Yu, Zhichao Lu, Yasuyuki Matsushita, Kun Zhou, Stephen Lin
Continuous Symmetric Stereo with Adaptive Outlier Handling.
IEEE International Conference on 3D Vision (3DV 2015)
Paper, Supplementary Material, Bibtex
A stereo matching technique that provides a good combination of accuracy and speed, allowing one to generate high-quality reconstruction of outdoor images even with complex occlusions. More details coming soon.
Lap-Fai Yu, Noah Duncan, Sai-Kit Yeung
Fill and Transfer: A Simple Physics-based Approach for Containability Reasoning.
IEEE International Conference on Computer Vision (ICCV 2015)
Project Page, Paper, Supplementary Material, Video, Bibtex
We use simple physics to reason about the fill and transfer directions of containers. Many thanks to the anonymous chef for cooking our food slowly while Noah and I were baking this idea in a food court (and the food was yummy).
Lap-Fai Yu, Sai-Kit Yeung, Yu-Wing Tai, Stephen Lin
Shading-based Shape Refinement of RGB-D Images.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013)
Paper (High Res), Paper (Low Res), Video, Bibtex
I had much fun playing with the Microsoft Kinect. To me, this paper also serves as my travel log in 2012, as the RGB-D images were captured in different countries throughout my travel while the formulation was written mostly during my flight over the Pacific Ocean.
Lap-Fai Yu, Sai-Kit Yeung, Yu-Wing Tai, Demetri Terzopoulos, Tony F. Chan
Outdoor Photometric Stereo.
IEEE International Conference on Computational Photography (ICCP 2013)
Oral presentation
Paper (High Res), Paper (Low Res), Video, Data, Bibtex
This work brings photometric stereo out of the lab.
Journals
Wei Liang, Jingjing Liu, Yining Lang, Bing Ning, Lap-Fai Yu
Functional Workspace Optimization via Learning Personal Preferences from Virtual Experiences.
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2019
(Special Issue on IEEE Virtual Reality 2019)

Project Page, Paper, Video, Bibtex
By learning working habits of users from virtual environments, our approach generates personalized workspace designs automatically.
Yujia Wang, Wei Liang, Jianbing Shen, Yunde Jia, Lap-Fai Yu
A Deep Coarse-to-Fine Network for Head Pose Estimation from Synthetic Data.
Pattern Recognition (PR), 2019
Project Page, Paper, Video, Bibtex
Training a deep learning classifier using rendered head pictures to estimate head poses for applications such as VR/AR.
Lifeng Zhu, Benyi Xie, Yongjie Zhang, Lap-Fai Yu
Cartonist: Automatic Synthesis and Interactive Exploration of Nonstandard Carton Design.
Computer-Aided Design (CAD), 2019
Paper, Video, Bibtex
An interactive tool for exploring and creating interesting nonstandard carton designs.
Biao Xie*, Yongqi Zhang*, Haikun Huang, Elisa Ogawa, Tongjian You, Lap-Fai Yu
*Equal contributors
Exercise Intensity-driven Level Design.
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2018
(Special Issue on IEEE Virtual Reality 2018)

Featured on IEEE Xplore Innovation Spotlight
Project Page, Paper, Video, Bibtex
Our approach generates game levels that carry the right amount of exercise intensity one needs to meet his or her training goal. Have fun while exercising in virtual reality!
Chenfanfu Jiang, Siyuan Qi, Yixin Zhu, Siyuan Huang, Jenny Lin, Lap-Fai Yu, Demetri Terzopoulos, Song-Chun Zhu
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars.
International Journal of Computer Vision (IJCV), 2018
Paper, Video, Bibtex
By sampling stochastic scene grammars, different realistic scenes can be synthesized for purposes such as graphics for vision and virtual reality.
Tomer Weiss, Alan Litteneker, Noah Duncan, Masaki Nakada, Chenfanfu Jiang, Lap-Fai Yu, Demetri Terzopoulos
Fast and Scalable Position-Based Layout Synthesis.
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2018
Paper, Video, Bibtex
We present a fast, physics-motivated, continuous layout synthesis technique that can arrange hundreds of objects (e.g., chairs in a theater) in a few seconds.
Haikun Huang, Ni-Ching Lin, Lorenzo Barrett, Darian Springer, Hsueh-Cheng Wang, Marc Pomplun, Lap-Fai Yu
Automatic Optimization of Wayfinding Design.
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2017
Project Page, Paper, Supplementary, Video, Bibtex
An automatic tool to help wayfinding designers create better wayfinding designs for architectural spaces, and to help level designers create better game levels.
Changyang Li, Wei Liang, Chris Quigley, Yibiao Zhao, Lap-Fai Yu
Earthquake Safety Training through Virtual Drills.
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2017
(Special Issue on IEEE Virtual Reality 2017)

Project Page, Paper, Video, Bibtex
Experience the shake and learn to protect yourself via immersive virtual environments.
Mahdi Jampour, Chen Li, Lap-Fai Yu, Kun Zhou, Stephen Lin, Horst Bischof
Face Inpainting based on High-Level Facial Attributes.
Computer Vision and Image Understanding (CVIU), 2017
Project Page, Paper, Supplementary, Video, Bibtex
Combining high-level and low-level cues for inpainting occluded face images.
Mahdi Jampour, Mohammad-Shahram Moin, Lap-Fai Yu, Horst Bischof
Mapping Forests: a Comprehensive Approach for Non-linear Mapping Problems.
Journal of Mathematical Imaging and Vision (JMIV), 2017
Paper, Bibtex
A new and robust mapping approach based on regression transformations for computer vision applications.
Duc Thanh Nguyen, Binh-Son Hua, Lap-Fai Yu, Sai-Kit Yeung
A Robust 3D-2D Interactive Tool for Scene Segmentation and Annotation.
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2017
Project Page, Paper, Video, Bibtex
Details of our 3D scene annotation tool for annotating the SceneNN dataset.
Lap-Fai Yu, Sai-Kit Yeung, Demetri Terzopoulos
The Clutterpalette: An Interactive Tool for Detailing Indoor Scenes.
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2014
Project Page, Paper (High Res), Paper (Low Res), Video, Bibtex
Having trouble detailing your virtual worlds? The Clutterpalette helps you quickly add scene details by using the clutter object distributions learnt from real-world RGB-D scene data.
Workshops
Quang-Hieu Pham, Minh-Khoi Tran, Binh-Son Hua, Lap-Fai Yu, Duc Thanh Nguyen, Sai-Kit Yeung, et al.
SHREC'18: RGB-D Object-to-CAD Retrieval.
Eurographics Workshop on 3D Object Retrieval, 2018
Project Page, Paper, Bibtex
A report of our RGB-D Object-to-CAD retrieval contest.
Binh-Son Hua, Quang-Trung Truong, Minh-Khoi Tran, Quang-Hieu Pham, Lap-Fai Yu, Duc Thanh Nguyen, Sai-Kit Yeung, et al.
SHREC'17: RGB-D to CAD Retrieval with ObjectNN Dataset.
Eurographics Workshop on 3D Object Retrieval, 2017
Project Page, Paper, Bibtex
A report of our RGB-D object-to-CAD object retrieval contest.
Haikun Huang, Ni-Ching Lin, Lorenzo Barrett, Darian Springer, Hsueh-Cheng Wang, Marc Pomplun, Lap-Fai Yu
Analyzing Visual Attention via Virtual Environments.
Virtual Reality meets Physical Reality Workshop, SIGGRAPH Asia, 2016
Paper, Bibtex

Theses
Lap-Fai Yu
Data-driven Optimization for Modeling in Computer Graphics and Vision.
Ph.D. Dissertation, Computer Science Department, UCLA, 2013
Advisor: Demetri Terzopoulos
Lap-Fai Yu
3D Reconstruction and Synthesis of Facial Expressions using a Manifold Alignment Framework.
M.Phil. Thesis, Department of Computer Science and Engineering, HKUST, 2009
Advisor: Dit-Yan Yeung
My first attempt to apply data-driven techniques to computer graphics, in this case, to train a controller to synthesize 3D facial expression. The inspiration was from the dull faces I always saw in video games.
Patents
Lap-Fai Yu, Sai-Kit Yeung, Demetri Terzopoulos, Tony F. Chan
Clothing Matching System and Methods.
US patent application no. 61/725, 073
Lap-Fai Yu, Sai-Kit Yeung, Chi-Keung Tang, Demetri Terzopoulos, Tony F. Chan, Stanley J. Osher
Automatic Optimization of Scene Configuration.
US patent application no. 61/675, 251