映维网(Paper) https://paper.yivian.com 影响力虚拟现实(VR)、增强现实(AR)产业信息数据平台 Tue, 10 Dec 2019 02:32:06 +0000 zh-CN hourly 1 https://wordpress.org/?v=4.8.11 https://paper.yivian.com/wp-content/uploads/2019/11/cropped-YV-400-150x150.png 映维网(Paper) https://paper.yivian.com 32 32 Meet the AI Choreographer: This New Model Can Help You With Your Next Dance Video https://paper.yivian.com/120 Tue, 10 Dec 2019 02:28:38 +0000 https://paper.yivian.com/120 Title: Meet the AI Choreographer: This New Model Can Help You With Your Next Dance VideoTeams: University of California, NVIDIAWriters: Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan KautzPubDate: December, 2019Project: Dancing2MusicAbstractDancing to music is an instinctive move by humans. Learning to model the music-to-dance generation process is, however, a challenging problem.

Meet the AI Choreographer: This New Model Can Help You With Your Next Dance Video最先出现在映维网(Paper)

]]>

Title: Meet the AI Choreographer: This New Model Can Help You With Your Next Dance Video

Teams: University of California, NVIDIA

Writers: Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz

PubDate: December, 2019

Project: Dancing2Music

Abstract

Dancing to music is an instinctive move by humans. Learning to model the music-to-dance generation process is, however, a challenging problem. It requires significant efforts to measure the correlation between music and dance as one needs to simultaneously consider multiple aspects, such as style and beat of both music and dance. Additionally, dance is inherently multimodal and various following movements of a pose at any moment are equally likely. In this paper, we propose a synthesis-by-analysis learning framework to generate dance from music. In the analysis phase, we decompose a dance into a series of basic dance units, through which the model learns how to move. In the synthesis phase, the model learns how to compose a dance by organizing multiple basic dancing movements seamlessly according to the input music. Experimental qualitative and quantitative results demonstrate that the proposed method can synthesize realistic, diverse, style-consistent, and beat-matching dances from music.

Meet the AI Choreographer: This New Model Can Help You With Your Next Dance Video最先出现在映维网(Paper)

]]>
An Integrated 6DoF Video Camera and System Design https://paper.yivian.com/119 Wed, 04 Dec 2019 02:01:28 +0000 https://paper.yivian.com/119 ...

An Integrated 6DoF Video Camera and System Design最先出现在映维网(Paper)

]]>

Title: An Integrated 6DoF Video Camera and System Design

Teams: Facebook, Red Digital Cinema

Writers: Albert Parra Pozo, Michael Toksvig, Terry Filiba Schrager, Joyce Hsu, Uday Mathur, Alexander Sorkine-Hornung, Richard Szeliski, Brian Cabral

PubDate: November , 2019

ProjectFacebook360 Depth Estimation Pipeline

Abstract

Designing a fully integrated 360◦ video camera supporting 6DoF head motion parallax requires overcoming many technical hurdles, including camera placement, optical design, sensor resolution, system calibration, real-time video capture, depth reconstruction, and real-time novel view synthesis. While there is a large body of work describing various system components, such as multi-view depth estimation, our paper is the first to describe a complete, reproducible system that considers the challenges arising when designing, building, and deploying a full end-to-end 6DoF video camera and playback environment. Our system includes a computational imaging software pipeline supporting online markerless calibration, high-quality reconstruction, and real-time streaming and rendering. Most of our exposition is based on a professional 16-camera configuration, which will be commercially available to film producers. However, our software pipeline is generic and can handle a variety of camera geometries and configurations. The entire calibration and reconstruction software pipeline along with example datasets is open sourced to encourage follow-up research in high-quality 6DoF video reconstruction and rendering.

An Integrated 6DoF Video Camera and System Design最先出现在映维网(Paper)

]]>
Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera https://paper.yivian.com/116 Tue, 03 Dec 2019 07:55:17 +0000 https://paper.yivian.com/116 Title: Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth CameraTeams: Max Planck Institute for Informatics (GVV Group), Saarland Informatics Campus , Universidad Rey Juan CarlosWriters: Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A.

Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera最先出现在映维网(Paper)

]]>

Title: Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera

Teams: Max Planck Institute for Informatics (GVV Group), Saarland Informatics Campus , Universidad Rey Juan Carlos

Writers: Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A. Otaduy, Dan Casas, Christian Theobalt

Project: TwoHands

Publication Date: July, 2019

Abstract

We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user’s hand shape. In order to achieve this, we embed a recent parametric hand pose and shape model and a dense correspondence predictor based on a deep neural network into a suitable energy minimization framework. For training the correspondence prediction network, we synthesize a two-hand dataset based on physical simulations that includes both hand pose and shape annotations while at the same time avoiding inter-hand penetrations. To achieve real-time rates, we phrase the model fitting in terms of a nonlinear least-squares problem so that the energy can be optimized based on a highly efficient GPU-based Gauss-Newton optimizer. We show state-of-the-art results in scenes that exceed the complexity level demonstrated by previous work, including tight two-hand grasps, significant inter-hand occlusions, and gesture interaction.

Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera最先出现在映维网(Paper)

]]>
FaceDrive: Facial Expression Driven Operation to Control Virtual Supernumerary Robotic Arms https://paper.yivian.com/109 Mon, 02 Dec 2019 03:09:11 +0000 https://paper.yivian.com/109 Title: FaceDrive: Facial Expression Driven Operation to Control Virtual Supernumerary Robotic ArmsTeams: 慶應義塾大学 理工学部Writers: Masaaki Fukuoka, Adrien Verhulst, Fumihiko Nakamura, Ryo Takizawa, Katsutoshi Masai, Maki SugimotoPublication Date: September, 2019AbstractSupernumerary Robotic Arms (SRAs) can make physical activities easier, but require cooperation with the operator. To improve cooperation, we predict the operator’s intentions by using his/her Facial Expressions (FEs).

FaceDrive: Facial Expression Driven Operation to Control Virtual Supernumerary Robotic Arms最先出现在映维网(Paper)

]]>

Title: FaceDrive: Facial Expression Driven Operation to Control Virtual Supernumerary Robotic Arms

Teams: 慶應義塾大学 理工学部

Writers: Masaaki Fukuoka, Adrien Verhulst, Fumihiko Nakamura, Ryo Takizawa, Katsutoshi Masai, Maki Sugimoto

Publication Date: September, 2019

Abstract

Supernumerary Robotic Arms (SRAs) can make physical activities easier, but require cooperation with the operator. To improve cooperation, we predict the operator’s intentions by using his/her Facial Expressions (FEs). We propose to map FEs to SRAs commands (e.g. grab, release). To measure FEs, we used a optical sensor-based approach (here inside a HMD), the sensors data are fed to a SVM classifying them in FEs. The SRAs can then carry out commands by predicting the operator’s FEs (and arguably, the operator’s intention). We made a Virtual reality Environment (VE) with SRAs and synchronizable avatar to investigate the most suitable mapping between FEs and SRAs. In SIGGRAPH Asia 2019, the user can manipulate virtual SRAs using his/her FEs.

FaceDrive: Facial Expression Driven Operation to Control Virtual Supernumerary Robotic Arms最先出现在映维网(Paper)

]]>
Recycling a Landmark Dataset for Real-time Face Tracking with Low Cost HMD Integrated Cameras https://paper.yivian.com/106 Mon, 02 Dec 2019 02:03:36 +0000 https://paper.yivian.com/106 Title: Recycling a Landmark Dataset for Real-time Face Tracking with Low Cost HMD Integrated CamerasTeams: Disney ResearchWriters: Kenny MitchellPublication Date: November 2019AbstractPreparing datasets for use in the training of real-time face tracking algorithms for HMDs is costly. Manually annotated facial landmarks are accessible for regular photography datasets, but introspectively mounted cameras for VR face tracking have incompatible requirements with these existing datasets.

Recycling a Landmark Dataset for Real-time Face Tracking with Low Cost HMD Integrated Cameras最先出现在映维网(Paper)

]]>

Title: Recycling a Landmark Dataset for Real-time Face Tracking with Low Cost HMD Integrated Cameras

Teams: Disney Research

Writers: Kenny Mitchell

Publication Date: November 2019

Abstract

Preparing datasets for use in the training of real-time face tracking algorithms for HMDs is costly. Manually annotated facial landmarks are accessible for regular photography datasets, but introspectively mounted cameras for VR face tracking have incompatible requirements with these existing datasets. Such requirements include operating ergonomically at close range with wide angle lenses, low-latency short exposures, and near infrared sensors. In order to train a suitable face solver without the costs of producing new training data, we automatically repurpose an existing landmark dataset to these specialist HMD camera intrinsics with a radial warp reprojection. Our method separates training into local regions of the source photos, \ie mouth and eyes for more accurate local correspondence to the mounted camera locations underneath and inside the fully functioning HMD. We combine per-camera solved landmarks to yield a live animated avatar driven from the user’s face expressions. Critical robustness is achieved with measures for mouth region segmentation, blink detection and pupil tracking. We quantify results against the unprocessed training dataset and provide empirical comparisons with commercial face trackers.

Recycling a Landmark Dataset for Real-time Face Tracking with Low Cost HMD Integrated Cameras最先出现在映维网(Paper)

]]>
Study of 3D Virtual Reality Picture Quality https://paper.yivian.com/104 Fri, 29 Nov 2019 04:34:48 +0000 https://paper.yivian.com/104 Title: Study of 3D Virtual Reality Picture QualityTeams: University of Texas, Facebook Reality LabsWriters: Meixu Chen, Yize Jin, Todd Goodall, Xiangxu Yu, Alan C. BovikPublication Date: October, 2019AbstractVirtual Reality (VR) and its applications have attracted significant and increasing attention.

Study of 3D Virtual Reality Picture Quality最先出现在映维网(Paper)

]]>

Title: Study of 3D Virtual Reality Picture Quality

Teams: University of Texas, Facebook Reality Labs

Writers: Meixu Chen, Yize Jin, Todd Goodall, Xiangxu Yu, Alan C. Bovik

Publication Date: October, 2019

Abstract

Virtual Reality (VR) and its applications have attracted significant and increasing attention. However, the requirements of much larger file sizes, different storage formats, and immersive viewing conditions pose significant challenges to the goals of acquiring, transmitting, compressing and displaying high quality VR content. Towards meeting these challenges, it is important to be able to understand the distortions that arise and that can affect the perceived quality of displayed VR content. It is also important to develop ways to automatically predict VR picture quality. Meeting these challenges requires basic tools in the form of large, representative subjective VR quality databases on which VR quality models can be developed and which can be used to benchmark VR quality prediction algorithms. Towards making progress in this direction, here we present the results of an immersive 3D subjective image quality assessment study. In the study, 450 distorted images obtained from 15 pristine 3D VR images modified by 6 types of distortion of varying severities were evaluated by 42 subjects in a controlled VR setting. Both the subject ratings as well as eye tracking data were recorded and made available as part of the new database, in hopes that the relationships between gaze direction and perceived quality might be better understood. We also evaluated several publicly available IQA models on the new database, and also report a statistical evaluation of the performances of the compared IQA models.

Study of 3D Virtual Reality Picture Quality最先出现在映维网(Paper)

]]>
Aero-plane: A Handheld Force-Feedback Device that Renders Weight Motion Illusion on a Virtual 2D Plane https://paper.yivian.com/102 Fri, 29 Nov 2019 04:18:07 +0000 https://paper.yivian.com/102 Title: Aero-plane: A Handheld Force-Feedback Device that Renders Weight Motion Illusion on a Virtual 2D PlaneTeams: Korea Advanced Institute of Science and Technology, University of Chicago, Dartmouth CollegeWriters: Seungwoo Je, Myung Jin Kim, Woojin Lee, Byungjoo Lee, Andrea Bianchi, Xing-Dong Yang, Pedro Lopes Publication Date: October, 2019AbstractForce feedback is said to be the next frontier in virtual reality (VR).

Aero-plane: A Handheld Force-Feedback Device that Renders Weight Motion Illusion on a Virtual 2D Plane最先出现在映维网(Paper)

]]>

Title: Aero-plane: A Handheld Force-Feedback Device that Renders Weight Motion Illusion on a Virtual 2D Plane

Teams: Korea Advanced Institute of Science and Technology, University of Chicago, Dartmouth College

Writers: Seungwoo Je, Myung Jin Kim, Woojin Lee, Byungjoo Lee, Andrea Bianchi, Xing-Dong Yang, Pedro Lopes

Publication Date: October, 2019

Abstract

Force feedback is said to be the next frontier in virtual reality (VR). Recently, with consumers pushing forward with untethered VR, researchers turned away from solutions based on bulky hardware (e.g., exoskeletons and robotic arms) and started exploring smaller portable or wearable devices. However, when it comes to rendering inertial forces, such as when moving a heavy object around or when interacting with objects with unique mass properties, current ungrounded force feedback devices are unable to provide quick weight shifting sensations that can realistically simulate weight changes over 2D surfaces. In this paper we introduce Aero-plane, a force-feedback handheld controller based on two miniature jet propellers that can render shifting weights of up to 14 N within 0.3 seconds. Through two user studies we: (1) characterize the users’ ability to perceive and correctly recognize different motion paths on a virtual plane while using our device; and, (2) tested the level of realism and immersion of the controller when used in two VR applications (a rolling ball on a plane, and using kitchen tools of different shapes and sizes). Lastly, we present a set of applications that further explore different usage cases and alternative form-factors for our device.

Aero-plane: A Handheld Force-Feedback Device that Renders Weight Motion Illusion on a Virtual 2D Plane最先出现在映维网(Paper)

]]>
Fast Depth Densification for Occlusion-aware Augmented Reality https://paper.yivian.com/99 Thu, 28 Nov 2019 13:06:14 +0000 https://paper.yivian.com/99 Title: Fast Depth Densification for Occlusion-aware Augmented RealityTeams: Facebook AI, University of WashingtonWriters: Aleksander Holynski, Johannes KopfPublication date: November 2018AbstractCurrent AR systems only track sparse geometric features but do not compute depth for all pixels. For this reason, most AR effects are pure overlays that can never be occluded by real objects. We present a novel algorithm that propagates sparse depth to every pixel in near realtime.

Fast Depth Densification for Occlusion-aware Augmented Reality最先出现在映维网(Paper)

]]>

Title: Fast Depth Densification for Occlusion-aware Augmented Reality

Teams: Facebook AI, University of Washington

Writers: Aleksander Holynski, Johannes Kopf

Publication date: November 2018

Abstract

Current AR systems only track sparse geometric features but do not compute depth for all pixels. For this reason, most AR effects are pure overlays that can never be occluded by real objects. We present a novel algorithm that propagates sparse depth to every pixel in near realtime. The produced depth maps are spatio-temporally smooth but exhibit sharp discontinuities at depth edges. This enables AR effects that can fully interact with and be occluded by the real scene. Our algorithm uses a video and a sparse SLAM reconstruction as input. It starts by estimating soft depth edges from the gradient of optical flow fields. Because optical flow is unreliable near occlusions we compute forward and backward flow fields and fuse the resulting depth edges using a novel reliability measure. We then localize the depth edges by thinning and aligning them with image edges. Finally, we optimize the propagated depth smoothly but encourage discontinuities at the recovered depth edges. We present results for numerous real-world examples and demonstrate the effectiveness for several occlusion-aware AR video effects. To quantitatively evaluate our algorithm we characterize the properties that make depth maps desirable for AR applications, and present novel evaluation metrics that capture how well these are satisfied. Our results compare favorably to a set of competitive baseline algorithms in this context.

Fast Depth Densification for Occlusion-aware Augmented Reality最先出现在映维网(Paper)

]]>