Lightning Artist Toolkit: A Hand-Drawn Volumetric Animation Pipeline

Abstract

This research contributes a set of methods for freely integrating live-action volumetric video with hand-drawn volumetric animation. The Kinect, the first consumer depth camera, arrived in 2010; in 2016, the HTC Vive headset introduced the first mass-market 6DoF controllers. Combined, these two advances unlocked a new approach to creating frame-by-frame animation with 6DoF drawing tools, which my research has developed as the Lightning Artist Toolkit—a complete pipeline for hand-drawn volumetric animation; at the time of writing, the only open-source example of its kind. The goal of the project is to make creation in 3D as expressive and intuitive as creation in 2D, by retaining the human gesture from its origins in hand-drawn animation on paper. Importing and manipulating scanned photographic images alongside handmade drawings has been a core feature of 2D image editing and animation tools for over fifty years. Initially, applying raster editing capabilities to real-world animation production was impractical—so the earliest hand-drawn computer-animated short films used 2D vector strokes. Today, operating naïvely on 3D voxels similarly requires excessive computational resources to be scaled up for even a few minutes of high-resolution footage, and working with 3D vector graphics representations offers a promising solution. At this project’s core is a collection of applied machine learning systems that transform live-action volumetric video into a sequence of volumetric brushstroke vectors. Integrated into a conventional animation workflow, this is suitable for the practical production of hand-drawn 3D animated short films in an XR drawing system. The contribution is less a computer vision challenge with an objective goal, as with for example point cloud segmentation, than it is an attempt to approximate the aesthetics of human vision—to generate a collection of brushstrokes from a point cloud that resembles what an artist might draw from scratch in XR, in imitation of a drawing process that records as markings the information from a scene that was subjectively important to an individual artist. In addition to supporting animation production through this workflow, this project also contributes a large public dataset of 3D drawings that may be usable in new and unexpected ways.