Interactive Video Cutout

Jue Wang1, Pravin Bhat1, Alex Colburn2, Maneesh Agrawala2 and Michael Cohen2

1. University of Washington

2. Microsoft Research

Paper: [pdf, 58Mb]  Video : [Divx, 37Mb]

Abstract

We present an interactive system for efficiently extracting foreground objects from a video. We extend previous min-cut based image segmentation techniques to the domain of video with four new contributions. We provide a novel painting-based user interface that allows users to easily indicate the foreground object across space and time. We introduce a hierarchical mean-shift preprocess in order to minimize the number of nodes that min-cut must operate on. Within the min-cut we also define new local cost functions to augment the global costs defined in earlier work. Finally, we extend 2D alpha matting methods designed for images to work with 3D video volumes. We demonstrate that our matting approach preserves smoothness across both space and time. Our interactive video cutout system allows users to quickly extract foreground objects from video sequences for use in a variety of applications including compositing onto new backgrounds and NPR cartoon style rendering.

 

Keywords: Interactive video processing, min-cut, graph-cut, mean-shift segmentation, alpha matting

System Overview

 

Our Contributions

1. Volumetric painting interface. A novel user interface for allowing the user to indicate foreground and background regions within the video by painting of surfaces within the spatio-temporal volume.

 

2. Hierarchical segmentation algorithm. We introduce a hierarchical mean-shift over-segmentation preprocess to allow for solving global graph-cut optimizations in interactive time (a few seconds).

    Note : In the paper we reported that each sequence will take 10-15 seconds. We have worked on optimizing the algorithms so now our latest system can segment a whole sequence (100-150 frames) in 4-7 seconds!

3. Local color and edge costs. We define new spatially local color and edge models within a graph-cut framework to leverage the advantages video offers.

4.

4. Spatio-temporal alpha matting. We extend the 2D alpha matting presented in [Rother et al. 2004] to the 3D spatio-temporal video object to preserve both spatial and temporal smoothness of the alpha matte.

Some Results

                                                        Basic Cutout                                                                                           Application: NPR

               

       Application: Free-Form Deformation                                                        Application: Composition