Check out our online demo for exploring hundreds of results and the code used to create them!
I haven't tried running the code on other machines yet, so let me know if you have any trouble. I'll probably be updating / transplanting features from a codebase that's connected to other projects for a while.
Abstract
We present a visual analogue for musical rhythm derived from an analysis of motion in video, and show that alignment of visual rhythm with its musical counterpart results in the appearance of dance. Central to our work is the concept of visual beats — patterns of motion that can be shifted in time to control visual rhythm. By warping visual beats into alignment with musical beats, we can create or manipulate the appearance of dance in video. Using this approach we demonstrate a variety of retargeting applications that control musical synchronization of audio and video: we can change what song performers are dancing to, warp irregular motion into alignment with music so that it appears to be dancing, or search collections of video for moments of accidentally dance-like motion that can be used to synthesize musical performances.Sample Results
Barney Don't Need No Education
Barney and Friends retimed to “Another Brick in the Wall” by Pink Floyd.
If you're happy and you know it
The owners of the coffee shop I regular have a todler who is often watching this show when I go to get my coffee. They also like rap music, so I made this.
Kitty Shake Shake It
Cats dancing. Enough Said.
Psy's Best Friend
This example was a lot of fun to create. I found this Youtube channel for Dancing Nathan – a dog that does this strange flailing trick on a chair, which kind of looks like off-beat dancing.
Spot The Dancing Robot
Remix of Spot The Dancing Robot from Boston Dynamics.
Visual Rhythm And Beat (Overview Video)
A video giving an overview of the project and what we can do.