Zenith

The Zenith Project is the major project that Freestyle students make their senior year using the skills they learned in their Freestyle classes.

Originally, I planned on making a documentary film about about a company that developed new technology for pointe shoe padding. Unfortunately, I became extremely ill early on in the Zenith process and couldn’t complete my project as planned. I was unable to sit up or stand and had to create a new project that I could do while lying in my bed.

After brainstorming ideas with my parents, I decided to attempt to create what they call the Maddy-cracker. I have danced ballet since I was seven years old and in almost every year participated in my studio’s Nutcracker. My dutiful father has seen me perform all of the parts I have played over the years, from a Mother Ginger child to the Snow Queen. As much as he loves to watch me dance, he has no desire to see other amateur ballet. He has described his experience of watching the Nutcracker as “blah, blah, blah, Maddy! blah, blah, blah.” Ever since I started learning ways to edit video in Freestyle, he has encouraged me to create a combined video of my studio’s Nutcracker in which I play all the parts. A Maddy-cracker.

For my Zenith, I decided to take footage from five different years and attempt to put myself in the place of those five parts for the opening scene of Act Two. It seemed like a good scene to attempt to try because it was short but I had portrayed many different parts in it. My original plan was to place myself playing the parts of Dewdrop, Flower Girl, Russian and Clara onto the footage of myself portraying the Arabian lead. I used the Arabian footage as my base layer since it involved two people and had minimal camera movement.

In order to do this project, I would to learn how to rotoscope in After Effects. Rotoscoping is where you create a matte or trace of an object to put it on a different background.

In After Effects you can rotoscope by using masks, the rotobrush tool or Mocha AE. In film we had briefly learned how to use the rotobrush tool in After Effects but I didn’t think that that would be effective enough for this project. After some research I determined that I would try to use as much masking as I could and then use Mocha AE to finish.

Attempt #1

In my first attempt I used the VOB files that I had downloaded from the DVDs. I lined the clips up according to the music and adjusted the sizes and placement as I went.

This attempt worked mostly well for combining the Dewdrop and the Arabian, but it got more complicated for the other parts, especially as the camera moved around differently with the different clips. After I created the masks I tried to import the first clip into Mocha and discovered that VOB files are not compatible with Mocha, so I abandoned this attempt.

The only reason it works well for the Dewdrop, though, is because the Arabian man is played by the same person, standing in the exact same place, in both the Dewdrop and Arabian clips. After Effects masks work best for rotoscoping simple shapes. With some more feathering and color correcting, I could make the line where the clips overlap less noticeable, but it will never truly go away.

Attempt #2

Since I thought Mocha wouldn’t work with my files, for my second attempt I decided to just use the rotobrush. I restarted and lined the clips up the same way as before and put a mask over the Dewdrop in the Arabian clip. The rotobrush tool works similarly to the quick selection tool in Photoshop, where you roughly mouse around the image you want selected and then add or subtract to the selection until you get what you want.

Due to the fact that the clips were not very high resolution, I found it very hard to get an accurate selection and had to go frame by frame. This got even harder as the Dewdrop moved into the background and became more out of focus. My computer also became extremely slow as I kept creating new layers for each addition or subtraction to the selection. In the end I created 2,022 different layers.

I gave up this attempt because it was too slow and ineffective. Even though I went frame by frame there are still times parts of the Dewdrop go missing. The rotobrush works best when there is a lot of contrast between the shape you are trying to rotoscope and the background. This attempt could be improved with more work but it will never work that well.

Attempt #3

After these attempts, I talked to my teacher and realized that I could convert the VOB clips using Adobe Media Encoder. Once I did this, I was able to import the clips in Mocha. In Mocha I used the X-spline layer tool to create a complete outline of myself. The X-spline tool uses anchor points that can be adjusted to create sharper or smoother lines and curves. I then had Mocha track the Dewdrop shape.

Because the Dewdrop shape changed so much, Mocha had a hard time tracking it. I had to use the pick tool to create a lot of keyframes that were right next to each other. This was somewhat of an improvement on the rotobrush because my selections were more precise, but it was still very tedious. I ran into issues when I tried to import the shape data back into After Effects. Because the aspect ratio had been changed when I put the clip into Mocha, the data was misaligned in After Effects.

After some research I was able to get the aspect ratio sorted out but I was still not happy with how Mocha was working out. I thought that it might be more effective to create shapes of single limbs and compile them all together to make a Dewdrop. I tried to create a mask for just one leg to see if Mocha would have an easier time tracking that and I would have less manual work to do. For more advanced rotoscoping, a composite of multiple masks is generally the best way to get a precise mask and good-looking final product. In my case, however, I found this approach even less effective because of the way the legs moved and crossed over each other.

      

At this point I realized that my initial goal was far too ambitious, so I started over again to try and use Mocha AE to just track the Dewdrop shape.

On the final attempt using Mocha, I tried to let Mocha track through the clip a bit and not create new keyframes for each frame. I tried to only adjust the mask when it got significantly off from the Dewdrop shape. Unfortunately, the track got off almost every couple of frames. I learned the zoom was my best friend in order to get an accurate mask even though the low resolution and motion blur could make it hard to tell where the Dewdrop shaped ended and the background began. I also learned that when I made a new keyframe and adjusted a mask, it worked best to start adjusting the part that was the most accurate and move toward the part that was the least accurate. As I got more familiar with Mocha and the different tools, I was able to more effectively manipulate multiple points at once, which did help speed up the process somewhat.

Mocha did work well for the point where the Dewdrop crosses behind the Arabian man because I was able to make a mask around the Arabian shape and delete that from the Dewdrop mask. This technique also helped me for the entrance of the Dewdrop where I made a mask around the stage wing.

Once I finished tracking the Dewdrop shape in Mocha, I went back and added even more keyframes to make the track more accurate. Adding keyframes that are close together is generally not a good idea because it can make the track look shaky but with the movement of the Dewdrop shape, I didn’t really have a choice. Although I ran into some of the same problems with Mocha that I had with the rotobrush, I didn’t have the computer processing issues and I had more manual control over what I was selecting. This made for a much more enjoyable experience.

I then went back into After Effects where I used a simple mask to cover the Dewdrop from the Arabian clip. I had to use multiple versions of the Arabian clip that I froze at various times in order to eliminate the Dewdrop while keeping the Arabian man. This worked pretty well but it was complicated by the changes in camera movement and lighting.

Lastly, I copied the shape data from Mocha and pasted it onto the Dewdrop clip in After Effects. Unfortunately, the data still did not copy correctly and the mask was off on the footage.

Attempt #4

After a lot of trouble shooting, I was able to fix this problem so that the mask I made in Mocha worked with the footage in After Effects. My final version still has many flaws and it would take hundreds to thousands more hours of work to make it look seamless.

Overall, I learned that good rotoscoping is quite hard and requires a lot of time and work. Rotoscoping dance is especially difficult due to all the movement and in retrospect was probably not the best thing to start with. Ballet creates very unnatural shape movement with the way different limbs of the body cross over each other and go in and out of view. With this project I was also limited by the quality of the source material. The clips I used are shot from different places, have different camera movements, are not well lit or in focus, and are low resolution.

It is disappointing that even though I put a lot of time and effort into this project, I still don’t have a finished product that I am satisfied with. At the current time I don’t have the skill or source material required to make the Maddy-cracker. I did, however, learn a lot through this process and I became much more comfortable with software that I was previously unfamiliar with.