Misery Business

We had footage of Dorothy and the Wicked Witch in the can. I say in the can, but... y'know... digitial. In the card doesn't sound as cool, though. They had been filmed anyway, is the point. I say filmed, but... y'know... digital. The scene was then firmly in the post-production phase. Time for editing and effects.

I'd already edited this scene twice at this point. Once when I edited the Dorothy footage, and once when getting it ready to present at Banner Elk. I'll be uploading both versions to Youtube sometime in the near future. The Banner Elk version was pretty much a disaster. I'd wanted a reveal for the Witch, I wanted the scene to build up all this tension, and then boom, there she is. I actually had the idea for her stepping into the light while editing for Banner Elk. But I was forced to compromise, because we'd had such trouble with the Witch mask up to that point.

So the Banner Elk people got to see a bastardized version of the scene, which played out the Witch's monologue in shaky close-ups. I was very dissatisfied with what we were able to present there, but I only had a couple of weeks to put it all together. This time, I had a couple of months, and I was going to take advantage of every second of it. You'll see soon I mean that literally.

I really enjoy editing, so I had no problem going back and starting the process from scratch. I wanted to make sure the scene was as good as I could make it. It came together pretty naturally. Even with just the action playing out on green-screen and a pixel-y pre-cursor to the final shot of the tower, Sean and I liked this version of the scene the best yet. It had the build-up and pay-off that I wanted so badly.

The edit only changed once after that. The first time through when I edit, I always trim out as much as possible. Jump into a shot late, leave early. As I started seeing the effects come together, I realized I could draw the whole thing out just a bit more. The longer you can go without cutting in a scene like this, the more tension there is. So I extended the first shot slightly, let the second shot play out longer after the Witch had turned around, and kept in an additional foot-step on the foot shot. They were little additions, about eight seconds total, but every second of film you choose to include or cut out is very important. Sometimes the trimming can come down to a frame or two.

I had the edit I wanted, and in order to let it remain precisely so, I would do my effects shots with the same file name as the original raw footage. A copy and replace into the folder containing the shots would obliterate the old one, and Sony Vegas would never know the difference. As long as the files were also AVI's, and the durations and frame-rates remained constant.

Which brings us to the effects. I have a love/hate relationship with visual effects. I have a deep love for them on a conceptual level. They open up amazing possibilities in film. I love watching a good visual effect, and I spend a lot of my free time on websites like fxguide, or watching visual effects breakdowns and documentaries.

But DOING visual effects... Eh. There are times when I get a lot of joy out of it. That's usually when an effect works easily, and ends up looking exactly the way I want it to. But the rest of the time I just... don't really care for it. I find it frustrating and repetitive.

That's why I'm more than happy to hand off as much work to Drew as I possibly can. Sadly, in this case it wasn't too much. I probably could've gotten some compositing out of him had I scheduled this whole thing better, but I didn't. That's not to downplay his input, though. What he did do for me was crucial stuff.

For a start, there was the motion-tracking. In case you don't know, motion tracking is how you replicate a camera move in the computer. There is software that will track various points in the frame to determine how the camera moved, and export that data in ways that 3D packages and compositing programs can use to make virtual cameras that move with the scene. It's not as automated as that, though. I have little experience in that field, which is one of the many reasons why it's good to have Drew on the team.

I promised more on Drew's lack of input on the Dorothy footage last time. I was in charge of the tracking marker situation that time around and... well... I wasn't thinking third dimensionally. The flat markers I placed on the screen are fine for 2D tracking, but they don't really do anything to indicate depth. Which has meant some hassle for Mr. More in getting this stuff successfully motion tracked.

But by gum he did it, and with great speed. He tracked the shots according to a priority list I'd made for him, and sent me the data as he had it. In addition, he was responsible for... visual effects supervision. When I've got my comps starting to come together, I'll show them to Drew, and he'll be able to articulate exactly why it's not looking right to me. His input is the difference between a shot that looks like this:

And the final:

So that's the sort of vague workflow, but how about the nitty-gritty? I'm going to break down three shots that run the range of process and technique. Between these three, you'll have a good overview of how the effects went together in the entire sequence.

THE PLANNED

This is a shot that was in the game plan from the word go. A movie can't play out entirely in close-ups, and sometimes we need to see Dorothy in the same shot with these other characters. Being the director, camera operator, and editor, storyboards were quite rare in shooting the Dorothy material. It was very run and gun. Sean and I would plan out the environments in broad strokes before hand, and then with that in my head I'd shoot what I'd need of Dorothy to get the shot I want later down the line.

Occasionally I'd frame a shot knowing that I'd want another character in the picture with her, and I'd tell Mare what action to do, and work out the timing. In this case, I told her to start by looking down to her right, and then when I gave the word, she nodded. The scripted reaction to that point in the Witch's monologue.

Shooting the Witch was a different process. And for any scene that featured Dorothy, much less free-form. I would edit the scene using the filmed Dorothy material and cheesy animations, so I knew exactly what we needed from the Witch for a successful scene. I'd show it to Marie when necessary, and that allowed us to work out the timing properly.

In the case of this shot, since we were going to composite her into a handheld shot of Dorothy, the Witch had to be shot on a tripod. Otherwise there would be no way to match her into the shot.

To begin the composite, I would start just adjusting the timing of the two separate character layers. The Witch element was just placed over top of the Dorothy element with 50% opacity, allowing me to make sure the nod was inserted at the right time.

I then began the composite proper. All the composites in this sequence were handled with planes at different depths, rather than rendering out actual tracked 3D backgrounds. I'd done some tests in Softimage XSI, and the parallax you get from that was so subtle as to not really be noticeable, which made the increased render times not worth it.

The composite then becomes just flat video images at different depths. They only look right from the one angle the virtual camera is shooting from. If you look at them from any other direction, it's just nonsense. Or a scene from Tron.

So obviously we have our two character plates. The castle element is a still of my battlement CGI model rendered at the appropriate angle. The cloud plates are elements I shot out the kitchen window during hurricane Irene a couple months ago. In this particular sequence the Witch's anger and the magic with the cap is sort of swirling up the clouds and causing them to twist around the castle.

Once all these layers are put together, it's all about the color grade. Adjusting the contrast and saturation of the various elements so they look like they belong together, and then grading the image as a whole. That's what really helps sell it.

THE CHEATS

This is a shot that shouldn't exist. There's really no way it could have. The keys are in the script, but not in any of the Witch designs up to the point of filming Dorothy. So I didn't have any idea what they'd look like, and the mere mention of keys didn't spark any creativity on my part. But Sean found this awesome set of keys, and did a really great job aging them. They looked so cool hanging off her waist that I HAD to showcase them in some way. But I didn't want to just insert a random shot of them dangling around.

Enter the shot that I had dubbed during the Banner Elk edit the "Leone Keys." I thought it'd be really cool to do the standard gunslinger kind of shot, but on those keys, with feeble little Dorothy sitting in the background. And I did, during our costume only day, where Marie didn't have on any of the prosthetics. But I didn't have that idea until 2011, which meant there was no corresponding Dorothy element for this shot.

I took a lot of spare footage of Mare on her last day. But it's mostly close-ups, rotations, and walking stuff. To get me out of any continuity issues that may arise. The teaser is comprised solely of this type of footage. And the original version of the key shot, for Banner Elk, was too.

But only being able to show her from the waist up, I had to cheat the angle of the background. And it looks quite lame. Luckily, as I searched through the raw footage, I found that we had some nice tripod footage of Mare sitting in the bell dress. It was perfect for this shot. And we had a flubbed take that contained about five seconds straight of Mare not speaking. So I nabbed that. She doesn't have the appropriate expression of terror, but knowing that the element was going to be out of focus, it didn't matter.

It was when working on this shot that I had the idea to adjust the blocking. You'll notice in the original that Dorothy was sitting in the middle of the wall. This made all the shots very flat. I decided I'd literally back her into a corner, which would liven up the visuals just slightly. Being a tripod shot, this is a straight up 2D composite. Sky footage, CGI Battlement, Dorothy, Witch. Again, the color grade is what really helps it pop. Here it is before:

THE LUCAS

If the last shot shouldn't exist, then this shot REALLY shouldn't. I didn't even have this idea until two days before the scene needed to be finished. The original shot was to be a better version of this, which made it into the Banner Elk version:

But I just wasn't happy with it, on multiple levels. It's a pretty flat, uninspired shot. Dorothy's all centered and just sitting there. It doesn't do much to ratchet up the tension, and at this point in the scene that's pretty crucial. Also, while I got it to work pretty well, I was having a hard time getting handheld Dorothy to sit on the CGI floor without sliding around. So I had failed the shot compositionally, and on a visual effects level. At the last minute I decided I wanted to replace it with something else.

Sean and I discussed and in some cases sketched out the environments before ever Shooting Dorothy, so I always knew what the space would look like when I was filming her. But sometimes you don't know how to best showcase that environment until you've seen it built. After working with the battlement model for all these months, I had an idea for a replacement shot that I thought would be really cool. And I figured I could make that work with the same sitting Dorothy element from before.

But as I said, I had this idea two days before the 31st. In order to get Dorothy behind the foreground element, and in front of the background element, I needed to render them out as separate passes. And the shot had to be 136 frames, to match the duration of the shot it was replacing. I didn't have the time to render out 271 frames in the time allotted. So, first things first, I rendered out the move just using the XSI viewport camera, which looks terrible, but renders very quickly.

I then rendered out a very large plate of the wall, with all the same frills as the other renders, high anti-aliasing, global illumination, final gathering. I didn't have time to render the foreground element like that, because it would take an hour per frame. Since I knew it would be out of focus, I brought everything down and lost the GI and final gathering, sticking with just an ambient occlusion pass instead.

When that was done, all that was left was compositing. I brought that viewport render in and animated a point to stick to the tip of the Dorothy stand-in's nose. I then positioned the nice render of the wall and the Dorothy element according to this reference, and parented their movement to that animated point. That left me with the two elements moving in a way that was correct to the shot. All that was left was to drop the foreground elements over top, blur, and do the color grade. Wham bam poetry slam, finished shot.

But I had a concern about it, and when I showed Sean the scene in its current form, his concern mirrored my own. Since the Dorothy element was never intended for this use, Mare's expression was not really appropriate to the scene. She looked kind of bored. She doesn't have a lot to do in this scene, but what was required of her she did very well. It wouldn't be right of me to make her performance look sub-par in this shot, especially when it's because of me that it is that way. So I plowed through the raw footage again, looking for a more suitable element.

What I discovered was that the scene I stole the piece from in the first place was the only time bell-dress Dorothy is sitting in a tripod shot. Which meant a more suitable element didn't exist. This is where a certain type of person is going to start hating me. Because I decided to create a more suitable element. I went back to the raw footage and picked out an alternate take of the shot this was meant to replace.

It's a handheld shot, like many in the film, so I couldn't just paste that more suitable head onto the other body. I picked the clearest still frame from the sequence, planning to use that as a base. Then I took the five frames of her averting her gaze, and in Photoshop pasted those eyes over the head element I'd chosen. I now had a steady head that wasn't just a still frame.

I shrunk, flipped, and masked that into position over the other body, with the eye animation taking place just at the end of the shot. And that was it, finished.

As I said, this is sure to raise the ire of a certain type of person, and I can certainly sympathize with their point of view. But I needed to get a different shot in there, and I didn't want to do a disservice to my lead actress. With her being too old now for reshoots, and me being hundreds of miles away anyhow, this seemed like the only way forward.

I ended up finishing the scene at seven AM on the 30th, after staying up all night working on it. We were having a pretty bad snow storm, and our lights kept flickering. As we frequently lose electricity in storms, I wanted to make sure I had it done and online on time, so our reveal "event" wasn't ruined. I uploaded it immediately after finishing, so Sean would have time to incorporate it into the web design. In fact, some of you got a sneaky peek when Youtube randomly decided I was kidding when I made it unlisted.

So there you have it. A VERY in depth breakdown of the visual effects process. If you stuck it through to the end, congratulations...