Picture this -- Sicily 2019 (forgive my Golden Girls reference). A stalwart Patrick steps out of a vehicle with his trusty sidekick, Sancho, ready for their commercial shoot for a mid-tier paper supply company. Sancho meticulously plans the shot, consulting the light gods, while Patrick consults with the client.
We start filming and everything looks good, great even, when all of a sudden the client wants another shot set up despite the fact that the interviewee is showing signs of fatigue from standing in the hot sun. Nevertheless, the client decides that we need to get a lower angle to get some distant signage.
Our answer is always, "Yes ... we can do that."
Flash forward and we see Patrick sending the client a "fine" draft, a version that is in the final stages of post-production, and the client has noticed something in the background - a trash can. A trash can that was moved from the first shot but then appeared in the second shot due to the rushed camera change. What do we do?
My suggestion is to ditch that shot and replace it with something else - something from one of the other cameras that we had rolling during the interview (we always have at least 2 cameras). Itʻs a common problem with several workarounds. But the client doesnʻt want workarounds, they want me to paint the trash can out.
Paint. The. Trash. Can. Out.
“Painting a fixed object out of the background of a still picture isnʻt that hard and people do it all the time!”
Painting the trash can out of a still picture isnʻt impossible but painting it out of a moving image can be incredibly labor intensive which ultimately means more time and cost to the client.
Here’s why:
Yes, artists and technicians paint still images out of frame all the time. But a moving image is exponentially more challenging because of the calculations involved in manipulating that image. A video camera in the United States most likely will be running at either 24 or 30 frames per second. Each frame is a still picture. If the shot lasts for 5 seconds then that is 120 and 150 still pictures, respectively. That is a staggering amount of images for such a small amount of time. Now, there are programs designed specifically for this task but using them correctly still takes time and skill.
For each frame the computer hypothesizes what you will do next based on what you have done before. However, the computerʻs guess is often far from perfect because the computer is reading the colors in the image not the object in the image. It doesnʻt know that you are painting out the trash can. All it knows is that those colors are somewhat similar and the gradations on the colors are changing slightly and that they were similar colors before so they may be similar colors after. Great. What that means in practice is that I need to check all the pictures, by hand, to make sure that the computer hasn't collected other things in addition to the trashcan and accidentally grouped them together. And, if it has, which is usually the case, I need to manually correct it and replace it.
This process is known as rotoscoping. Rotoscoping is the most painstakingly tedious process to ever be conceived by rational beings.
So what happens next?
Well, I’m glad you asked. What Iʻm left with is a trashcan shaped hole in the image, or more accurately, in each layer of the image. The trash can was covering a lower wall, a middle wall, some grass and concrete. That’s 4 different distinct colors each with distinct derivations that now need to be matched in each frame. To do this I use the same tool from the previous step to duplicate the correct color and texture from other parts of the image to backfill the hole the trash can left. I painstakingly repeat this process for each and every frame in the scene cropping, feathering, reducing, darkening and skewing the image to match the nonexistent trash can 120 or 150 times, as needed, all for a 5-second shot.
This process is tough for a seasoned pro. The best of the best can do it in 8 hours.
I’m pretty good -- I can do it in 16.
Is it worth it?
Some people will argue, yes, it is our job to make sure everything within the frame is perfect and composed in a way that enhances, and doesnʻt detract from the scene. A trash can is ... gross. Definitely a detraction.
But, ultimately, I would argue that it isn’t worth it. The better answer is to do it right the first time by setting up the shot differently or using different footage that we can splice into the scene.
For the record, thatʻs exactly what we ended up doing. That said, there will be times when rotoscoping is your best, or only, option and everyone, clients and producers, should know that itʻs available to them.
If there’s one takeaway from our adventure in Italy, itʻs this - when the tripod is down, have a look around ...
Hello! This is the first article Iʻve published and would love to get feedback! Let me know if you liked the story and if you have had a similar experience on or off set! Please feel free to like and share