OCT 2024
Maintaining the visual consistency of a character or product over time in an AI-generated shot has always been tricky.
In this test I decided to not even try.
Instead, I filmed the product with my phone and used AI to generate the environment around it. I also generated a lighting pass to help integrate the product into the new environment.
In the pre-AI-vid2vid era, creating something like this would require 3D-tracking the camera move, laying out geometry, texturing/shading the geometry, lighting and rendering the scene and then compositing the isolated shoe back into the new CG environment. Now you can get a similar result in less steps.
One limitation of environment-generation derived from pre-shot footage is how closely the AI retains the structure of the original footage. You can see that it is basically reskinning the existing objects in the original environment (this happens no matter how far you push the structure attribute in Runway, I wish you could push it further). I have experimented with creating a full CG build instead of shooting, where only the product is a textured/rendered model and the surrounding environment is proxy geo. This workflow lets you change the composition of the environment quickly but the ‘reskinning’ effect is still an issue.
Fun times, feels like a new way to 'render', albeit with less control, but far faster and less specialized skillsets required.