Advancements in AI for 3D and video production

Quite some time has passed, and we have witnessed the lightning speed of development in generative AI. While we’ve already gotten used to seeing new graphical wonders coming from text-to-image AIs, the rise of ChatGPT came as a surprise for many.

It has been a truly explosive development. I remember how the general media picked up on the topic, and all of a sudden, ChatGPT was used for almost everything. It has been the general public’s turning point in generative AI use. ChatGPT has also woken up the last of the sleeping dogs, and fear and rejection toward generative AI quickly reached new heights. 

While the AI wars rage on and this new technology still faces angry opposition from many different directions, it is becoming obvious that there is no way back. Like it or not, generative AI is here to stay.

While many are seriously afraid to lose their jobs or even see their whole industry on the verge of destruction, others have already begun seeing the expected benefits of this new technology.

According to Goldman Sachs, generative AI has the potential to boost global GDP by 7% in the next decade while also improving productivity and efficiency. McKinsey Global Institute estimates that generative AI could add trillions of dollars to the global economy, revolutionizing industries and enterprises. In any case, the impact of generative AI on economies and businesses will be profound.

Today, I’d like to focus on two areas that are just starting to be affected by the use of artificial intelligence (AI) in the media industry.

Use cases continue to expand, with exciting new developments emerging all the time. In particular, the use of AI for 3D modeling and video production has been making waves in the industry lately.

3D modeling and animation

Traditionally, creating 3D models has been a time-consuming and labor-intensive process. However, with the help of AI, this is starting to change. One of the most significant developments in this area is using generative adversarial networks (GANs) to automatically create 3D models. GANs consist of two neural networks - a generator and a discriminator - that work together to create a model that is as realistic as possible. This process is well known to those around you who have been working with text-to-image AIs.

Another exciting development in AI for 3D modeling is reinforcement learning. This involves training an algorithm to improve its performance over time by rewarding it for making correct decisions. This technology could help 3D modeling software automatically learn how to create more sophisticated and detailed models. And we are not only talking about a general use model. This training can also be used in media creation companies to have the trained AI stick to their design house rules.

As we enter the age of text-to-3D, the possibilities are endless. It all starts with a prompt, just like in an image or text AI. But the resulting output is on a whole new level. Instead of being a flat image, the 3D creation already consists of different 3D elements that can then be refined and rearranged by the user. These changes can then be made traditionally using manual inputs or the prompt system. And since we are looking at individual elements, they can be changed one by one. These changes can be made using plain human language.

You can picture it like this:

You start by requesting: Create a small square room with a bed, table, and chair, made in the style of a cute comic.

Now, you end up with the room but want to avoid the created table.

Your next request is: Change the table to be round.

Et voilà: Here is your round table.

Next, you might want to change the objects’ textures and use the prompt system once again.

The exciting part is still to come: You can also use prompts to define the position of objects tied to a timeline. That means you can directly convert a story into a full 3D animation using text input.

The best part is that you will always have the option of going over to your old-fashioned way of doing things at any point because this new tech will be available inside your 3D software.

At the moment, Spline is already in the public beta phase of implementing AI, and it is believed that Blender will soon follow.

Video production

AI is also making an impact on the world of video production. One of the most notable applications of AI in this field is using computer vision to automatically identify and tag different elements within video footage. This makes it easier for editors to search for specific clips and helps to significantly reduce the time it takes to create an edit.

And when you, as an editor at this point, thought: “Oof, great, I am safe,” you unfortunately might be wrong. Once the AI has learned to identify the footage, advancing to automated editing is just one more step away. Of course, a human decision-maker might still be needed, but it is quite obvious that the amount of human involvement will decrease.

Another area where AI is making a difference is in the creation of special effects and CGI. In the past, creating realistic-looking effects required a lot of time and effort. However, with the help of AI, the process is becoming faster and more precise.

For example, machine learning algorithms can automatically create realistic-looking textures and materials for 3D models, helping them blend seamlessly into live-action footage. This, once again, is powered by the ever-improving ability of these systems to identify and learn. Soon, the AI can decide whether an effect looks realistic or not.

And then, lastly, there is the elephant in the room - or, if the elephant is an actor and a soundstage - it might just be that there is no elephant in the room at all. But the outlook toward realistic actorless movie-making is worth a full article so I will leave it at that for now.

Challenges and limitations

While the use of AI in 3D modeling and video production is undoubtedly exciting, challenges still need to be overcome.

One of the main issues is the need for large amounts of data to train neural networks. This can be a challenge in industries where data is scarce or where there are privacy concerns around the use of consumer data. Being forewarned by the developments in the art scene, the big studios by now will have made it quite clear what material can NOT be used to train models. So, it is more plausible to see proprietary AIs used by specific studios trained with their own data.

Another challenge, as mentioned before, is the need for human intervention at key stages of the process. While AI is making the creation of 3D models and video production easier and faster, some tasks still require human creativity and judgment. At least so far. This includes tasks such as scriptwriting and directing. While scriptwriting is already in the danger zone, directing cannot be automated using current AI technologies just yet.

The next big thing(s)

One thing that will once again accelerate the use of generative AI is speech. And it has already started, too. Voice-controlled devices have not only become commonplace by now, but they also finally work really well. This is, of course, again powered by advances in AI technology. Using plain, spoken language to request what you need a generative AI to create for you breaks down the last barriers. This is the point at which you need no technological skills anymore to use AI, and this step is just around the corner.

The other technology that ties right into this is AR. While VR has remained a rather niche tech, new products like Vision Pro will eventually bring widespread adoption of AR into everyday life. This technology relies heavily on using only your body and voice to control and communicate with it.

I think this might be the final frontier towards the age of AI. Looking at the development of generative AI in the last five years, it should be clear that soon, we will fully rely on this technology.

Conclusion

Overall, the use of AI in 3D modeling and video production is an exciting area with enormous potential. As technology continues to develop and improve, we expect to see more exciting developments in this field.

While there are challenges to be overcome, AI’s dangers and benefits for media professionals are becoming increasingly clear, and we will inevitably see this technology being used more widely in the coming years.

This article is written by Luvcraft as a part of enter.blog's bounty program. Do you have an interesting topic, series or subject you think would be fitting for enter.blog? 

You can now submit your articles to enter.blog and get paid for your contributions!
Read more and submit your article here.