ChatGPT maker OpenAI has now unveiled Sora, its artificial intelligence engine for converting text prompts into video. Think Dall-E (also developed by OpenAI), but for movies rather than static images.
It’s still very early days for Sora, but the AI model is already generating a lot of buzz on social media, with multiple clips doing the rounds – clips that look as if they’ve been put together by a team of actors and filmmakers.
Here we’ll explain everything you need to know about OpenAI Sora: what it’s capable of, how it works, and when you might be able to use it yourself. The era of AI text-prompt filmmaking has now arrived.
OpenAI Sora release date and price
In February 2024, OpenAI Sora was made available to “red teamers” – that’s people whose job it is to test the security and stability of a product. OpenAI has also now invited a select number of visual artists, designers, and movie makers to test out the video generation capabilities and provide feedback.
“We’re sharing our research progress early to start working with and getting feedback from people outside of OpenAI and to give the public a sense of what AI capabilities are on the horizon,” says OpenAI.
In other words, the rest of us can’t use it yet. For the time being there’s no indication as to when Sora might become available to the wider public, or how much we’ll have to pay to access it.
We can make some rough guesses about timescale based on what happened with ChatGPT. Before that AI chatbot was released to the public in November 2022, it was preceded by a predecessor called InstructGPT earlier that year. Also, OpenAI’s DevDay typically takes place annually in November.
It’s certainly possible, then, that Sora could follow a similar pattern and launch to the public at a similar time in 2024. But this is currently just speculation and we’ll update this page as soon as we get any clearer indication about a Sora release date.
As for price, we similarly don’t have any hints of how much Sora might cost. As a guide, ChatGPT Plus – which offers access to the newest Large Language Models (LLMs) and Dall-E – currently costs $20 (about £16 / AU$30) per month.
But Sora also demands significantly more compute power than, for example, generating a single image with Dall-E, and the process also takes longer. So it still isn’t clear exactly how well Sora, which is effectively a research paper, might convert into an affordable consumer product.
What is OpenAI Sora?
You may well be familiar with generative AI models – such as Google Gemini for text and Dall-E for images – which can produce new content based on vast amounts of training data. If you ask ChatGPT to write you a poem, for example, what you get back will be based on lots and lots of poems that the AI has already absorbed and analyzed.
OpenAI Sora is a similar idea, but for video clips. You give it a text prompt, like “woman walking down a city street at night” or “car driving through a forest” and you get back a video. As with AI image models, you can get very specific when it comes to saying what should be included in the clip and the style of the footage you want to see.
To get a better idea of how this works, check out some of the example videos posted by OpenAI CEO Sam Altman – not long after Sora was unveiled to the world, Altman responded to prompts put forward on social media, returning videos based on text like “a wizard wearing a pointed hat and a blue robe with white stars casting a spell that shoots lightning from his hand and holding an old tome in his other hand”.
How does OpenAI Sora work?
On a simplified level, the technology behind Sora is the same technology that lets you search for pictures of a dog or a cat on the web. Show an AI enough photos of a dog or cat, and it’ll be able to spot the same patterns in new images; in the same way, if you train an AI on a million videos of a sunset or a waterfall, it’ll be able to generate its own.
Of course there’s a lot of complexity underneath that, and OpenAI has provided a deep dive into how its AI model works. It’s trained on “internet-scale data” to know what realistic videos look like, first analyzing the clips to know what it’s looking at, then learning how to produce its own versions when asked.
So, ask Sora to produce a clip of a fish tank, and it’ll come back with an approximation based on all the fish tank videos it’s seen. It makes use of what are known as visual patches, smaller building blocks that help the AI to understand what should go where and how different elements of a video should interact and progress, frame by frame.
Sora is based on a diffusion model, where the AI starts with a ‘noisy’ response and then works towards a ‘clean’ output through a series of feedback loops and prediction calculations. You can see this in the frames above, where a video of a dog playing in the show turns from nonsensical blobs into something that actually looks realistic.
And like other generative AI models, Sora uses transformer technology (the last T in ChatGPT stands for Transformer). Transformers use a variety of sophisticated data analysis techniques to process heaps of data – they can understand the most important and least important parts of what’s being analyzed, and figure out the surrounding context and relationships between these data chunks.
What we don’t fully know is where OpenAI found its training data from – it hasn’t said which video libraries have been used to power Sora, though we do know it has partnerships with content databases such as Shutterstock. In some cases, you can see the similarities between the training data and the output Sora is producing.
What can you do with OpenAI Sora?
At the moment, Sora is capable of producing HD videos of up to a minute, without any sound attached, from text prompts. If you want to see some examples of what’s possible, we’ve put together a list of 11 mind-blowing Sora shorts for you to take a look at – including fluffy Pixar-style animated characters and astronauts with knitted helmets.
“Sora can generate videos up to a minute long while maintaining visual quality and adherence to the user’s prompt,” says OpenAI, but that’s not all. It can also generate videos from still images, fill in missing frames in existing videos, and seamlessly stitch multiple videos together. It can create static images too, or produce endless loops from clips provided to it.
It can even produce simulations of video games such as Minecraft, again based on vast amounts of training data that teach it what a game like Minecraft should look like. We’ve already seen a demo where Sora is able to control a player in a Minecraft-style environment, while also accurately rendering the surrounding details.
OpenAI does acknowledge some of the limitations of Sora at the moment. The physics don’t always make sense, with people disappearing or transforming or blending into other objects. Sora isn’t mapping out a scene with individual actors and props, it’s making an incredible number of calculations about where pixels should go from frame to frame.
In Sora videos people might move in ways that defy the laws of physics, or details – such as a bite being taken out of a cookie – might not be remembered from one frame to the next. OpenAI is aware of these issues and is working to fix them, and you can check out some of the examples on the OpenAI Sora website to see what we mean.
Despite those bugs, further down the line OpenAI is hoping that Sora could evolve to become a realistic simulator of physical and digital worlds. In the years to come, the Sora tech could be used to generate imaginary virtual worlds for us to explore, or enable us to fully explore real places that are replicated in AI.
How can you use OpenAI Sora?
At the moment, you can’t get into Sora without an invite: it seems as though OpenAI is picking out individual creators and testers to help get its video-generated AI model ready for a full public release. How long this preview period is going to last, whether it’s months or years, remains to be seen – but OpenAI has previously shown a willingness to move as fast as possible when it comes to its AI projects.
Based on the existing technologies that OpenAI has made public – Dall-E and ChatGPT – it seems likely that Sora will initially be available as a web app. Since its launch ChatGPT has got smarter and added new features, including custom bots, and it’s likely that Sora will follow the same path when it launches in full.
Before that happens, OpenAI says it wants to put some safety guardrails in place: you’re not going to be able to generate videos showing extreme violence, sexual content, hateful imagery, or celebrity likenesses. There are also plans to combat misinformation by including metadata in Sora videos that indicates they were generated by AI.
+ There are no comments
Add yours