How Generative AI creates worlds in real-time
Exploring Game Development with AI
If you would like exposure to the biggest opportunities at the intersection of Generative-AI, Virtual Reality, Gaming and Web3, reply to this email to learn more about our upcoming Fund II or meet us in person at SALT NYC.
In January of this year, Valve announced a significant update requiring developers to disclose when their games use AI. Following this update, Ichiro Lambe, prev. Steam Labs discovered that over 1,000 Steam titles now incorporate generative AI, marking a transformative trend in gaming.
Nearly $700 million has been invested in this sector since 2022. Beyond well-known platforms like ChatGPT and Midjourney, this technology is reshaping several aspects of game development, including NPC interactions and animations.
The funding supports diverse applications. It helps studios improve game production and allows gamers to change their video game surroundings dynamically.
Generative AI in the Game Development Process
Startups like Scenario and Leonardo AI are leading the way in generating 2D asset designs. Still, 3D asset generation remains a more complex challenge. It involves creating models from 2D concepts and optimizing, rigging, animating, and integrating them into scenes. Given the complexities, 3D technology is progressing but at a slower pace compared to 2D.
Sloyd stands out as one of the top ten most promising Norwegian startups of 2024. It aims to apply AI across all production stages - from scene creation to single object generation and iterative object editing. Users will be able to input descriptions like "a living room" or "a downtown city block," and the AI will automatically arrange and adjust these scenes.
Another example is Blockade Labs, known for its Skybox AI, which enables the creation of AI-generated 3D worlds. Recently, the company has taken a significant step forward by upgrading its tools to Model 3. This enhancement not only boosts the visual immersion of their creations but also ensures that these detailed environments can be seamlessly integrated into popular game engines like Unreal and Unity.
Optimizer.AI transforms game development with its text-to-sound effect model. Users can easily create immersive sound effects for games, videos, and animations.
Google's Vertex AI enables easy customization of models like Gemini to enhance gaming environments. The Vertex AI Agent Builder can construct AI-powered search engines and agents. Google Genie generates interactive, playable environments from single image prompts, including photos and sketches. Trained on Internet videos without action labels, Genie can identify controllable elements and simulate consistent actions across various generated worlds.
These tools not only refine game environments but also deepen player involvement. Moving to Generative AI during gameplay, the distinction between player and creator blurs, allowing players to transform their virtual worlds dynamically.
Generative AI during Gameplay
This includes companies developing intelligent non-playable characters (NPCs), incorporating AI to create totally new gaming experiences, and allowing users to play a larger role in producing user-generated content (UGC).
Smart NPCs
The subject of intelligent NPCs has received the most media interest. InWorld AI was valued at $500 million in a key acquisition completed in late 2023.
Convai, another conversational AI platform, announced new features and collaborations at GDC 2024, including better NPC features for games like Second Life and the upcoming Stormgate.
Convai has integrated with Unity's AI-powered platform, introducing capabilities like Convai Connect for cost management, language support, and long-term memory for NPCs, allowing them to remember and react to player interactions.
Agents
Altera is working on the first AI Agent that you can interact with directly in Minecraft. Previously, AI NPCs were limited to speaking in games. This team is making buddies who can play games with you and are constantly available online.
DeepMind published research on the Scalable Instructable Multiworld Agent (SIMA) in March. This artificial intelligence is meant to learn and follow instructions in a variety of 3D video game environments.
SIMA was given training on nine different games, demonstrating its ability to generalize skills and follow language-based directions for activities such as resource collecting and navigation.
The Emergence of Text-to-Video
One of the most exciting developments in real-time creation is the evolution of text-to-video technology. As explored in a previous article, The Dawn of Photorealistic Video Creation, Platforms like Sora lead the charge, offering a glimpse into a future where game narratives evolve in real-time and respond dynamically to player decisions.
Shengshu Technology and Tsinghua University in China recently revealed Vidu, a text-to-video AI tool capable of creating 1080p films as long as 16 seconds. In response to OpenAI's Sora, Vidu emphasized China's AI accomplishments in the face of US tech export limitations.
Expanding the Horizons of Generative AI
The potential of generative AI extends beyond storytelling to the environments we explore. A noteworthy example is StreamDiffusion, a Stable diffusion-powered technique used by Federico Guardabrazo in Minecraft. It can currently generate videos at 20 frames per second.
The implications are vast - from changing weather systems that affect gameplay to evolving landscapes that challenge players to adapt.
Transforming 3D Environments with Gaussian Splatting
The 3D Gaussian Splatting method is another unique technology that has the potential to impact gaming significantly. This approach reconstructs scenes from several images with the same or higher accuracy as Neural Radiation Fields (NeRF) but with a substantially lower rendering burden. This capacity enables the production of photorealistic 3D scenes that may be produced in real-time, hence improving the visual quality and immersion of virtual landscapes.
Just last week, Researchers presented a hierarchical 3D Gaussian format intended for real-time visualization of very big datasets. This can improve the capacity to generate large game environments in real-time, assuring good visual quality even for scenarios of enormous size.
The approach uses a hierarchy of 3D Gaussians to generate efficient Level-of-Detail (LOD) solutions, allowing for smoother transitions and more effective level selection as the player moves about the game area.
The Road Ahead
The future of generative AI in gaming lies in combining human creativity with artificial intelligence, resulting in games that are diverse, adaptable, and exciting.
This is only the first wave of companies to rise at this intersection. Real-time generative AI presents a number of challenges for creators, requiring not just capable artificial intelligence but also a fresh approach to game creation.
Disclaimers:
This is not an offering. This is not financial advice. Always do your own research.
Our discussion may include predictions, estimates or other information that might be considered forward-looking. While these forward-looking statements represent our current judgment on what the future holds, they are subject to risks and uncertainties that could cause actual results to differ materially. You are cautioned not to place undue reliance on these forward-looking statements, which reflect our opinions only as of the date of this presentation. Please keep in mind that we are not obligating ourselves to revise or publicly release the results of any revision to these forward-looking statements in light of new information or future events.