Authors: David Silver, Richard S. Sutton
Abstract
We stand at the threshold of a new era of artificial intelligence that promises unprecedented capabilities. A new generation of agents will learn primarily through experience and acquire capabilities beyond those of humans. This article explores the key features of that coming era.
The Era of Human Data
In recent years, artificial intelligence (AI) has made significant progress through training on vast amounts of human-generated data, followed by fine-tuning with expert demonstrations and preferences. Large language models (LLMs) exemplify this approach and have made striking advances in versatility. An LLM can now handle tasks ranging from writing poetry and solving physics problems to diagnosing medical issues and summarizing legal documents.
Although imitating humans can produce satisfactory results in many areas, imitation alone cannot deliver breakthrough capabilities. These models tend to reproduce existing human ways of thinking and frameworks of knowledge, limiting their ability to create new knowledge or discover new strategies.
The Arrival of Autonomous AI Experience
We believe the next frontier of AI will be defined by agents that learn through experience. Instead of relying on human demonstrations, these agents will develop capabilities by interacting with their environments, experimenting, and learning from failure. The process resembles how infants or animals gain knowledge by exploring the world themselves.
This paradigm shift has already been demonstrated in reinforcement learning. Reinforcement-learning agents have surpassed human experts in games, robot control, and other challenging tasks.
Learning through experience has several key advantages:
- Discovery: agents can discover solutions humans have not explored.
- Adaptability: agents can relearn and adapt when environments or rules change.
- Continuous progress: without the limitations of a fixed dataset, agents can keep evolving through continued interaction.
- Less dependence on human knowledge: particularly where knowledge is scarce or human biases are strong, experience-based learning can avoid inheriting those biases.
The Rise of Autonomous Capabilities
In the era of experience, agents will gain autonomous capabilities: completing tasks by setting their own goals, discovering strategies, and learning, rather than depending on human instructions. Unlike models that rely on human labels, prompts, or feedback, these agents will:
- Ask questions independently.
- Set their own goals.
- Evaluate outcomes themselves.
- Gradually build complex bodies of knowledge and combinations of skills.
These capabilities emerge from a combination of reinforcement learning, evolutionary algorithms, planning, and search, which together enable agents to develop in a more lifelike way.
For example, some Go agents have demonstrated strategies humans had never used. Robots have learned unconventional but more effective movements in the real world. Virtual agents in complex three-dimensional worlds have developed sophisticated navigation, planning, and even social strategies from basic actions.
Autonomous Experience Outperforms Imitation
Imitation learning, or learning from human demonstrations, is useful early on but has clear limits:
- Imitation can learn only what humans already know.
- It is constrained by the quality of human performance.
- It generalizes poorly to unfamiliar situations.
- It may inherit human biases and mistakes.
In contrast, experience-based learning can:
- Discover entirely new strategies and knowledge.
- Show greater adaptability and creativity when facing unfamiliar problems.
- Correct itself and continue evolving.
- Break out of established human paradigms and explore superhuman capabilities.
Experience-based learning can also help agents develop a kind of intuition or understanding: grasping the underlying patterns and structures rather than merely remembering examples.
Toward General Intelligence
The real potential of experience-based learning is that it opens a path toward artificial general intelligence (AGI). Human intelligence is powerful because we can:
- Solve new problems without explicit instructions.
- Transfer experience flexibly between contexts.
- Continually adjust strategies through failure.
- Above all, learn through experience, rather than imitation alone.
An agent that grows through experience can show adaptability and creativity similar to those of humans. It can:
- Develop general skills, rather than techniques for narrow tasks alone.
- Transfer learning across tasks.
- Achieve lifelong learning.
- Exhibit goal-directed behavior, planning, and intrinsic motivation.
From AlphaGo to AlphaZero and MuZero, we have already seen experience-based learning move step by step away from dependence on human knowledge and toward fully autonomous intelligent systems.
Conclusion: A Vision for Autonomous AI Experience
We are entering a new era of AI centered on experience. In this era, agents will do more than reproduce human behavior. They will:
- Explore the world autonomously.
- Learn new skills.
- Continually evolve and adapt.
- Eventually develop capabilities and insights beyond those of humans.
This paradigm shift will transform AI across many fields and lead us to reconsider the nature of intelligence.
Future agents will no longer be merely tools. They will become collaborators, explorers, and even discoverers, potentially leading us into a true era of general intelligence.
Acknowledgments
We thank the pioneers and researchers who laid the foundations for experience-based learning. Their work has advanced technology and expanded our understanding of intelligence, learning, and consciousness.
