TL;DR
Moloch is not a person. It is a failure of coordination, and it needs no malice from any participant — only an environment that cannot coordinate.
- The shape of it: everyone wants the better collective outcome, and competition plus incentives slide everyone into a worse equilibrium together. Pointless overtime and advertising arms races both have this structure.
- The prisoner's dilemma numbers: both cautious pays 8 each; one aggressive gets 12 against 2; both aggressive pays 4 each. Whatever the other side does, aggression is individually optimal — so it settles at (4, 4) when (8, 8) was available.
- What is distinctive about AI is that it compresses the timescale of competition. Higher firm efficiency is not higher firm profit; faster trading is not better returns for every trader. Industry efficiency and participant profit are not the same thing.
- Technical progress does not automatically solve institutional problems. Computational problems yield to compute and data; institutional problems involve parties with different objectives, and even perfect calculation by everyone may not produce cooperation.
- The escape is not asking people to be kinder but changing the incentives that reward defection: make hidden risk observable, make commitments verifiable, bring external costs into the decision — and accept that coordination mechanisms fail too.
1. Moloch is not a person
Moloch comes from ancient religious and literary imagery. Scott Alexander borrowed the name in Meditations on Moloch (2014) for a recurring social mechanism: every participant wants a better collective outcome, and competition, incentives and coordination problems carry them all into a worse equilibrium together.
Overtime, for instance. Every firm would like to cut pointless overtime, but once one of them gains a short-term edge from longer hours, the others may be forced to match. Hours rise across the industry and staff welfare falls, while relative competitive position may barely move at all.
Or advertising. If nobody advertised, every firm would save a great deal. But as soon as one can buy attention, the rest have reason to follow. Everyone ends up carrying higher marketing costs, and consumer attention is exactly as scarce as it was.
This is not to say all advertising is waste, or that competition produces bad outcomes generally. The problem is that some competitive behaviour mainly shifts relative position between participants without creating social value on a comparable scale.
When private benefit separates from social cost, rational behaviour can produce collective irrationality.
2. Writing it out as a game
A minimal two-player game. Each firm chooses safe development or aggressive competition:
| Other is safe | Other is aggressive | |
|---|---|---|
| We are safe | 8, 8 | 2, 12 |
| We are aggressive | 12, 2 | 4, 4 |
Under these assumptions, aggression is the best response whatever the opponent does: 12 > 8 against a safe opponent, 4 > 2 against an aggressive one.
So both settle at 4 apiece, when choosing safety together would have paid 8. A textbook prisoner's dilemma.
These numbers are a model, not an estimate of real payoffs in the AI industry. In reality safety investment may also raise customer trust, cut failure costs and improve product quality, which is a competitive advantage directly — and that changes the matrix.
But the underlying point holds: knowing a better collective outcome exists does not make it reachable. Without commitment, monitoring and credible enforcement, cooperation does not persist.
3. AI amplifies productivity and coordination failure at once
What makes AI distinctive is that it raises not only individual capability but the speed of the whole market.
Scaling a company used to mean hiring, building teams, training and building management structures. Increasingly many tasks go to software agents, so a firm's rate of expansion partly detaches from traditional headcount constraints.
The timescale of competition compresses:
- Market analysis that took weeks may take hours
- Content production that depended on human review can run around the clock
- Strategies a trader executed can be researched and executed by many agents in parallel
The problem appears once every participant has the same capability.
| Looks like it follows | Often does not |
|---|---|
| Firm efficiency rises | → firm profit rises |
| Trading gets faster | → every trader earns more |
| Content production scales | → public discussion improves |
Competitive outcomes depend not only on absolute productivity but on scarce resources, market structure and the relative behaviour of participants.
If technology mainly lowers the cost of competing without changing the incentives to compete, society may simply arrive at the same inefficient equilibrium faster.
Finance shows it plainly. Suppose more and more trading agents hunt arbitrage using the same public signals. Early entrants may do well; as competitors arrive the spread compresses while spending on compute, data and execution infrastructure keeps climbing. Some strategies may end up converting most of their returns into technology infrastructure costs.
Society does get more efficient price discovery. The individuals competing do not necessarily earn more for it. Industry efficiency and participant profit are not the same thing.
4. Why technical progress cannot solve institutional problems by itself
A common technological optimism holds that a sufficiently advanced tool will eventually solve society's problems.
But institutional problems differ from computational ones in kind.
A computational problem, given a clear objective function, generally improves with more compute, better algorithms or more data.
An institutional problem involves multiple parties with different objectives, holding different interests, power and information. Even if every one of them calculates an optimal strategy perfectly, cooperation may still not happen.
Two countries may both know an arms race is expensive and keep spending anyway, because neither can be sure the other will honour an agreement. This is not a shortage of computing ability — it is credibility of commitment, strategic uncertainty and the security dilemma acting together.
AI can help analyse a situation; it cannot dissolve a conflict of interest through better reasoning. A firm with an agent that predicts markets accurately still cannot make its competitors behave in the industry's collective interest.
Intelligence improves decisions. It does not automatically change the structure of the game.
5. How to escape
The key is not asking participants to be kinder but changing the incentives that make defection advantageous.
First, make hidden risk observable. When the market cannot tell a safe system from an unsafe one, firms have little reason to keep investing in safety. Published reliability metrics, independent evaluation and incident disclosure can turn invisible safety quality into a competitive variable the market can price.
Second, make commitments verifiable. A firm stating that it follows a safety principle and an outsider being able to confirm that it does are different things. If a high-risk agent's execution permissions can be constrained by an independent policy engine, the safety claim no longer rests entirely on self-attestation.
Third, bring external costs into the decision. If the loss from a failure falls entirely on users while the firm keeps the efficiency gains, the market will underprice risk control. Liability allocation, insurance, audit obligations and contractual arrangements all change that calculus.
Fourth, accept that coordination mechanisms fail too. Regulation can be captured by large incumbents and turned into a barrier to competition; safety standards can decay into expensive compliance theatre; disclosure can leak trade secrets or open new attack surfaces.
So a good institution must not only constrain participants but be transparent, contestable, updatable and capable of correcting itself.
6. The infrastructure that matters may be coordination infrastructure
If the software revolution mainly lowered the cost of processing information, the agent era may lower the cost of acting and organising.
We are heading for a world with many software entities that can research, negotiate, procure, trade and execute independently. That produces an important shift:
The number of acting entities in society may grow sharply, while trust, oversight and accountability mechanisms do not scale with them.
One firm may run thousands of agents; one user may authorise several to act for them in different markets; agents belonging to different organisations may trade, collaborate and exchange information directly.
When action gets cheap, coordination and accountability become the bottleneck.
So the infrastructure worth building is not only models, compute and data, but verifiable identity, bounded authorisation, shared execution protocols, independent audit, dispute resolution, and credible cross-organisational coordination.
None of this necessarily requires a blockchain, and none of it necessarily comes from government — it may come from open standards, market contracts, industry consortia or verified software. What matters is that it genuinely changes incentives and constraints rather than wrapping an inefficient structure in a new technical shell.
7. Civilisation is not only about making everyone stronger
We tend to read progress as progress in tools: more powerful machines, more efficient production, more accurate prediction.
But civilisation has a second path of equal importance: getting individuals with conflicting interests to cooperate. Property rights, money, firms, markets, law and the scientific community can each be read as a coordination mechanism humans invented. None is perfect; together they made cooperation at scale possible.
AI may produce a society where individual intelligence and capacity to act rise dramatically. But a system made of many clever actors is not necessarily a clever system.
If we cannot change bad incentives, then the stronger the intelligence and the fiercer the competition, the faster collective failure may arrive.
The real challenge is not only building smarter machines, but building institutions that give increasingly capable actors a reason to keep cooperating.
Second in a four-part series on the problems of the AI era, on coordination. The others: cognition, power, institutions and distribution.