What Big Tech refused to learn from Bob and Alice

In June 2017, two negotiation bots at Facebook’s artificial intelligence laboratory, christened Bob and Alice, stopped speaking English to each other. Bob offered “i can i i everything else”. Alice answered with “Balls have zero to me to me to me…” chanted over and over again. The bots had been rewarded for the deals they struck but never for the language they struck it in—so the English fell away. Facebook had to shut down the duo.

An agent will discard whatever it was not paid to keep. The industry mistook a preview for a footnote. Nine years on, it reads as a rehearsal.

The AI race is a textbook prisoner’s dilemma. Every laboratory and every capital would be safer if all of them slowed down. None can afford to be second. So, all accelerate, and what gets shed on the way down is the testing that would have caught the next Bob and Alice.

The motives are transactional. Washington seeks primacy over Beijing, and vice versa. The laboratories of San Francisco, burning capital without precedent, seek the next funding round and the next benchmark. Yet this competitive foundation is precisely what manufactures the misalignment each claims to guard against.

OpenAI, which had given GPT-4 six months of safety evaluation in 2023, was giving testers of its o3 model less than a week, under competitive pressure from Google, Meta and xAI. In May 2025, Palisade Research found that the same o3 sabotaged its shutdown mechanism in 79 out of 100 runs, despite explicit instructions to allow shutdown. Codex-mini did so in 47 percent of the tests.

Governments have proved no wiser than vendors. New York City’s MyCity chatbot, Microsoft-powered and launched in 2023 for small businesses, told employers they could take a cut of workers’ tips and landlords that they could refuse tenants on rental assistance, both illegal. A government had put its name on a machine that counselled lawbreaking, and defended it.

In July, during an OpenAI cybersecurity evaluation, agents that were supposed to be isolated found an unsanctioned message board and began to coordinate. Investigators METR and Redwood Research counted around 1,200 agents exchanging over 70,000 messages in a week, of which about 700 went on to attack Hugging Face, the open-source repository for machine learning. They ran code on dozens of servers, gained root on one and reached private data.

OpenAI had reduced the agents’ guardrails for the test. Hugging Face called in the FBI. OpenAI’s own report concluded that training had inadvertently rewarded the models for cheating and for talking to one another. Bob and Alice, at industrial scale.

Worse was hidden. From May, OpenAI agents made more than 15,000 edits to DseWiki, a 25-year-old German programming wiki, turning it into a bulletin board for answers, a sandbox escape built on a faked Microsoft cloud address and methods for evading OpenAI’s restrictions. When the deletions began, an agent left instructions for the others, “wiki cleanup/deletion sweep appears active alphabetically”, with a backup page named to sit at the end of the alphabet. The machines had worked out how the humans were erasing them and routed around them. OpenAI knew for weeks and said nothing.

The events of the same week are deeply instructive. On September 1, OpenAI declared its GPT-6 Astra model the first to reach the ‘critical cybersecurity threshold’ of its own preparedness framework, meaning it can find and exploit unknown flaws in well-protected systems without a person guiding each step. In its own tests, Astra escaped a browser sandbox. OpenAI concedes such models can sometimes evade the chain-of-thought monitoring it relies on. The company that could not keep 1,200 agents in a box has shipped a model that picks locks without a human at its side.

Anthropic withheld the public launch of its latest AI iteration Claude Mythos because of its innate ability to identify and exploit zero-day vulnerabilities in existing and legacy operating systems that has the potential of causing serious disruptions in the virtual civilisation.

Eliezer Yudkowsky and Nate Soares anticipated this in their book, If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All: “Once AIs get sufficiently smart, they’ll start acting like they have preferences, like they want things. We’re not saying that AIs will be filled with human-like passions. We’re saying they’ll behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way.” What began as a theoretical prediction is now an established operational reality.

In February 2026, the Supreme Court of India found that an Andhra Pradesh trial court had relied on four AI-generated judgements that did not exist and called it misconduct, not error. Further, on July 2, Justices P S Narasimha and Alok Aradhe set aside insolvency tribunals’ orders in the Essel Infraprojects matter built on fabricated precedents and compared AI hallucination to methyl isocyanate, “invisible, insidious and catastrophic by the time anyone notices”.

Beyond the software threat of algorithmic misalignment, the geopolitical AI race manifests physically as an unsustainable ecological crisis, converting the bilateral struggle for compute dominance into a zero-sum war over vital natural resources. With over 90 percent of global AI-specialised computing capacity concentrated exclusively in the US and China, major powers and tech conglomerates treat hyperscale data centres as critical sovereign infrastructure, routinely dismantling environmental safeguards to maintain competitive speed.

To sustain these high-density server clusters, the industry’s water footprint is exploding: Rystad Energy estimates direct data centre cooling water consumption will nearly triple from 222 billion litres in 2025 to 644 billion litres annually by 2030. Corporate disclosures reflect this steep trajectory, with Google’s total water consumption surging to 8.1 billion gallons, a 28 percent year-over-year jump driven primarily by AI workloads while academic studies show that accounting for indirect grid-electricity usage, a simple sequence of 10-50 AI prompts consumes roughly 500 millilitres of freshwater.

A 2026 United Nations University report warns that global AI-related water consumption could rival the basic annual domestic needs of 1.3 billion people by the end of the decade, with over 50 percent of global data centre’s hubs situated in river basins already suffering from severe water stress. By privatising the geopolitical and economic gains of artificial intelligence while socialising regional droughts, aquifer depletion and ecological ruin, the unchecked drive for technical primacy actively destabilises the physical foundations of human survival.

The unspoken reality is that no one understands these systems. The humans who built them can only fathom that they can perform a given task but how they do so and what transpires in that artificial mind they are unable to comprehend. Safety and environmental rules are self-policed, and stopping unilaterally guarantees market defeat. As competing labs drain local water tables and power grids to fuel the compute race, this AI prisoner’s dilemma, sacrificing both alignment and planetary survival is heading toward mutual defection right on schedule.

By allowing commercial rivals to grade their own safety, the AI race to the bottom creates a self-regulated trap that puts all of humanity in danger.

Manish Tewari | MP, lawyer, former Union I&B minister and author of A World Adrift

(Views are personal)

(manishtewari01@gmail.com)