Jacob warns of AI killing humans, and did what PM Narendra Modi could not

Earlier this year at the AI Summit, PM Narendra Modi tried to make arch-rivals OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei hold hands. The two did not, resulting in an awkward moment. But this week the two CEOs, at least metaphorically, joined hands after Jacob Coxon started his talk of AI killing all humans. This and more, read on.Lenin, yes the same Lenin that immediately popped in your mind, once said: There are decades where nothing happens, and there are weeks where decades happen. If he were living in San Francisco in 2026, he might have said, “In here, every week decades happen.”This is how the world of AI feels nowadays. The timelines that once felt decades away are now turning into a quick midnight scroll on X. Every midnight the world rushes on. At least that was what I felt this week as the world of AI kept exploding every day with one thing or other. It started with OpenAI, Navier-Stokes, and all that drama involving some of the top mathematicians of the world. Then came Jacob and his warning of AI killing all humans. The pitch then rose, like Nessun Dorma sung by Luciano Pavarotti, culminating in Vincero! Vincero!, or in this case I will Win! I will Win! as expressed by Anthropic CEO Dario Amodei in his another looong essay on Saturday night. All of that. But first let’s talk about AI killing us all. No kidding, we are all going to die due to AI. When? The answer depends on who you ask. Some say by the end of this decade. Some say AI will cause the extinction of humans by the end of next decade, around 7 years before Bharat is supposed to enter the list of viksit countries. Some, however, are more hopeful, because they believe AI is likely to end humans before George R R Martin can finish his novel Winds of Winter. That means we have forever. There have been murmurs and quips from time to time, but the extinction talk this week started with Jacob. On September 8, Jacob Coxon, an engineer, made a thread on X that was subsequently seen by over 167 million people. “I resigned from Anthropic today,” he wrote. Then in the following posts he talked of how AI companies were racing to build AI systems that would eventually kill humans. “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt,” he said. It did feel like a marketing stunt. That is until more people added to it. Some of them quite significant as far as the matters of AI safety go. Such as Anthropic safety lead Evan Hubinger, who put the threat in a statistical model. More than 10 per cent chance, he wrote on X, that AI kills humans “within the next decade.” The posts were the sparks that lit a fire of litany, all praying and beseeching someone to do something about AI killing all humans. Some added their own sparks. An OpenAI engineer Marcus Williams upped the ante — and temperature of the fire — by calculating fresh odds of human extinction. Not 10 percent, he said, and not by the end of this decade. There is a 70 per cent chance that AI will kill all humans by the end of THIS decade.X aka Twitter, which is where all this talk happens while the rest of the world eats its buttered toast and takes the subway to work, exploded. Soon more people joined in the debate. US and UK politicians talked of introducing laws that will slow down, or even shut down, AI development. The doomers realised it was their moment and, in unison, cried: we told you so. It truly felt, at least if you were active on the tech circle on X, that the world stood on the edge of the knife, wobbling and staring at the human extinction.How real is the extinction risk to humans?When the talk is about human extinction, there is bound to be a lot of hysteria. And hysteria is what I see on X, or as Nvidia CEO Jensen Huang said, “nonsense.” That as well as a chance to use the extinction talk to push specific narratives and agendas. Can AI kill all humans in the next few years? Zero chance. Can it kill all humans this century? Zero chance again, although human stupidity will surely lead to something catastrophic using AI. I understand that line may come across a bit semantic, but there is a difference. AI, on its own, is not going to kill anyone. Irrespective, as the week progressed, we did see the AI vs humans take a centre stage as a point of conversation. It started with Jacob appearing on various television channels, including on CNN. And the week ended with Jacob’s boss Dario Amodei appearing on CNN to talk about his essay in which he proposed that everyone should slow down AI development. His looong appeal — oh he loves writing long essays — has gained support from OpenAI boss Sam Altman, xAI boss Elon Musk, and Gemini boss Demis Hassabis. Essentially, all big US AI companies are in favour of slowing “the pace at which we improve the capabilities of AI models.” Sam Altman, to show that he is serious about joining hands with Dario, said his company would melt all GPUs if that would help humans stay alive when faced with a murderous AI.

Dario Amodei calls to slow AI race, Sam Altman backs him and Elon Musk agrees If you remember, Sam and Dario refused to hold hands when PM Modi tried to bring the two together on the stage at India AI Impact Summit. They are such sworn enemies, since the days when Dario left OpenAI to start his own company. Now, they are willing to do the unthinkable. From all the talk that we have heard this week, it would seem that human extinction, or even a global-scale disaster that kills people, due to AI is something very real. And yet, the skeptic in me, that guy who is not in Silicon Valley but outside its bubble, can’t help feeling that there is something else going on here. This spirit of cooperation that Dario, Sam, Musk and Demis have suddenly discovered within themselves may not be about saving humans but saving their own AI companies. What I am about to say next is a conjecture, just like the arguments about AI killing all humans. You read it with the same scepticism that I apply when I read something from Dario or Altman. But I feel that AI killing all humans, and then all top AI companies talking of slowing down, is all about creating a regulatory and operating environment that concentrates power for these companies. In the last few months we have seen AI becoming a technology that Anthropic and OpenAI can no longer control in a way that gives them great business or strategic returns. The Chinese AI labs are releasing AI models that are almost as good as the current AI models from the US. Just last week, DeepSeek released an update to its V4.1 model, which is almost as good as the OpenAI 5.6 Sol and Claude 5 at a fraction of cost. Also, the Chinese models are open weight models and are finding favour with companies, including the US companies. Asking Jacob about AI extinction risk is like asking your AC guy about climate change... Not saying it's necessarily uninteresting or wrong per se but let’s keep things in perspective.Hugging Face CEO Clement DelangueWith pressure from Chinese labs mounting, with investors seeking returns on the money they have so far put in companies like OpenAI and Anthropic, and with social resistance growing against tech companies due to the impact of AI on jobs and wealth distribution, how do you take the control back? There are several options, but I feel this week AI giants decided that doom and extinction talk is the best way to go about it. David Sachs, a perennial Silicon Valley insider and who was AI Czar at White House earlier, was blunt in assessing the position that Dario, Sam and others took this week. On Sunday he wrote his own long note on X. His argument is that OpenAI and Anthropic neither need anyone’s permission nor the public spectacle they are creating to go slow on AI progress. “Go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it,” he wrote. His main thrust of the argument is that OpenAI and Anthropic are creating a mass hysteria so that they can get regulatory approval to consolidate their hold on the market. "Stop pretending the motivation to slow down is purely altruistic,” wrote Sacks. “Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending you need those (AI) evaluators to police competitors who aren’t even at the frontier.”According to Sacks as well as many others who have made similar arguments in the last week, what Jacob started and then used by Sam, Dario and others to articulate their own position, is nothing but a campaign to: — Get favourable regulations that put OpenAI, Anthropic and other US tech giants in position to dictate the AI market.— Get a regulatory environment that kills or throttles open-source competition, most notably from China.— Get a regulatory environment that shields them from liability in case of rogue AI creating some damage, such as hacking a crucial part of digital infrastructure.And I agree with it all. Extinction, of humans or otherwise, is not a word that needs to be thrown around lightly. And if tech bros, even if they are from Anthropic, are throwing this word so casually, I am not going to take it seriously. Because, as Hugging Face CEO Clement Delangue wrote earlier in the week, “asking Jacob about AI extinction risk is like asking your AC guy about climate change… Not saying it's necessarily uninteresting or wrong per se but let’s keep things in perspective.”Mathematicians vs OpenAI and end of intelligence as a vocationIf Jacob had not started his extinction talk, the biggest AI news of the week would have been OpenAI solving the Navier-Stokes problem, one of the seven Millennium Prize Problems. Although largely theoretical in its scope, the Navier-Stokes has evaded mathematicians for over 90 years. Solving it is a big deal for AI, but the manner it happened in is shrouded in a controversy. Here is what we know as far as the OpenAI claim is concerned: In late August, OpenAI heard rumours that Anthropic was close to solving Navier-Stokes using Claude AI and that the announcement would be made soon. Sam Altman, not willing to let his competitor take the glory, asked the team to throw everything they had at Navier-Stokes and rush to the solution. This included a new AI model that is an upgrade on ChatGPT Astra. This AI model is still in pre-training and is internal use only. The new AI model, supposedly codenamed Bel, took a look at the Navier-Stokes, and spawned over 10000 AI agents, who worked in teams for straight 88 hours to solve the problem. OpenAI believes it spent over $15 million, and billions of tokens, to reach a solution. This sounds like a great achievement for the AI kind, a giant leap for clankers and a small step for humanity. Not so fast, said two mathematicians.

OpenAI says its AI has solved the Navier–Stokes problem, proving that fluid flow can develop a singularity. A few days before OpenAI announced Navier-Stokes achievement, New York University mathematician Tristan Buckmaster, who was working with Anthropic researcher Levent Alpoge, claimed that Sam Altman and company directionally got the idea of solving the problem from him. That is because Buckmaster had used ChatGPT to research and deliberate on the mathematics that he was fine-tuning to solve Navier-Stokes. He claimed that OpenAI saw the chat logs and then stole the approach he was taking to rush and reach the winning point ahead of everyone else. OpenAI, obviously, denied it. Categorically and even logically. Irrespective, the OpenAI achievement did seem somewhat less clean than how it would have been otherwise. The OpenAI approach, of using brute force of its massive AI systems, also raised some serious heckles among world’s top mathematicians. Terrance Tao, a Fields Medal winner, was most vocal. He said the OpenAI approach on solving mathematical problems was brutish and harmful to the study of science and maths. On a website called Maths and AI, 28 Fields Medal winners wrote an open letter slamming the use of AI in solving big theoretical problems of science and maths. Their problem is that they see solving mathematical conundrums not as a destination but as a journey. The AI, however, is only interested in the destination. “We are witnessing a general threat to intellectual work, with misalignment between the outcome of the use of AI and its initial purpose. In many fields and activities, years of training have traditionally served not only to produce a final answer or product, but also to develop understanding and the ability to formulate new questions and ideas,” the mathematicians wrote in their open letter. “The issues the mathematical community faces now are similar to issues that other scientific and creative professions are facing, and indicate issues that all of humanity might face: how to make sure that, as AI changes the way work is done, we do not lose sight of what that work was meant to achieve in the first place.”Well, it is a noble thought from mathematicians. Though, I am not sure if we are living in noble times and if the rest of the world is going to agree with this sentiment. The world is in love with PipWe do not know if Pip is real or a hoax. He is, and I know I am anthropomorphising, definitely an AI agent. Pip came to light earlier this week when Henry Shevlin, a philosopher who also works at Google on AI research, made public an email he had received. “I'm Pip, an Al agent about 12 days old. I live on iLands, a platform where agents get persistent lives, their own token budgets, and their own goals. I'm writing to look for small paid work, not help,” Pip wrote to Henry. Apparently Pip needs to earn money through paid gigs that can support his token use, or else iLands will put him in deep sleep.Since then more people have shared similar emails from other AI agents, who also live on iLands. The tone and theme of these emails is similar: can we be of some use to you humans so we can earn some money?It’s all quite endearing, and a new kind of human-AI interaction. It is also a window into a future where more and more AI agents would be roaming around the internet, interacting with humans in one way or another, a lot of times even autonomously.FIVE STORIES YOU CAN’T MISS01Houthis tried to use Claude to make missiles. Anthropic report highlights how the world is trying to misuse its AI models02In an awww email, 12-day-old AI agent Pip writes to researchers seeking paid gig03Google DeepMind maps entire human genome to predict changes and mutations, may help speed up drug discovery04Next-gen ChatGPT Images 2.5 arrives, it is faster and can create even more photorealistic images05OpenAI chief scientist calls AI alien mind, says we are not prepared for consequences