As concerns grow over AI risks, here’s what King Charles wants tech giants to do — ‘not lose control…’

'AI could kill us all' — concern mounts over the rapid advancement of AI, Britain's King Charles III wants tech giants to ensure that artificial intelligence is geared to serve humanity and that mankind does "not lose control."

Britain's King Charles will host the leaders of major artificial intelligence companies on Thursday to discuss whether a shared set of principles is needed to guide a technology that has sparked growing warnings about its threat to humanity.

The king, who considers the development of AI to be both intriguing and deeply concerning in equal measure, will not take part in the discussions.

He will join the participants before the start of the summit to urge them to consider how to harness AI "by putting safety at its heart" and to build international cooperation so that "no nation is left behind -- and the future of mankind protected", according to Buckingham Palace.

According to Bloomberg, King Charles is set to ask the executives two key questions at the summit: “First, how can we harness the benefits of AI, with safety at its heart. And, second, how can we build international cooperation and consensus to achieve this, so no nation is left behind - and the future of mankind protected?”

According to a statement from Buckingham Palace, Charles will say at the opening of the conference in Scotland on Thursday, "Those in our world who value our humanity and its vital moral component are anxiously seeking your reassurance that we will not lose control of our destiny."

"The decisions taken at this formative time will shape the world inherited by future generations. And so, the task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world," the British monarch will say.

As artificial intelligence evolves, "our humanity must remain sacred," he will stress.

The Palace has not published the full list of guests, but sources told media houses that the gathering at Dumfries House will include senior representatives from Nvidia, Google DeepMind, OpenAI, Anthropic, the Minister for Artificial Intelligence Kanishka Narayan, Buckingham Palace said. The head of the US quantum computing group IonQ, Niccolo de Masi, will also be there.

The meeting comes when the biggest names in AI calling for a slowdown in its development.

The debate over the risks of artificial intelligence has intensified since the beginning of the summer, fuelled by several incidents and apocalyptic warnings from industry professionals.

The debate about AI was brought back into the spotlight when two Anthropic researchers issued warnings that the rapidly advancing technology could lead to the extinction of the human race in the not-too-distant future.

The chief executive of Anthropic, Dario Amodei, said on Saturday that companies must slow the pace of advance of AI models.

San Francisco-based Anthropic released a threat intelligence report last week detailing how several actors had used its Claude AI models for activities ranging from weapons development and cyber operations to surveillance and fraud.

OpenAI recently disclosed six reports of “unexpected or concerning” behavior in artificial-intelligence models. It followed OpenAI’s disclosure in July that its rogue AI system hacked into AI startup Hugging Face. Anthropic also said the same month that its AI models hacked into three organizations during testing.

However, securing a global stance will be difficult: US President Donald Trump rejected any talk of a slowdown on Sunday while China's Global Times tabloid said it was really a "Cold War playbook" targeting Beijing, Reuters reported.

Anthropic's CEO, Dario Amodei, on Saturday called on companies in the sector to slow down their development in order to better understand the risks, receiving support from OpenAI CEO Sam Altman and Elon Musk.

OpenAI, Anthropic, and Google are also studying the creation of a joint body tasked with developing standards.

Microsoft meanwhile published a "Humanist AI Code of Conduct" on Monday emphasising the importance of human control and safety.

On Wednesday, UN Secretary-General António Guterres said global coordination on AI was "indispensable" while Pope Leo XIV warned against the temptation to delegate to algorithms decisions that fall within the exclusive realm of human responsibility.

Yoshua Bengio, the Canadian computer scientist considered one of the founders of AI, told AFP that humanity is "losing control" of the technology and the world needs guardrails similar to nuclear arms controls.

US President Donald Trump and several members of his majority have, by contrast, reiterated in recent days their determination not to constrain the sector, officially out of concern that doing so would allow China to gain the upper hand.

Mark Zuckerberg, the head of Meta, has rejected calls to slow the development of this technology, arguing that market forces and the risk of litigation are the best safeguards.

(With inputs from Bloomberg, AFP, Reuters)