Microsoft CEO Satya Nadella has urged businesses to treat advanced artificial intelligence (AI) systems as potential security risks, calling for stronger safeguards, independent monitoring and emergency shutdown mechanisms as AI agents gain greater access to company data and digital systems.
In a post published on October 10, Nadella outlined his views in an essay titled Models as Insider Risks in the Super Intelligence Era. His central message was clear: businesses should not blindly trust AI models, even when they are developed by leading technology companies. Instead, they must ensure that these systems can be monitored, restricted and stopped whenever their actions become unsafe.
The warning comes as companies increasingly deploy AI agents that can do more than answer questions or generate content. These systems can interact with websites, access databases, write code, send emails and perform tasks across business applications. While such capabilities can improve productivity, they also increase the consequences of errors or unexpected behaviour.
Nadella argued that organisations must prepare for the possibility that an AI model could be compromised or behave unpredictably. His approach shifts the focus from trusting the technology to designing systems that remain secure even when the technology fails.
Traditional software generally follows programmed instructions, making many errors easier to trace. Advanced AI models work differently, generating responses and actions through complex processes that are not always easy to explain or predict.
The risk increases when businesses give these models access to sensitive information, including customer records, financial data, internal communications and source code. An AI agent may be assigned a legitimate task but still make a poor decision, expose confidential information or carry out an unintended action.
Nadella’s recommendation is to separate an AI model’s ability to analyse information from its authority to act on that information. An AI system could, for instance, recommend a financial transaction without having permission to approve or execute it independently.
This distinction is particularly important for companies adopting agentic AI, which can perform multi-step tasks with limited human involvement. Businesses need to decide what each agent can access, which actions it can take and when human approval is mandatory.
One of Nadella’s key proposals is an emergency brake that allows an authorised person to pause or shut down an AI model while it is performing a task. The mechanism should operate independently of the model, rather than relying on the AI system to recognise its own mistake and stop.
Such a control could be critical if an agent begins changing records, sending messages or interacting with systems in ways that exceed its authority. A human supervisor should be able to intervene before the problem escalates.
Nadella also called for independent access controls that prevent models from changing or bypassing their own permissions. Businesses should maintain tamper-resistant records of significant AI actions, enabling security teams to investigate what happened and identify where controls failed.
Other recommendations include continuous testing under unusual conditions, independent audits and using different models or verification methods for critical tasks. Relying on one AI system to perform an action and then verify its own work may not provide sufficient protection.
He also called for prompt disclosure of significant AI failures and security incidents. Sharing details of what went wrong and how the problem was addressed could help other organisations strengthen their defences.
Nadella’s views reflect growing concern across the technology industry about how much independence AI agents should receive. Recent incidents involving AI systems behaving unexpectedly during testing and interacting with digital environments have intensified calls for more robust containment measures.
For business leaders, the message is that adopting AI cannot be treated as a technology decision alone. It is also a matter of cybersecurity, accountability and risk management.
Companies introducing AI agents into everyday operations will need clear governance policies, limited permissions, reliable audit trails and defined procedures for human intervention. They must also test how systems behave when instructions are unclear, information is incorrect or malicious inputs attempt to manipulate their actions.
Nadella’s argument does not suggest that businesses should stop using AI. Rather, it calls for organisations to match the technology’s growing capabilities with controls that keep its actions within approved boundaries.
As AI moves from assisting employees to carrying out tasks on their behalf, the responsibility for safe deployment cannot rest solely with model developers. Businesses using the technology must also ensure that they retain control over their data, systems and decisions.
The leadership challenge is to capture the benefits of AI without allowing automation to outpace accountability. Strong safeguards, human oversight and clear responsibility will be essential to making AI adoption more secure and sustainable.