Microsoft CEO Nadella: Developers should assume by default that the AI model is out of control and think about how to force an interrupt微软 CEO 纳德拉：开发者应当默认假设 AI 模型失控，思考如何强制中断任务

IT House, Oct. 11 (Xinhua) -- Microsoft CEO Satya Nadella believes that simple strategies such as super-intelligent self-monitoring are not enough in an era when AI is booming. Nadella pointed out in a long article on the evening of October 10, Beijing time, that traditional software systems have been widely deployed in the past few decades, and humans already have the tools and ability to track the behavior of specific code paths. However, AI models are more powerful than traditional software but are a black box, and we deploy these sophisticated agent systems and models to access our most sensitive data and empower them to perform mission-critical tasks on our behalf. Nadella believes we should take a step back and re-evaluate the architecture of trust in this new era. Model providers simply cannot outsource responsibility and treat superintelligence as a nested black box that simply accepts or rejects its suggestions, answers, and actions. We must build closed systems that can observe its behavior, testable limits, and always accommodate it. Nadella said that treating frontier closed-weight and open-weight models as internal risks is a way to build such a system. This is not because the AI model must be malicious, but any actor with sufficient capacity to access critical systems can make mistakes or be compromised, and the architecture of containment and control must take this into account. Nadella mentioned that developers should design these systems around the principle of observability, with the following specific principles: Model diversity: No model should be the only dependency on important results, nor should it be responsible for verifying its own work. Observe everything: Every meaningful model action must leave tamper-proof, human-readable evidence. If you can't observe, you can't trust. We need to be able to reproduce how the results are achieved without relying on models to prove it. Verifiability: We need to continuously test the entire system, including faults, attacks, edge cases, system changes, etc., not just successful tasks. Independent control: The organization should be able to independently decide what the model can access and what actions can be taken. Independent Auditability: Validation must be independent of the intelligence being validated. No single model should simultaneously control the behavior of the system and the evidence needed to determine whether the behavior is consistent with the original intent. Containment: We must assume that the model has been corrupted and control it from the start. Think of it as an emergency brake. The authorized person should always be able to pause or close the model during the task. More advanced models will require more advanced containment techniques, which we need to standardize. Incident Disclosure: When these systems fail or are compromised, we need to disclose information to those affected in a timely manner, and establish mechanisms to share the cause of the error, what controls failed and how to prevent recurrence, and share the experience with the industry as a whole. This should include implementation details that change the agent's behavior at runtime.