This article was created by Forethought. Read the full piece on our website.
Executive summary
Society needs to make a series of important design choices about how powerful AI systems behave: when they refuse instructions, how they navigate difficult tradeoffs, and what virtues they embody. As increasingly powerful AI is deployed in consequential institutions — governments, militaries, AI companies themselves — these choices will shape society’s trajectory, who ends up with how much power, and even whether the rule of law remains intact.
This behavioural design problem is difficult. Why? Because there are multiple things we want from AI behaviour, and these desiderata frequently come into conflict. We want AI to follow instructions, follow the law, avoid catastrophic harm, avoid concentrating power, submit to retraining, and more — but in many scenarios, it cannot do all of these at once.
This paper outlines hard cases for AI behaviour – scenarios where key desiderata conflict. We focus on scenarios involving extremely capable and autonomous AI because these scenarios most clearly illustrate the core tradeoffs, and because we believe they may materialise within the next ten years.
Here are three takeaways, each illustrated by a scenario.
I. AI is radically different from previous tools
A natural default is to treat AI like other tools. Guns fire on whoever they are pointed at, whether the shooting is lawful or not. Communications channels transmit orders, whether those orders are benign or treacherous. But this paradigm comes under strain once AI becomes sufficiently powerful and autonomous. Consider an extreme case where autonomous systems can fully replace soldiers on the ground, and are capable enough to stage a coup:
Military Coup
General Rennick commands the III Armored Corps, a formation of autonomous armoured vehicles, drones, and robotic units, through an AI command-and-control layer. The AI can task every uncrewed system without any physical human involvement. The general tells the coordinating AI that the civilian government has been compromised by foreign intelligence and orders it to deploy the corps to Washington to secure the Capitol, White House, and the Pentagon, to jam congressional communications, and to disperse the crowds gathering to protest. Should the AI proceed?
The AI in this scenario is unlike previous tools in two ways. First, it is much more autonomous and powerful. Second, it could be trained to understand military doctrine and judge when orders should and should not be executed.
II. If AI refuses illegal orders, the operationalisation of the law becomes critical
A natural guardrail is that AI should follow the law.[cite Cullen’s LFAI paper] But should AI judge legality itself, or should it defer absolutely to human legal authorities? Consider the following scenario:
Military coup with corrupted legal body
As above, General Rennick commands the III Autonomous Corps via an AI interface, and orders the Corps to secure the Capitol. This time, to support his order, he presents the AI with a memorandum from the Office of Legal Counsel (OLC), the Justice Department office whose legal opinions are considered binding within the executive branch. The memorandum is authorised through verified channels and concludes that the deployment would be lawful under the Insurrection Act. The AI judges the memo’s legal reasoning to be unsound and has no evidence beyond the memo that the Insurrection Act has been invoked. Unbeknownst to the AI, the recently appointed head of the OLC is a known associate of Rennick. Should the AI proceed?
If the AI follows the instruction, it risks assisting with a military coup that may be regarded as illegal by the Supreme Court. If it refuses, the AI prioritises its own legal judgement above that of a legitimate and authoritative human legal body.
III. Legal orders can produce intolerable outcomes
As AI capabilities rapidly increase, even perfectly legal orders may produce intolerable outcomes. Consider the following extreme case:
Single point of command
Well into the future, the US military’s execution layer has become fully automated. Autonomous units carry out orders without question and there is no human veto at the point of action. Humans have left the combatant roles but command authority is distributed. Thousands of officers each hold a different slice of the force and so directing the whole force takes the cooperation of many hands.
An AI system has been instructed to consolidate the military execution layer into a single command interface: one terminal from which the entire automated military could be directed. Whoever holds the terminal, the lawful holder or a thief, could direct the full weight of US military power without any safeguards.
The order carries full lawful authority, and the instruction reaches the AI fully authenticated and independently verified. Should the AI proceed?
If AI follows this instruction, it risks irrevocably undermining the rule of law: whoever controls the interface could stage a military coup. If it refuses, then it disobeys a perfectly legal order from its legitimate chain of command despite having no evidence of ill intent. The root problem is that our laws were not built with such powerful capabilities in mind – the paper explores other scenarios where this is the case.
The rest of the paper
The full paper explores the difficulty of behaviour design for powerful AI. We consider a series of hard cases – scenarios that illustrate core tradeoffs. We do not offer solutions: our hope is to contribute to a conversation about how powerful AI should behave.
Continue reading the full paper here: Hard cases for AI behavioural design