Single points of failure in military AI command

This article was created by Forethought. See all our research on our website.

Abstract

In today’s militaries, no single mistake, compromise, hack, or illegal order can cause catastrophe because the officers and soldiers who would project force are, in very narrow circumstances, asked to exercise independent judgment.

Deployment of advanced AI could change this: as autonomous systems replace the humans in a command network and on the ground, control can collapse toward a single point of failure at the top. In the limit, it would only take one external hack, an insider staging a coup, or a misaligned system to turn a decisive share of the force.

Nuclear command-and-control has faced a structurally similar problem: overwhelming force under the control of a handful of people. In response, nuclear states have developed three types of safeguards that could be adopted for such concentrated control of military AI:

• Integrity of command ensures the force acts only on genuine orders, through cryptographic authentication and the hardening of the systems that convey authority.

• Containment ensures that one compromised node results in a partial rather than a total catastrophe, by limiting how much of the force any one node can reach, and avoiding a single shared vulnerability.

• Soundness of command checks the orders given at the top, by requiring more than one party to commit the force or through downstream refusals based on narrow criteria like legality.

Integrity of command and containment, which protect against external compromise and misaligned systems, are well-incentivized because they protect the control of the commander. Soundness of command, on the other hand, is often resisted because it constrains the commander, who typically has to agree to implement it.

Military command structure balances obedience and independence

Militaries are organized as hierarchical networks in which orders flow from the top down toward the edge where force is applied. In most cases, intermediate nodes decompose high-level orders into lower-level orders, which they then issue to their subordinate nodes. That’s how “Take and hold this island” is gradually transformed into a myriad of low-level orders like “Steer this ship to these coordinates”, “Prepare the landing craft”, and “Engage this enemy target”. The authority to give commands becomes control of the force because subordinate nodes decompose and carry out orders faithfully. As a result, militaries are able to act coherently, even though they’re composed of tens of thousands of individual agents.

This system relies on lower-level nodes to recognize the authority of the nodes above them and to subordinate their own independent will to them. Otherwise, the system stops acting as a coherent agent and degrades into different parts acting at cross-purposes.

At the same time, the potential for each node to act independently is an important safety valve for the rare case in which a higher-level node becomes corrupted. Because downstream nodes can, under some circumstances, decide not to act on an order, corruption does not automatically spread to all subordinate nodes.

In human militaries, this balance is expressed as a general requirement of obedience paired with marginal room for independent judgment, which runs from simply questioning an order that doesn’t fit what a unit sees on the ground, to slow-walking a reckless command, to outright refusal because it is judged to be “manifestly unlawful”.

Automation removes independence

What happens as human nodes in the network of a military are replaced with artificial ones that do not share the same marginal independence and are perfectly loyal instead?¹

This process is already underway at the edge: in the emerging “loyal wingman” model (e.g., the US Air Force’s Collaborative Combat Aircraft), a single crewed aircraft directs a handful of autonomous drones. The human wingmen who could have questioned, hesitated, or refused orders have effectively been replaced with a “loyal” artificial node. The next rung up is already visible: rather than a human directing each drone, a lead node (e.g., a drone mothership) takes a high-level objective like “suppress the air defenses in that sector” and decomposes it into individual maneuvers and engagements across the swarm on its own. Now the intermediate node that coordinates the edge is artificial too, and the human has been pushed one level up, to tasking the lead node of the swarm.

Nothing, in principle, confines this replacement process to a single layer: it’s conceivable to have swarms coordinated by higher autonomous nodes, and those by higher ones still, until the only human left sits at the top, above a fully autonomous network that faithfully decomposes and carries out their will.² Such a military would act with extraordinary speed and coherence. However, the loss of independence as a safety valve also means that any corrupted order would propagate through the network without any friction or obstacle. Ultimately, such a fully integrated network of autonomous nodes would create a single point of failure at the top that, if compromised, would yield control of the entire network.³

This is the most extreme case, but the risk rises before that limit is reached: it is a function of how few nodes control⁴ a decisive share of the force, which, by default, falls with every human removed from the chain.⁵ It’s simply much easier to compromise a handful of nodes than a thousand.

Such a concentration of control could be exploited by three different actors with different affordances:

1. External adversaries. They could attempt to compromise nodes with control over large parts of the network to turn it against the rest of the force (e.g., via hacking).

2. Insider threats. The remaining humans in the network (in the limit, that’s just the top-level authority) could use their control of the network to stage a military coup.

3. Misaligned systems. The autonomous nodes themselves could make catastrophic mistakes, or be misaligned and use their position to disempower the human government.

Pressures toward autonomy and integration

Given this risk, the obvious response is not to build such concentration in the first place. However, it is likely to emerge on its own as the byproduct of militaries pursuing more capability at lower cost. Geopolitical competition may even make that pursuit hard to resist: once an adversary automates and integrates, matching them can become a forced move. As capabilities of autonomous systems increase, pressure to deploy them will increase (both in the application of force and in the command-and-control layer). Their potential advantages include:

• Higher endurance & reliability. Autonomous systems don’t tire, and don’t freeze or panic.

• Reduced political pressure. Having fewer humans at risk is politically valuable in democracies.

• Kill-chain compression. Autonomous systems promise to reduce sensor-to-shooter time.

• Greater and higher-quality information processing capacity. Humans have limited capacity to process and integrate information. Autonomous systems have the potential to remove this bottleneck.

• Higher & faster scale. Autonomous systems can be scaled more easily than manpower, which relies on humans who must be slowly recruited and trained.

• Lower cost-per-unit. An autonomous force may simply be far cheaper than human personnel.

Beyond these capability benefits, there is an organizational pull toward concentration itself: to leaders at the top, marginal independence can seem like a cost rather than a benefit. Given the same capability level, they may simply prefer subordinates who don’t question or slow-roll their orders.⁶

To be clear, even assuming the capability benefits, there are real pressures the other way, too: automation threatens the careers of soldiers and officers as well as the power of military institutions and services. Bureaucracies tend to defend themselves ferociously, which will probably meaningfully slow the slide.

However, if capability advantages and geopolitical pressures keep increasing, this resistance will likely be overcome. For example, when drones first threatened the careers and self-image of Air Force pilots, the service delayed until wartime demand forced a rapid expansion anyway (Secretary Gates likened the process to “pulling teeth”). Such resistance can buy time, and the time is worth using to put safeguards in place.

Nuclear command-and-control illustrates possible safeguards

A nuclear weapon capable of destroying a small country could, in principle, be launched by a single person. In response to this, most nuclear nations have developed a suite of safeguards in their nuclear command-and-control structure. Broadly speaking, they fall into three categories:

1. Integrity of command: Safeguards against action without genuine orders from the top. They ensure that force is only applied in line with the intentions of the command authority. In the case of nuclear weapons, they prevent rogue or compromised operators from launching a strike, accidental launches, as well as adversaries forging launch orders or stealing warheads.

2. Containment: Safeguards limiting the blast radius of a single compromise, even if the first set of safeguards fails. In the case of nuclear weapons, no single crew or launch center controls the whole arsenal, and different systems are locked with different codes, so compromising one does not cascade to the rest.

3. Soundness of command: Safeguards against bad orders from the top. They ensure that only orders that pass certain criteria are given or relayed. In the case of nuclear weapons, some systems require more than one person to authorize a launch, and officers are, in principle, obligated to refuse an order they judge to be manifestly unlawful.

Because nuclear weapons are an existing case of overwhelming force under the control of a handful of nodes, the following sections describe how their safeguards could apply to the case of an integrated autonomous force.⁷ They lean heavily on examples from the United States because it’s the best-understood nuclear command-and-control system.

Integrity of command

Integrity of command ensures that any command that results in force being applied comes from a legitimate authority. It has two core mechanisms:

1. Resilient authentication of authority

2. Node integrity

Resilient authentication of authority

Authentication ensures that any use of force requires a command that verifiably comes from someone authorized to issue it. The standard mechanism is a cryptographic key that confirms their identity. In the nuclear case, this is implemented through mechanisms like Permissive Action Links and Sealed Authenticators in combination with the Gold Codes. In principle, this system could be easily adapted to the fully integrated autonomous force: each machine can be built to require orders to be signed by a specific cryptographic key.⁸

However, there is a key difference from the nuclear case that limits the usefulness of this setup for the autonomous force. In the nuclear case, all nodes only ever need to relay an authenticated order from the top; they don’t need to change or decompose it. That means even the lowest node can check whether the order they receive has been authorized by the top.

This doesn’t work in our case because at least one lower-level node needs to decompose orders from the top: e.g., “Take and hold this island” into “Ship 1 move here” and “Ship 2 move there”. Crucially, they cannot sign these lower-level orders they generated with the key from the top (which they don’t have), and it’s impractical to have the top level approve all such lower-level orders. A decomposition node can only sign them with its own keys, which means it needs to have independent authority that’s recognized by systems subordinate to it. That means when it’s compromised, captured, or misaligned, it can still exercise independent control over the systems subordinate to it, even with the authorization scheme working perfectly. That’s where the second principle of node integrity comes into play.

Node integrity

All nodes that have significant authority over subordinate systems, i.e., orders signed by them carry weight in the network, need to be defended against compromise. Otherwise, their authority could be captured to issue bad but properly authorized orders. Three components need to be guarded:

• Keys. Access to the keys allows an actor to impersonate the node, thus giving them authorized control over all their subordinate systems. That’s why the Gold Codes are so closely guarded.

• Software. Even without access to the keys, the software decomposing orders could be corrupted by an adversary. This needs to be prevented through hardening and audits, just as the nuclear command-and-control infrastructure is secured against intrusion.

• Humans. To the extent that humans are still involved, either in the chain of command itself, or through the training and updating of key software, they would need to be secured against compromise as well—for instance, through regular vetting and a two-person rule, both of which are best practice in the nuclear case.

The safeguards above only protect against a node that is subverted from the outside. They do nothing about the third risk named earlier: the system itself being misaligned. Even with no adversary involved, a decomposition node could turn “take the island” into orders that aren’t what the top intended because it is covertly optimizing for goals of its own. Ensuring that it isn’t is an unsolved technical problem, sometimes called the alignment problem. The safeguards in the following sections are only partial mitigations to it.

Containment

Containment ensures that any compromised node below the top results in partial rather than total catastrophe by limiting the control of any one node and by ensuring no vulnerability is shared among all nodes. The corresponding two mechanisms are:

1. Compartmentalization

2. Decorrelation

Compartmentalization

Compartmentalization prevents a single compromise from cascading into the whole system by limiting the authority of any single part of the network below the top. There are a few ways of implementing this:

• By domain/theater. This would require any node below the top to only reach a limited number of subordinate systems. For example, the system could be set up such that no single node can send commands to both the air force and ground force or to different geographic commands (e.g., NORTHCOM and CENTCOM).⁹ The top-level authority would have to use different keys and systems for sending commands to those separated parts of the force. By design, this would require stopping short of the full integration of command-and-control as imagined by this article.¹⁰

• By scope. Orders from the top, or from any key-holding node, could arrive at subordinate nodes with specified bounds that need to be relayed together with any decomposed lower-level orders (e.g., bounds on geography, time, target class). That would limit how much any compromised node could deviate from the intention of the higher-level nodes.

• By function. Command of the autonomous forces could be separated from access to the underlying system itself (e.g., its model weights and update pipeline). This prevents the commander of the force from easily removing any guardrails placed on the use of the system. Custodial separation is a loose nuclear analogy in which the weapons are held by a different institution from the one authorized to order a launch.

Decorrelation

Decorrelation prevents a shared vulnerability between subordinate nodes that allows an adversary to easily compromise the whole force simultaneously, even without necessarily capturing higher-level nodes. For instance, if the same AI system is used in all drones, inserting a backdoor into this one system compromises the entire force. If a diverse set of systems is used instead, compromising a single type does not result in full control.

This could be implemented through the diversification of models and vendors as well as the separation of update pipelines, each requiring different humans and keys. Notably, this doesn’t have a nuclear analogy because nuclear weapons do not share an identical software layer that could be captured or compromised.

Soundness of command

Integrity of command and containment do not protect against bad orders by the highest authority. Soundness of command addresses that gap: safeguards in this category check the decision at the very top, against an authority that could be hacked, coerced, bent on seizing power, or simply reckless. Because there is, by definition, no higher authority, the only way to check it is to use independent peers or subordinates.¹¹ They constitute the two mechanisms:

1. Command concurrence

2. Downstream refusal

Command concurrence

Command concurrence requires that committing the force takes the approval of more than one party, at least in some high-stakes cases: each party decides independently on the merits whether to approve or not. For autonomous systems, this would mean requiring multiple independent authorizations before a command takes effect: nodes only act on an order carrying several independent signatures. The more independent parties required, the more collusion or coercion it takes to misuse the force, and the harder that is to do quietly.

This has a famous precedent in nuclear command-and-control: during the Cuban Missile Crisis, when Soviet submarine B-59 found itself out of contact with Moscow and under depth-charge attack, a nuclear torpedo launch required all three senior officers present to concur; one of them, Vasili Arkhipov, refused, which prevented a nuclear war.

Downstream refusal

Command concurrence is a broad safeguard: the parties can withhold approval for any reason, down to simple disagreement about what is prudent, as was the case with Arkhipov. Downstream refusal, or escalation, is narrower: it checks the order against a small, fixed set of criteria. Two stand out: legality and factual premises.

Legality. In democratic countries, all members of the armed forces are typically already obligated to refuse manifestly unlawful orders, and are trained to recognize them. Autonomous systems could be built with guardrails such that they refuse to act on orders they independently assess as manifestly unlawful.¹² However, this would require them to make difficult and contextual judgments, sometimes with high-stakes consequences. That means such guardrails should only be included in systems that can make these judgments reliably. A single drone, for instance, might lack the required situational awareness; an autonomous command-and-control system that decomposes high-level orders into mission commands would likely need such awareness to function at all.¹³

Factual premises. Soldiers routinely ask a superior to confirm an order when their own picture of the world contradicts the officer’s. The same could be built into autonomous systems: if their independent sensors show a target to be friendly rather than hostile, they could be trained to ask for human re-authorization, or to refuse outright.¹⁴ Outright refusal is more robust against compromised nodes and insider threats but more vulnerable to spoofing; asking for re-authorization protects at least against accidents and some external attacks. This has a loose analogy in the nuclear dual-phenomenology requirement, under which a launch conference is convened only when two independent sensor families corroborate an attack.

There is a predictable adoption gradient

Every safeguard limits the freedom of some nodes in the network on grounds that run from the mechanically checkable to the purely discretionary. Whether a given safeguard is adopted depends on three questions:

1. How much does it limit the people who’d have to adopt it? (stronger limits mean lower adoption)

2. How predictable are those limits? (less predictability means lower adoption)

3. How much does it impair force readiness or capability? (more impairment means lower adoption)

Strong integrity of command is often mechanically checkable (e.g., cryptographic signatures) and does not limit any relevant options for the commander. For that reason, it is already best practice in nuclear command-and-control. In the nuclear case, there can still be a trade-off with force readiness because minutes, or even seconds, can matter. This will likely be less binding in the case of autonomous systems because they can perform authentication much faster than humans can.

Most containment measures do not directly constrain the top-level authority, so deciders usually have no special reason to resist them; it’s an established military principle precisely because it prevents failure cascades. However, the same pressure toward a seamlessly integrated force that creates the concentrated points of failure in the first place also pushes against keeping the force in compartments. So containment will probably be contested: adopted in some cases and abandoned in others, depending on the specific trade-offs with force capability.

Soundness of command is where adoption typically falls off a cliff because these safeguards limit the highest authority, often the people who would have to implement them, or at least acquiesce to them.

Command concurrence is the most resisted because it constrains the highest authority directly and unpredictably: it introduces another independent decider who uses their own judgment to potentially veto a decision for any reason. The evidence is discouraging: for a US nuclear launch, concurrence does not exist at all. The president holds sole authority even for a first strike (which does not require decisions in minutes), and every attempt to require independent approval has been fought off.¹⁵ The same is probably true for almost all other nuclear nations (except, reportedly, Israel).

The only reason for any hope is that, for an integrated autonomous force, command concurrence is in the strong self-interest of other powerholders, e.g., the legislature, because an autonomous force responding to a single person could be used to overturn the constitutional order itself. Domestic deployment of an autonomous force is also the domain where command concurrence is most plausible and constitutional, e.g., US law already requires something like it for domestic WMD emergencies, where the military can act only on the joint determination of the Secretary of Defense and the Attorney General (10 U.S.C. § 282) rather than sole determination of the President.¹⁶

Downstream refusals also constrain the top and are sometimes resisted for that reason. The extent of resistance depends on the grounds for refusal. The nuclear record is instructive: Major Harold Hering was removed from Minuteman duty and ultimately discharged for asking how he could know a launch order came from a sane president. However, Air Force General John Hyten stated publicly that he would refuse illegal orders when given, seemingly without repercussions. And the dual-phenomenology requirement for attack confirmation has even been proactively adopted.

The problem is in translating refusals into autonomous systems, which are currently much less predictable than humans, lack the required situational awareness, and can be spoofed much more easily by adversaries, with potentially catastrophic consequences. So adoption will depend on whether these problems can be solved, which will probably become easier as the systems become more capable.

At the point when refusals are accurate, predictable, hard to spoof, and have an extremely low false-positive rate, adoption is overall more likely than command concurrence because guardrails run on narrow, clearly defined, and predictable criteria, and can be implemented by defense contractors, even without congressional mandate.

Looking back at the three risks (external adversaries, insider threats, misaligned systems), it becomes clear that adoption incentives are worst for safeguards against insider threats, especially toward the top of the chain of command. Safeguards against external compromise and misaligned systems mostly live in the integrity and containment categories, which have good adoption prospects.

That being said, two other interventions could make adoption of at least some safeguards more likely:

• Transparency. It may be hard to recognize when stronger safeguards have become necessary because concentration of control will likely happen incrementally, with each step looking safe on its own. Reporting of capabilities, usage, safeguards, and force integration, as well as whistleblower protections, can counteract this. For instance, much of what the public knows about the fragility of nuclear command-and-control came from insiders willing to break ranks who often faced prosecution as a result.

• Coordination. Geopolitical competition is a key pressure pushing toward increased autonomy and force integration. Arms-control-style agreements can ease this by keeping a safeguard from becoming a unilateral military handicap. Adversaries share at least the loss-of-control risk, which makes it a common problem rather than a zero-sum one. For instance, the US and China already agreed to maintain human control over the decision to use nuclear weapons.

It’s also important that, once established, safeguards cannot be removed quickly or quietly, so they should be implemented at the most durable level possible. This principle favors technical implementations over simple legal requirements. It also favors statute over revocable policy, and hardware over software because the latter is typically easier to reprogram. It also suggests that safeguards should be paired with verification and tripwires, so that removing or circumventing them is not merely difficult but detectable. Lastly, any safeguard should have a different custodian than the one it is designed to guard against.

Conclusion

For all of history, wielding overwhelming military force has required the cooperation of countless human officers and soldiers. Once autonomous systems replace them, militaries become vulnerable to being turned against what they were designed to protect, because those systems lack the independent will to refuse.

Some countries faced the same structural problem with nuclear weapons, and the record is both encouraging and sobering: we know what the safeguards are, but the ones able to constrain the most concentrated point of failure at the top are least likely to be adopted.

To be clear, the potentially catastrophic version of this problem is not imminent. The raw capabilities for sufficient system integration are years away, and militaries field frontier technology slowly. But that is not a sufficient reason to wait. Precursors of the relevant systems are deployed now. System integration is proceeding now. At the point where safeguards are retrofitted, it may well be too late.

This article was created by Forethought. See all our research on our website.

1. This does not mean that such autonomous nodes are not able to use their own judgment in carrying out orders. They may well be perfectly capable of operating in a “mission command” model where they receive goal-level orders that they independently translate into actions designed to achieve a given objective. In fact, AI agents are already capable of doing so in the context of programming. However, they might lack the independence to question orders, push back on them, or refuse them altogether.

2. In the US, there is no legal barrier to centralizing autonomous command authority in this way. The president has the legal authority to restructure the operational chain of command as they see fit, including giving direct orders to any military officer. There also does not seem to be any legal barrier, in principle, to them having their own private terminal for any weapon system. Advisors and deputies would probably strongly advise against this but are ultimately bound by the executive authority of the commander in chief.

3. One might object that integration is the opposite of a single point of failure. This is exactly how the US military’s flagship integration program, CJADC2, is described: a “disaggregated, distributed, resilient kill web” with so many redundant paths that no single link failing can break it. But this conflates two different kinds of resilience. The kill web is built to survive an adversary destroying parts of it (a property of availability). It does nothing to prevent whoever controls or compromises the command surface from directing the entire force (a property of control). A system can be maximally resilient against losing the force and maximally fragile against the force being turned.

4. “Control” in the sense that there are no other nodes that can refuse their orders.

5. Technically, this does not require replacing any human nodes. It’s also imaginable that the human military stays the same and that an autonomous military is added in a separate branch of the network. If the autonomous branch controls sufficient force, we’d still end up with the same risk.

6. A malicious leader may even attempt to centralize beyond military rationale to serve their own ends.

7. One difference in emphasis matters for the translation. Nuclear safeguards need to cover both the weapon systems at the edge and the command-and-control layer because even a single stolen or accidentally launched nuclear weapon could be catastrophic. The concentrated points of failure of an integrated autonomous force, however, sit only at the command layer; a single compromised drone does not pose a large risk.

8. Like nuclear command-and-control, such a setup would need to be resilient to two edge cases: (1) Incapacitation of the principal. To prevent the force from becoming inert, the authorization scheme would need a pre-defined line of succession. In the case of an autonomous force, this also creates an incentive for those on the lower rungs to incapacitate the ones above in an attempt to inherit control because the inheritance would deliver unprecedented control without independent human checks on their legitimacy. A possible safety valve for this failure mode is requiring independent confirmation for authority transfer, but this re-introduces the possibility of contested authority and risks incapacitating the force before confirmation arrives. (2) Loss of communication. Jamming has become a standard tactic on modern battlefields, and autonomous systems are even a potential solution because they can operate in comms-denied environments. However, this may risk unwanted escalation. The nuclear case illustrates that it can make sense to set different defaults for different action classes in the case of a silent command channel: escalatory commands default to stopping and defensive or retaliatory commands default to continuing. For instance, nuclear bombers launched on warning returned to base without authenticated strike orders, whereas launched missiles have no recall option.

9. This setup does not imply that separate people need to issue the command. It just means that the command has to route through different systems. By default, there will still be a single person at the top.

10. Sensing and information processing could still be fully integrated.

11. Instead of introducing veto points, mandatory transparency of high-stakes orders to independent witnesses could enable opposition to react faster. For example, the Insurrection Act already requires a public proclamation before domestic troop deployment.

12. Short of outright refusal, they could also re-route the order for additional (human) legal review. This also exists in the human case where a soldier may ask for a Judge Advocate to acknowledge that an order is in fact legal. That being said, in the status quo, such legal review does not have the power to overturn orders.

13. Such systems could even be allowed or required to consult a system specialized in such legal assessments.

14. For example, friendly units could identify themselves via a cryptographically signed signal they emit.

15. The one exception is nuclear weapons that would have to be delivered by NATO partners under some nuclear-sharing agreements.

16. Implementing this on a technical level with dual keys would still raise a number of concerns: e.g., how does the system recognize that a particular command requires command concurrence? How does command concurrence impact force readiness? What if keyholders are incapacitated?