A nightwatchman on every probe

Superintelligent surveillance to prevent galactic anarchy

This article was created by Forethought. See the original on our website.

One utopian vision for space expansion involves a libertarian explosion of diversity, with everyone and their robot setting up different sorts of societies and governance structures.¹ We might call such a future ‘galactic anarchy’.

On Earth, while there is anarchy between nation states, within a country the government enforces contracts, protects property rights, and provides for collective defence against external aggressors. A government that does nothing else (e.g. provide a social safety net) is Nozick’s ‘nightwatchman state’. Without a Leviathan to uphold the rule of law, we are left with Hobbes’s war of all against all.

But in the cosmic future, time and space make it far harder to have a centralised Leviathan punishing violations ex post. Suppose an aggressor conquers an already-occupied galaxy. It may take millions of years for nearby galaxies to learn about this infringement and prepare a response. And whose job is it anyway to go and punish the aggressors? That might be difficult and costly, and all the nearby galaxies prefer to mind their own business while strengthening their defences. Any inter-galactic governance system that relies on observing and punishing wrongdoing across millions of light-years is doomed to failure. Justice must be swift. And swift justice must be local,² not from the next galaxy over.

If space is defence dominant, each star (or galaxy) can self-govern in peace without much fear of being conquered or destroyed by a neighbour. Even so, there are three huge negative externalities that a single rogue star system could impose on the rest of the galactic community:

• Unconstrained expansion. In a free-for-all—where anyone can use whatever resources they can grab and defend—people could be incentivised to race as fast as possible to be the first to lay claim to as many resources as possible, even if this involves ‘burning the cosmic commons’ by wasting resources to accelerate more and more probes closer to lightspeed. It’s pretty unclear how significant this waste would be—how feasible would it actually be to quickly harvest most of the energy from a star to speed up probes or launch even more of them?—but in the worst case, this could eat up a lot of the possible value and favour the spread of ‘locust’ value systems that only want to acquire and consume resources.

• Galactic X-risks. Jordan Stone catalogues many (scientifically uncertain) possible ways that a reckless or omnicidal civilisation could drag all others in its lightcone down with it (the ‘vulnerable universe hypothesis’). For instance, it may be possible to trigger false vacuum decay, destroying all particles in a bubble expanding at light speed.³

• Hopefully physics does not allow for this. But it is possible that at technological maturity, rogue star systems or galaxies could destroy the universe. If so, retroactive enforcement and punishment won’t suffice.

• Suffering risks. A misguided or malevolent civilisation could create astronomical amounts of suffering, many orders of magnitude worse than e.g. factory farming on Earth. Given defence dominance, the rest of the universe may be powerless to intervene and punish, or even stop, this.

Even if the chance of any given star system going off the rails in one of these ways is very low, across billions of galaxies each containing millions of star systems that are slowly diverging from their parent civilisations through cultural evolution and drift, it is a statistical inevitability.⁴

There are also some big upsides that might be harder to achieve under galactic anarchy, such as long-distance moral trade. Suppose that Earl on Earth and Andy in the Andromeda galaxy agree that Earl will spend some resources on hedonium and Andy will spend some resources running copies of Earl. It will plausibly be difficult by default to verify that counterparties are holding up their end of the bargain (from space, compute running hedonium plausibly looks about the same as compute running Earl uploads, so it might be hard for Earl to be confident that Andy is actually running copies of him and not hedonium).

And, even with verification, it might be difficult to enforce contracts from a distance, especially if there are time lags that make it slower to do a tit-for-tat strategy. Suppose that Earl received word that Andy was not running Earl-uploads, 2 million years later. Earl has been faithfully running hedonium for the past 2 million years, so even if Earl stops now, Andy has gotten a lot of value out of Earl. And it might take even longer to propagate the news that Andy is a cheat to all of Andy’s (potential) trade partners.

Funding public goods is also difficult under galactic anarchy. Currently, these are funded by governments coercively taxing their populations and then voting on how to spend the tax dollars. It’s possible that there will be galactic-scale public goods—scientific research, defence against aliens, or setting aside resources for pursuing impartial moral purposes—and the coercive taxation model might be the best way to fund them.

What then can we do? One solution⁵ is mentioned in passing in AI 2040’s space governance supplement:⁶

One way to do this [enforce property rights] might be to require that all probes sent to colonize other star systems carry a nightwatchman ASI to prevent the colony from sending out probes to seize space resources that belong to others.

Given the vast distances of space, the monitoring and enforcement governance regime must be dispersed throughout the inhabited universe, rather than in a distant galactic capital. We need to prevent violations before they occur, since we can’t punish them ex post.

Every single inhabited star system should have an unchallengeable governance system that can with 100% reliability⁷ enforce the universal code of respecting property rights, not destroying the universe, and not creating astronomical suffering. Perfect reliability is not achievable today, but computational systems with backups, error-correction, and formally verified software systems may allow this.

For concreteness, here’s a sketch of a proposal:

• Every probe leaving the solar system would be required to carry an ASI nightwatchman.⁸

• • This might entail delaying when probes are sent out, so that the design and ruleset of the nightwatchman can be decided upon after sufficient reflection. It may also be desirable to delay space expansion until we are near technological maturity, such that we know better what risks the nightwatchmen must defend against.

• The nightwatchman would maintain a decisive strategic advantage in the colony established by the probe. This might require that the nightwatchman monitor industrial build-up in the system and the creation of new ASIs within the system, to ensure that they are either aligned to the nightwatchman or lack the ability to interfere with the nightwatchman.

• The nightwatchman would prevent probes from leaving that star system unless they also carry a copy of the nightwatchman. It would also prevent probes leaving the system that violate any anti-racing rules that have been set to avoid burning resources wastefully.

• The nightwatchman would prevent anyone in the colony from carrying out any prohibited activities. This probably requires that the nightwatchman or a trusted delegate be able to audit how compute and other resources are being used.⁹

• Optionally, when making voluntary agreements with people in other star systems, people can make such contracts binding by specifying that the nightwatchman will monitor and enforce the contract. This allows positive-sum trades to be made that otherwise could not be.

• (Maybe) The nightwatchman collects taxes and spends them on public goods.

• The nightwatchman should receive updates to its policies from legitimate authorities.¹⁰

  • E.g. if everyone votes that a new prohibition be added to the list,¹¹ then the nightwatchmen in every star system change their policy in response.

  • E.g. if someone discovers a new way of destroying the universe, then this information should be shared to all the nightwatchmen so that they know to monitor for and prevent it.¹²

We’re obviously leaving a lot of governance questions unanswered—how people decide on prohibitions, how people choose which public goods get funded, etc. The space governance supplement has a bit more discussion on some possible answers to these questions. The ‘nightwatchman’ proposal is intended to be a platform on which many different governance systems could be built.

This proposal has some drawbacks. Here are a few of the most important ones.

It’s a lock-in event, and it would be bad if we locked in a bad policy for the nightwatchmen. If the code is too expansive, and too hard to update, it may close off moral and social progress and innovation. We are certainly glad that no government has historically had the power to lock in a legal code indefinitely. But if the code is too minimal, or too easy to update, rogue actors will find a way to route around the letter of the law, or simply alter the code in undesired ways.

The AI Futures Project’s space governance appendix discusses some possibilities for how the code could be chosen and revised (see the section titled “Mitigating downside risks”). In any case, we think that the final version of the code—if there is a final one—should only be decided after careful reflection.

That said, the nightwatchman proposal tries to preserve as much future human agency as possible without imperilling galactic civilisation. Alternate proposals could give up altogether on diversity (e.g. a ‘hedonium shockwave’) or freedom (e.g. a sovereign AI that is immensely wise and makes all important decisions in each star system). So this may be close to the minimum amount of lock-in necessary to avoid catastrophe.

It introduces a single point of vulnerability among all the colonies. If all the nightwatchmen were identical and it turned out that they had a bug, then we would be in trouble because they have a decisive strategic advantage over all the star systems we’ve settled. We might be able to mitigate this by designing different nightwatchmen in very different ways, even if they would all ultimately be enforcing the same policy. But the more different nightwatchmen designs there are, the higher the chance that one of them is flawed, which allows a star system to go rogue and destroy its lightcone. So there is some inherent tradeoff between diversity and security here.

Plausibly it inherently reduces the value of the future for some people (e.g. it is incompatible with some conceptions of freedom, self-determination, or privacy). Surveillance would probably have to be fairly extensive—the nightwatchman needs to detect attempts to create superweapons, to send spacecraft to other star systems, and to remove the nightwatchman’s ability to monitor and enforce its policies in the future. It’s plausible that it would need to monitor every large industrial facility in each star system. The details will be determined by what is possible at technological maturity, such as the minimum requirements to build and launch a near-light-speed probe.

But enforcement should and could be narrowly targeted to preventing these very serious, galactic-scale harms. Otherwise, the nightwatchman should allow the inhabitants of the star system to govern themselves in whatever way they wish. The universal code could also specify different levels of verification and enforcement: if surveillance is lax enough that a few people avoid paying their galactic taxes to contribute to moral public goods, that is not the end of the universe. But surveillance to prevent false vacuum decay must be incredibly robust.

The nightwatchmen could also be time-limited. After all reachable star systems are settled, and ideally once galaxies have drifted apart such that they are no longer causally connected, the nightwatchmen could switch themselves off. A massive governance failure in one galaxy (e.g. vacuum decay) then could not spread to others, containing the damage. On plausible cosmological theories, the vast majority of lives might be lived in this ‘twilight’ of many causally disconnected pockets. So even if extensive surveillance ends up having large costs, only a very small fraction of lives need be lived under such conditions.¹³

This would not prevent suffering risks, though.

Currently, we shouldn’t trust any government with this power, as surveillance tech designed to stop terrorists is dual-use and could also be used to e.g. detect political dissidents. But it will likely eventually be possible to build provably secure AI systems with precisely specified remits, in this case to enforce the universal code (see proposals on ‘structured transparency’ for a sketch of what this could look like).

This is a crux for us—if it’s not possible to get these strong guarantees that the nightwatchman would only enforce the narrow universal code, then we would probably oppose it.

Many questions of optimal space governance—such as what combination of democracy, markets, and wise ASI oracles should determine how space resources are used¹⁴—remain very open. But we think the question of whether colonists should be allowed to set off into the unknown and entirely self-govern can be answered in the negative. Encouragingly, we expect people will come to realise this before it is too late and the von Neumann probes have left the solar system. But this isn’t guaranteed! This post is our small attempt to make galactic anarchy less likely.

This article was created by Forethought. See the original on our website.

1. E.g. from Thiel’s ‘The Education of a Libertarian’: “Because the vast reaches of outer space represent a limitless frontier, they also represent a limitless possibility for escape from world politics.”

2. Kevin Kohler points out that historically, empires have been able to cohere with up to about 3 months of communication latency (e.g. the transit time of a ship sailing from England to Australia in the 19th century). It is unclear how quickly governance cohesion deteriorates as latency increases.

3. Damon Binder argues that it is ~25% likely to be possible to deliberately trigger vacuum decay, even at technological maturity.

4. Except for galactic X-risks, since e.g. vacuum decay may be physically impossible to trigger. But S-risks and unconstrained expansion are fairly independent for each galaxy or star system.

5. The idea is not original to the supplement authors. E.g. Avi Parrack in a brilliant, rambling, 2,000-word EA Forum comment notes that:

Provided we develop a strong science of intelligence it seems possible to create a very robust and capable yet verifiably restrained system. It could be comprised of highly intelligent but transparently/provably safe entities, narrow/programmatic ensembles of enforcement mechanisms/protocols or a mixture of the two.

And various FHI (RIP) and broader EA people have been thinking along these lines for a while.

6. The AI 2040 supplement discusses various market and democratic mechanisms to decide which rogue actions are sufficiently harmful that actors should be prevented from taking them. But none of these systems are possible to implement without the ability to enforce some decisions across the universe.

7. Or rather, with sufficient reliability that the expected number of failures across all civilisations and the whole future is hopefully <<1.

8. This is slightly more restrictive than needed, as probes leaving that are travelling very slowly, or that do not have self-replicating tech on board, pose no threats. The nightwatchman must be in place by the time the first probe leaves that could win a space race if it chose to initiate one, or if it is technologically advanced enough that it could trigger a galactic X-risk. We do not currently know where that minimum capabilities of concern threshold lies. It would be imprudent to send out probes first and nightwatchmen later, as any rogue star systems may be able to outrun or overpower a nightwatchman sent too late.

9. This is reminiscent of Bostrom’s ‘freedom tags’ to protect a vulnerable world, although it is unclear whether every individual will need to be personally surveilled: hopefully only large-scale industrial technology will need to be.

10. However, once Earth-originating intelligence has spread far enough and as the universe expands, distant settlements will no longer be able to receive messages from the centre.

11. Who should be included in these votes, and how these votes should be carried out, is another important question. This is discussed in some more detail in the AI Futures Project's space governance agenda, under the heading “Mitigating downside risks.”

12. It seems likely that we will have made lots of scientific progress within the solar system before expanding through the universe, such that we will have a very good idea of what galactic X-risks are physically realisable. So we don’t expect such updates to be needed.

13. And if civilisation aestivates for billions of years, fewer people again need to live under surveillance.

14. A worthy topic for a future post, perhaps!