Policy ideas to ensure responsible government deployment of AI
This article was created by Forethought. See all our research on our website.
Summary
Governments are going to deploy frontier AI in the domains where they have exclusive powers: military, intelligence, surveillance, and punitive law enforcement.
The existing checks on those powers are weak. AI deployment could weaken them further.
Companies aren’t well-placed to police government deployments (e.g., contracts often require zero data retention), legislatures often lack access and capacity, and, as AI replaces humans in government, nobody will be left to refuse or slow-roll unlawful or anti-democratic orders.
In the limit, this can enable full-blown coups; in the short term, it undermines checks & balances.
This post lays out the work I’m most excited about:
Governance of AI applications in government: policies on how AI should be procured and used in government, research on how to govern AI agents in particular, and field-building beyond the EA/safety crowd.
Interbranch oversight: transparency and oversight provisions, audit tech for an automated government, waking up lawmakers, and increasing AI uptake in the legislative and judicial branches.
Developing AI for government use: model specs/constitutions that say something meaningful about behavior in government contexts, and evals for the qualities we’d want such systems to have (such as epistemic integrity, non-sycophancy, lawfulness).
Strengthening civil society: monitoring and freedom of information infrastructure for government AI use, a coalition of ML researchers at frontier labs, and tools that preserve citizens’ ability to coordinate (AI-for-epistemics + privacy tech).
Many ideas are more tractable than they might seem: specs are being written now, must-pass bills come around every year, and a lot of the civil-society work just requires someone to start doing it.
If you’re interested in working on any of it, apply to EIP’s recent RFP. You could also fill out this expression of interest form.
Why care about responsible government deployment of AI
How could government deployment go wrong?
Governments will increasingly use the most powerful AI systems in domains where they have exclusive powers; particularly military, law enforcement, intelligence, and surveillance. This is most salient for the US government: Claude was reportedly already involved in the Maduro raid in Venezuela and assisted with targeting during the military engagement with Iran. With the recent Executive Order, at least some parts of the US government will have privileged access to frontier AI models for up to 30 days before release to other trusted partners.
Without appropriate oversight or constraints, this could ultimately lead to a literal AI-enabled coup where a commander uses military AI systems to seize control of a country. Less extreme scenarios still undermine important checks and balances.
This risk strikes me as similarly important as the risk from misalignment and there are way fewer people working on it. And importantly, alignment is not sufficient: even a perfectly aligned AI loyally serving a coup-staging principal is catastrophic.
Why is this not going to go well by default?
Oversight into these government domains is already very limited. Companies aren’t well-placed to police government deployments (e.g., contracts often require zero data retention) and legislatures often lack access and capacity. Activities in these domains are often classified/secret.
As AI systems replace human bureaucrats, military officers, and soldiers, this will further empower the government and remove some of the remaining checks:
Governmental accountability partially rests on citizen bureaucrats and soldiers who refuse or slow-roll orders that are clearly unlawful or undemocratic. In 2024, for example, various commanders in the South Korean military refused orders during a constitutional crisis, which probably prevented a greater catastrophe. Unconstrained or personally loyal AI systems could change that.
The power and reach of government have historically been constrained by the friction inherent in a large bureaucracy. AI systems could remove that friction and make it much easier for leaders to impose their will on a country, which could make it much easier to abuse their position.1
AI could enable pervasive surveillance & cheap data processing, which could allow prosecution of political enemies on an unprecedented scale.
What to do about it?
In the spirit of inspiring more action, I’ve compiled the efforts I’m most excited about below.
If you’re interested in working on this topic, I’d encourage you to apply to EIP’s recent RFP or to fill out this expression of interest form. You should probably also take a look at this research agenda.
Rules for AI in government
Legislatures should set rules for the procurement and deployment of AI in government, especially for the most sensitive applications of government power (military, intelligence/surveillance, and punitive law enforcement).2
I think this is actually tractable right now, at least in the US:
The recent Anthropic/DOW conflict has put this on the Congressional map for autonomy in weapon systems & surveillance.
While Congress is passing fewer and fewer bills each year, there are annual bills for the intelligence community and military that have to be passed (IAA, NDAA), and there’s a lower bar for including stuff in them.
In the future, crises or scandals will create legislative openings, and we should have proposals ready to go.
Work I am most excited about
Policy & advocacy: Develop and push for rules for AI systems in critical domains (or adjust existing ones to account for the use of AI systems). The most salient ones in my mind are:
Autonomous weapon systems: Currently, they are only governed by a DOD Directive, which could be rescinded at any point. I’d love for more people to think through rules that would make it harder to use these systems against domestic opposition (e.g., technical guardrails against abuse, limiting the amount of autonomous force any one human can command, multi-party authorization schemes, geofencing).
General-purpose systems in the military: They are currently not specifically regulated at all, but ultimately, they could orchestrate large-scale cyber attacks or command entire battalions. At that point, it will matter a ton whether they have any constraints at all.
Surveillance: AI will enable data processing on an unprecedented scale. It’s worth thinking through how to adjust current rules in light of that (e.g., closing the data broker loophole).
Policy & advocacy: Develop and push for cross-cutting requirements for AI systems in government (general-purpose ones in particular).
There is already an Executive Order requiring all AI systems in government to be truth-seeking and ideologically neutral. Those seem like great requirements to me that could be refined/improved and put into statute.
Another cross-cutting proposal is law-following AI, which roughly says that AI systems in government should be trained to follow the law rather than just instructions. I’m excited for more people working out the open questions of this agenda, e.g., which laws should they be required to follow, how should they decide whether a contemplated action is likely to violate the law, in what contexts should the law require that AI agents be law-following?
More minimally, AI systems could be required not to assist in undermining the constitutional order (though how to assess this is challenging in its own right).
Field-building: Build a broad-tent coalition for responsible deployment of AI in government.
Responsible deployment and oversight is something that lots of people and political parties should be in favor of. It could be good to build a coalition that is nonpartisan and includes AI safety advocates, constitutional lawyers, civil liberties orgs, defense intellectuals, and national security think tanks.
Oversight of AI in government
Rules need to be enforced and updated, even as the technology changes. That will require the legislative and judicial branches to be institutionally and technologically empowered.
Again, there are some reasons for optimism in the US:
Congress may soon be controlled by the opposite party from the Presidency, which creates incentives for increased oversight.
Crises or scandals can open up opportunities for legislation (e.g., FISA following the Church Committee).
Some non-legislative projects could still create significant value.
Work I’m most excited about
Policy & advocacy: Develop and push for the transparency & oversight provisions.
We will need democratic oversight of AI systems, especially around surveillance and the application of force (in military and law enforcement). This is currently not in place and not on track to happen.
Here are proposals I’m currently most excited about (in the US):
Requiring comprehensive logging of government use of AI (and designating such logs as federal records)
Automated flagging of suspicious AI behavior to oversight bodies (ideally, something like FISA courts or Congressional committees)
Whistleblower reform for national security domains (especially allowing whistleblowers to talk to Congress)
Increasing GAO’s capacity to audit deployment of AI in national security domains
Auditing tech: Pilot and mature the infrastructure required for overseeing widespread AI use.
As AI is increasingly used in government, it will be both a challenge and an opportunity for oversight. The speed of AI will make it harder for regular human oversight to keep up, but AIs are in principle much more auditable than humans because they think out loud and automatically create records that can be scrutinized by other AI systems that flag suspicious behavior.
There are already attempts to audit companies deploying large amounts of AI labor. I would love to see these piloted, matured, and adapted for classified government contexts.
Advocacy: Wake up lawmakers to the powers and challenges of AI.
Some of the proposals in this doc require ambitious changes to the way government is run. They are more likely to happen if lawmakers are aware of the stakes involved.
This could involve building trusted relationships, showcasing AI capabilities through easy-to-understand demos, and making insights from the AI safety community intelligible to people with much less context.
Products/programs: Increase AI uptake & unblock automation in legislative and judicial branches.
To provide meaningful oversight, it will become increasingly important to understand and utilize AI systems. I’d love for lawmakers and judges to be well-versed in these systems.
Examples of the things that I have in mind here include: providing training, building custom tools, bringing in technical experts through fellowships (e.g., TechCongress), and reconstituting the Office of Technology Assessment in Congress.
Policy: Coup-proof government-led AI projects.
If there’s ever a government-controlled AI project (e.g., public-private partnership), its shape will matter a lot. It will probably be designed in response to a crisis, which favors existing templates & plans. I’d like for somebody to create one that carefully distributes power (e.g., multi-stakeholder governance boards, oversight mechanisms that work under classification constraints, sunset clauses, mandatory external audits, access guarantees so one company doesn’t end up controlling the stack).
Research (speculative): Write new constitutions.
There is a chance the disruption caused by AI will create “constitutional moments”. At that point, many government functions may need to run at machine speeds: voter input, legislative deliberation, judicial review, law enforcement, or application of lethal force. How do we do that in a way that still preserves important democratic values, if that’s desirable? I don’t really have great answers to these questions, and would love for people with the right macrostrategy skill set to think about them.
(Of course, the more of these problems we can address without actually requiring constitutional changes, the better. Constitutional changes are hard!)
Developing AI for government
There are many free parameters when it comes to the question of how to build AI systems in government and what constitutes safe and responsible AI systems in critical domains (setting aside whether companies can constrain government use through contracts).
Again, I think progress on this front is pretty tractable:
Specs/constitutions for frontier models are being developed right now.
Systems are already being procured by governments across departments, and that will probably only increase.
Work I’m most excited about
Research: Design and stress-test model specs/constitutions for government use.
General-purpose models deployed in government should have some guardrails, e.g., they should not assist in staging coups to overthrow the civilian government. This is a tough line-drawing exercise where you don’t want models to overrefuse, but you also want some meaningful constraints. That makes me keen for people to work out desired behavior for lots of edge cases and create broad public buy-in for a set of minimal constraints.
Research: Build evals & datasets for desirable qualities in government-deployed AI systems.
The US government has already said it wants AI systems to be truth-seeking and ideologically neutral. That seems great to me. Other important qualities include: epistemic virtue / integrity, non-sycophancy, robustness to manipulation / adversarial pressure, and lawfulness / constitutional fidelity.
As far as I can tell, current methods of measuring this are insufficient. So I’d be excited for different projects to develop evals that track and incentivize qualities like this.
Strengthening civil society
Civil society is an independent check on government activities (e.g., transparency & monitoring efforts, pressure & advocacy campaigns, building valuable tools for empowering the citizenry). I expect that the same mechanisms will also provide some accountability in the case of AI (though it might be particularly tough in classified domains, which may sadly be the most relevant).
Luckily, this falls into the category of “you can just do stuff”, so I think it’s very tractable. My main worry is more about how much of a difference it will ultimately make.
Work I’m most excited about
Advocacy: Build an OSINT program or organization that informs about AI use in the government.
It would be good to have more transparency into government deployment of AI, so that civil society can provide a counterweight to government power. I imagine some watchdog or civil liberties orgs are already doing versions of this, but there may well be important bits that are missing.
Some examples of the kinds of things I have in mind:
Figuring out what to monitor, especially from the perspective of preventing concentration of power.
Building an automated tracker for relevant executive orders, agency directives, memos, emergency declarations, relevant procurement decisions, and reclassification decisions.
Submitting systematic FOIA requests targeting government AI procurement records, deployment & use decisions, usage logs, and other relevant documents.
Publicly reporting on the most important developments.
Labor-organizing: Organize ML researchers into a coalition around responsible use of AI.
You could build a coalition of employees at AI companies who are concerned about the use of the technology they’re building. They could advocate for policies and draw red lines around certain use cases, enforced by boycotts. Structuring it as individual pledges avoids antitrust issues.
These employees currently have a lot of bargaining power with regard to their companies (cf. their salaries). So they’d have to be taken seriously by their employers, and by extension the government.
This is inspired by the Federation of American Scientists, which was founded in 1945 by various contributors to the Manhattan Project. As I understand it, they supported the McMahon Act of 1946, which established civilian control over atomic energy (instead of military control), and their Nuclear Information Project became the gold-standard open-source tracker of global nuclear arsenals, published annually in the Bulletin of the Atomic Scientists.
Tech development: Build tools that preserve citizens’ capacity to coordinate against gradual concentration of power3
Here are the two categories I have in mind:
Shared sense-making (”AI for epistemics“). Authentication of authorship, provenance, and content, and tools that push toward a high-honesty equilibrium. Takeovers typically require secrecy because they violate the preferences of too many people to survive transparency, so a better-informed society is structurally harder to take over.
Privacy from state surveillance. It’s hard to target and influence what you can’t see. Secure communication and privacy tech raise the cost of preemptive coercion against organizers (with the caveat that the same tools can also shield collusion).
There’s already an active ecosystem of funders and builders here (e.g., ex/ante, Future of Life Foundation). I’d encourage people to plug in rather than starting from scratch.
This article was created by Forethought. See all our research on our website.
For what it’s worth, I think this same dynamic applies to other institutions like companies. They’ve similarly been constrained by requiring a large bureaucracy of humans, which creates various inefficiencies and checks, limiting the influence of any one person at the top. However, AI could change that, and we need a response to this. On the whole though, I do think that this presents a unique problem in government because of its sheer size and monopoly on the legitimate use of violence.
I'm aware that some of these rules may slow down adoption, and I think that’s a downside worth taking seriously. Protecting democracy only matters if the democracy survives foreign competition. So I’m most excited about proposals that add meaningful safeguards without significantly eroding capability/adoption.
This won't help against sudden takeovers involving advanced military technology, but in slower erosion-of-checks scenarios it could matter a lot.
 | A guest post by
|