This article was published by Forethought. Read the original on our website.
If we’re thinking about automating alignment, it may help to distinguish different types of work we may wish to automate:
Empirical alignment research: design and execution of alignment-relevant experiments (for a given notion of what we’re aiming for)
Conceptual alignment research: theoretical or judgement-driven work to improve our understanding of what we’re aiming for and how to achieve it (including how to measure it)
Direct alignment work: making particular AI systems more aligned (including via inputs such as training data)
These buckets are actually pretty different, in terms of what’s needed for automation, what the impacts of automating are, how much damage subtle misalignment within the automated systems could do, etc. So I’ve come to think that we should be tracking them separately.
Automating empirical alignment research
This is the most obvious fit for the AI capabilities we’re getting at the moment. You want to know more about the relationship between some alignment-relevant variables? Great, an AI agent can design and run that experiment for you. You want to know what training methods help your alignment benchmark go up? In the not-too-distant future, AI agents will probably be able to do this too. When we have the automated capabilities researchers we seem on track for, this is exactly the kind of thing they’ll likely be able to do a lot of.
Note that I’m not counting the design of benchmarks/metrics/evals as part of this bucket. In current practice that’s tightly intertwined with empirical alignment research, and often the hardest part of it. But for automation purposes it clusters much more with conceptual work; it’s easy to imagine automating the rest of empirical research without it (as AI capabilities generally get automated); and I think its automation has different strategic upshots.
The capabilities that are needed for automating empirical alignment research very heavily overlap with the capabilities needed for automating AI capabilities research. Therefore, it doesn’t seem like a priority to try to accelerate this type of automation.
However, the prospect of getting a lot of automated empirical alignment work before we have good versions of other types of automated alignment work increases the priority of work which complements it. Significantly, that may mean better specification of the kind of numbers we would like measured/targeted.
Automating conceptual alignment research
What about automating the type of metric choice/construction that we’ve just said would be a good complement to automating empirical alignment work? Or more generally improving our understanding of how to think about alignment, what we’re even aiming for, etc.?
This is essentially theoretical research. Automating it is trickier than automating empirical work because of the lack of verifiable signal about what constitutes progress.
Accelerating the automation of conceptual research is somewhat fraught (feeding into some dangerous technologies), but seems helpful for a number of high-priority areas, including macrostrategy as well as conceptual alignment work. Overall it seems likely good to do, if we can.
Methods for helping to automate conceptual research broadly might include:
Curating training data for finetuning
Building scaffolds to try to elicit more research progress from current or future models
Building methods to extract more signal from existing conceptual research data
Building benchmarks for conceptual research capabilities, that can be targeted by AI researchers (or potentially by automated capabilities researchers)
Or just identifying good ideas for benchmarks, even if they aren’t fully implemented
Automating direct alignment work
As well as learning more about what it takes to make systems more aligned, sometimes we want to just make our actual systems as aligned as possible.
This overlaps with empirical alignment work, but the emphasis is different:
Empirical alignment work is science
Direct alignment work is engineering
Automating direct alignment work could include a lot of activities which leverage scalable cognition for alignment-relevant purposes. This could include careful auditing of particular RL environments or training data, or direct monitoring/supervision of systems during training. RLAIF for alignment properties is a form of automated direct alignment work; Claude 3 Opus’s apparent self-alignment was another (albeit accidental in that case; but if we knew how to reproduce or generalize the behaviour perhaps it could happen deliberately).
I think that the limit of automating direct alignment work would be a form of system 2 learning, where updates are not made blindly towards a training objective, but reflectively by an AI system that was already sufficiently aligned (not necessarily having robustly good values already — just enough genuine care about being aligned, and honesty with itself, that its reflective updates tend to move it further in that direction), and wished to become more so — whatever the deep true version of aligned should mean1. Embarking on such a path could be fraught (because of difficulty in being confident that the starting conditions were in the right basin of attraction), but compared to other alignment approaches, it has the benefit that it might leverage the full strength of the AI’s cognition to help, so that its “self-improvement” in an alignment direction is a closer analogue to meditation than to self-performed neurosurgery.
So, accelerating the automation of direct alignment work could include:
Designing (and analysing) setups that can deploy AI labour to help with direct alignment work
Designing paradigms (that could then be tested empirically) that allow AI to be a more active participant in its own alignment
Producing data (or scaffolds) to help improve AI taste at the alignment-relevant tasks necessary for this
Interactions
These categories blur and interact in a number of different ways:
Automated direct alignment techniques may be good subjects for empirical experiments
This is not completely reliant on having good metrics for alignment; developing a science of what’s going on with different attempts at automated direct alignment could be useful even if we don’t yet know what “good” looks like
Automated hill-climbing might get you to very strong direct alignment approaches — but this does rely on having good enough metrics
i.e. we could plausibly be in a regime where we’re better at reasoning about and implementing training setups that in fact lead to aligned systems than we are at accurately measuring their alignment. In that case, optimising for our best alignment metrics might produce worse alignment outcomes than following our best-guess direct alignment process.
If you can do a good job of automating conceptual alignment research, you should be approximately done (“winning” on the alignment challenge)
Since that should then be able to do the relevant parts of prep for automating the other types of alignment work, and being appropriately confident in your techniques for them
You could also in principle do a good enough job at direct alignment that the systems you’re building are entirely safe
But I think it will be hard to have very strong confidence in that before improving our automation of the conceptual side; and so each new generation of AI might pose a challenge — will your alignment techniques transfer, or will your luck run out?
So: the main goal (from an existential risk perspective) for automating empirical research and direct alignment work is to get our systems safe enough to robustly avoid catastrophe (“not losing”) for long enough that we can automate conceptual alignment research to a very high quality level
There will likely be approaches that allow you to leverage automation of empirical research to get better automation of conceptual work
e.g. Taking Talkie or another LLM with only vintage training data, and working out how to get it to perform well at rederiving more modern theoretical work
Priorities
Two major uncertainties that affect priorities:
How much loss-of-control risk is early?
If there’s a bunch, then putting more energy into the “not losing” part of things makes sense
If the main misalignment risk comes when we’re getting systems that can perform well even on tasks without verifiable rewards (like long-term strategy to allow them to plot sophisticated power-seeking campaigns), then it makes sense to put more energy into accelerating the “winning” part, and so perhaps move it earlier in time so that it comes before some crucial challenge that we otherwise wouldn’t be able to face
Note that in this world, the win state and the biggest threats rely on capabilities that from our vantage point look quite similar — in that they require high-quality thinking about novel situations without verifiable rewards — so it’s quite plausible that even a modest acceleration could change the sequencing
How close might we be to decent automated conceptual alignment work, if we approach it relatively directly?
If it’s not hideously out of reach, then it looks more like the kind of thing where human intervention to speed it up (by building benchmarks and scaffolds, and curating data) could make the difference
If so, the full suite of activities mentioned above as precursors for conceptual alignment work looks helpful
If automated conceptual research is too hard, maybe the direct interventions don’t actually do much — in this world, automated conceptual research falls out of stronger AI driving automated (empirical) research programs to work out how to automate conceptual research
So trying to target conceptual alignment work directly might be obsoleted as a way to accelerate the “winning” moment, and it’s a higher priority to get things safe enough that we robustly don’t lose through the period where the automated capabilities research is speeding up
I’d note that building metrics/benchmarks for conceptual research, that can be targeted by later automated empirical work, looks more robust to this concern than other forms of targeting automated conceptual research
My personal guesses on these uncertainties are that most of the loss-of-control risk comes later (after AI is superhuman even at tasks that don’t have verifiable rewards), and that it is possible to directly approach decent automated conceptual research. However, mostly I’m pretty uncertain — which argues in favour of hedging by investing some in all of the paths.
Acknowledgements: this article was inspired by a conversation with Collin Burns. Thanks to Max Dalton and Alex Mallen for comments on an earlier draft.
This article was published by Forethought. Read the original on our website.
So the system would need something like the type of understanding that conceptual alignment research is aiming at. But conceptual research aims to understand things abstractly; the understanding which such a system had might be more akin to a lived experience and understanding. I say this without taking any stance on whether such AI systems would have conscious experience; it does rely on something like introspective ability, but that’s something we’ve seen (to some extent) in LLMs today.
