Data bottlenecks won’t prevent an intelligence explosion

But they will slow it down

This article was created by Forethought. See the original on our website.

Summary

One of the most important questions facing the world is whether AI will undergo a fast intelligence explosion and quickly automate most real-world work.

A key objection people raise is data bottlenecks. Data is a crucial input to AI training — how could it increase fast enough for such explosive progress?

I tried hard to find strong bottlenecks, but ended up sceptical that they would stop an intelligence explosion from accelerating over time.

Here are the most compelling bottlenecks I investigated and why I was ultimately unconvinced.

Bottleneck 1: you’ll need millions of trajectories of every job, which will take years or decades to collect (more)

Today’s AI algorithms are very data hungry. You would need a huge amount of data to automate the full economy.

But the plan isn’t to use today’s AI algorithms. The plan is to do a software intelligence explosion inside a data centre, where AI becomes superintelligent at AI research — and have that AI produce a highly sample-efficient learning algorithm.

Then you don’t need millions of trajectories per task. You need as many as a human learns from, or fewer. And the data can be messy, just like the data humans learn from. That data could be gathered in a few months.

Sceptic: “But what about sim to real transfer? Would a learning algorithm discovered using virtual tasks really translate to the real world?”

People get confused here. Trained neural nets generalise badly. A neural net trained on chess generalises badly to Go. But learning algorithms generalise very well. The same learning algorithm that masters chess can also master Go (AlphaZero). The Transformer architecture was developed for processing text, but it also works for sounds, images and controlling agents. The human learning algorithm was “designed” for hunting on the Savanna, but it also works for string theory.

So if AI designs a sample-efficient learning algorithm from inside a data centre, it will very likely translate to real-world tasks. (And AI will have access to some real-world tasks so can check and iterate!)

At this point the sceptic might retreat: “Fine, data won’t be a bottleneck if there’s a software intelligence explosion in a data centre. But data bottlenecks will prevent that from happening in the first place!”

So let’s focus on bottlenecks to a software intelligence explosion.

Bottleneck 2: AI progress has ridden exponential growth in training data. That can’t continue, let alone accelerate (more)

But forecasts of a software intelligence explosion extrapolate software progress. And software progress just is: getting the same capabilities from less compute and less data. So the core engine of the explosion doesn’t rely on increasing the amount of data at all.

Bottleneck 3: improving data quality relies on human experts, and that won’t be possible once AI is smarter than humans (more)

The core engine of the software intelligence explosion does rely on improving data quality. Today we augment high-quality internet data, pay humans for expert trajectories, and build RL environments where AI has to replicate human-built software.

But all these methods extract quality out of an existing human reservoir. Above human level, the reservoir is empty. AI will have to manufacture higher-quality data than any that exists, and do so from scratch.

This will slow the intelligence explosion.

But it won’t stop it in its tracks. We can already produce superhuman data quality via RL. And AI could produce higher-quality trajectories by thinking for longer.

Crucially, this won’t stop the intelligence explosion from accelerating. Losing the human reservoir makes it harder to get from AGI to AGI+ — but it makes it harder to get from AGI+ to AGI++ by roughly the same amount. The handicap is the same at every stage. So if you previously expected each step to take less time than the one before, you should still expect that. Every step might take 50% longer, but progress still accelerates.¹

Bottleneck 4: paradigm tax (more)

AI capabilities are spiky. When AI first matches humans at AI research, it will be much stronger than humans in some ways and much weaker in others. More parallel copies, faster thinking — but weaker generalisation and weaker sample efficiency.

Which means AI’s human-level performance will initially lean heavily on a mountain of human writing about Transformers.

When AI then discovers new techniques — no such mountain will exist. No textbooks, no blog posts, no code. Deeply understanding a technique is harder than stumbling upon it — we understand general relativity far better now than Einstein did in 1917. So AI may match humans on the techniques it inherited, but fall below humans on the ones it creates.

This paradigm tax can be paid. E.g., AI can generate millions of trajectories to illustrate new techniques. But paying the tax is a real cost that slows down AI progress.

Again though, this won’t stop the intelligence explosion from accelerating. It makes it harder for AI to master whatever replaces the Transformer — but it makes it harder to master whatever replaces that by roughly the same amount. Every new paradigm arrives without a corpus. This makes each step take longer than you previously thought, but it doesn’t change whether each step is faster than the one before.

My overall bottom line: Data bottlenecks will slow the early stages of a software intelligence explosion, but won’t stop it from accelerating over time. And then they won’t stop AI from quickly automating most economic work thereafter.

The rest of the post defends this conclusion in greater depth.

Three types of data bottleneck

Before discussing the specific data bottlenecks I find most plausible, I’ll give a quick taxonomy for three types of bottleneck and another taxonomy for three times at which a bottleneck could occur.

I like the following breakdown of data bottlenecks (h/t Herbie Bradley):

1. Data quantity. You need more data that’s in the same distribution of data that you already have some samples from. E.g. more Wikipedia data, or more RL coding environments at the same level of difficulty.

2. Data quality. You need data that teaches the same skills as existing data, but demonstrates those skills to a higher average quality level. E.g. expert-curated step-by-step solutions to difficult math problems, more difficult RL coding environments, or simply filtering to remove low-quality data.

3. Data coverage. Data showing knowledge and skills not already present in the dataset. E.g. data about how to operate machines in a factory, or expert trajectories showing step-by-step how to build a financial model in Excel (where your previous data only showed finished models, not the process for building them).

Three types of data bottleneck. The AI company has the black data points. The orange data points show absent data that could bottleneck progress.

We’ll make use of this breakdown in what follows.

Three times when a bottleneck could occur

Very roughly, I expect AI progress from today to go in three broad phases:²

1. Scaling. A continuation of recent scaling-driven progress until AIs match humans at AI R&D.

2. Software intelligence explosion. An SIE in a data centre during which AI capabilities at AI R&D increase very rapidly. (Capabilities at other far-flung economic tasks may increase much more slowly.)

3. Broad deployment. AI becomes expert in thousands of specific domains across the economy by learning from domain-specific data.

Three phases when data bottlenecks could delay AI progress.

Data bottlenecks could in principle delay any of these phases. I’ll discuss each phase in turn! Spoiler: ultimately we’ll be populating a table with this format:

Data bottlenecks to automating AI R&D

This first phase hasn’t been my focus, but I’ll briefly state that I think data bottlenecks are plausible.

Limited data quantity and quality are already somewhat reducing the gains to scaling pre-training. And automating AI R&D may require better data coverage — there’s abundant coding data but much less data demonstrating (e.g.) good research taste. Indeed, generalisation from RL still seems to be fairly limited — if this continues, automating AI R&D may require constructing data for most parts of the (hugely complex and varied!) AI R&D workflow. That could involve recording humans’ computer screens, or building lots of RL environments that tile the space of AI R&D tasks.

Ok, let’s discuss bottlenecks to the second phase: a software intelligence explosion.

Data bottlenecks to a software intelligence explosion

I’ll discuss two specific bottlenecks that will slow down the SIE.

But first, I’ll briefly explain why I’m generally not expecting big data bottlenecks here.

Let’s say AI is weak at some particular AI R&D task. AI companies will have unfettered access to the real-world deployment setting (AI R&D itself!) so can closely study what is going wrong and craft solutions accordingly. AIs could think for a long time to craft high-quality demonstrations for supervised fine-tuning, write sophisticated tests and rubrics to evaluate AI performance, and design RL environments to elicit the desired capabilities. Before humans are obsolete (which happens fairly deep into the SIE), human experts can input to all these approaches.

Ok, let’s turn to the first specific data bottleneck to the SIE.

Paradigm tax: Early in the SIE, conceptual progress in ML will make AI less capable at AI R&D

My argument here depends on two core claims. I’ll argue for each in turn.

Claim 1: Early in the SIE, AI will have weak sample efficiency and generalisation compared to humans.

My preferred milestone for the “start” of an SIE is AI-human parity. This is when an AI company would make roughly as much research progress using only its AIs (no human researchers) as it would using only its human researchers (no AIs developed after 2020).³⁴

Why is this a good milestone? As well as being fairly concrete, it is also roughly when AI is beginning to significantly accelerate AI software progress. If AI-alone would contribute as much as humans-alone, then combined they likely contribute much more than humans-alone, maybe 10x more. This is due to strong complementarities between AI and humans.⁵ So the AI-human parity milestone is also roughly when software progress starts to significantly speed up (see footnote for a BOTEC⁶).

AI capabilities are spiky. At AI-human parity, AI will be much stronger than humans on some dimensions, and much weaker on other dimensions. AI will be much more numerous, fast, knowledgeable and experienced. But it will have weaker sample efficiency (i.e. need much more data than a human to learn a new skill) and weaker generalisation (i.e. what it learns transfers less well to new situations) — these are the areas where it’s weaker today⁷ and this weakness is likely to persist.

(I think AI’s weak sample efficiency and generalisation are likely intertwined. Both imply that AI capabilities are weak in domains where there is little data - h/t Tom Cunningham. I will refer to both simply as “sample efficiency”.)

Once you accept spikiness, it’s very hard to avoid the conclusion that AI will be much weaker than humans in some dimensions when we reach AI-human parity. Sample efficiency seems very likely to be such an area. (See footnote for an objection.⁸)

Of course, if AI-human parity doesn’t happen for a decade and there’s a paradigm shift first, it’s much harder to predict the specific areas of weakness.

I think there are likely many interesting implications of this early-SIE spikiness (e.g. for when long-run schemers might emerge), but here I’ll just focus on the implications for data bottlenecks.

Claim 2: AI’s weak sample efficiency bites much harder once AI significantly pushes the frontier of AI R&D. New ML concepts won’t be present in the human-derived training data and so AI will understand them less deeply.

AI will reach parity with humans at AI R&D by training on orders of magnitude more relevant data: millions of human-written posts that teach it today’s ML paradigm, billions of lines of human-written code that teach it today’s programming languages, and a massive stock of RL environments — built up over years from real-world software — that teach it today’s best practices for software engineering and AI R&D.

In other words, AI will reach parity with humans by making significant use of human-derived data sources that give it capabilities in today’s AI R&D techniques.

But if AI significantly pushes the frontier of AI R&D, these human-derived data sources will be outdated:

• AI will have invented techniques as important as Mixture of Experts and Sparse Attention (two significant architectural improvements to the Transformer), and probably techniques as big as the Transformer and RLVR (the reinforcement learning technique behind today’s reasoning models). But the human-written pre-training data will be much less helpful for understanding these new techniques!

• AI will invent improved coding languages. And the pre-training data will be much less helpful for mastering these!

• It will invent new ways of structuring AI R&D workflows, rendering human-constructed RL environments less helpful.

• AI may invent an entire new paradigm completely absent from pre-training data.

By the time AI has done these things, the SIE will be in trouble if AI capabilities are still reliant on human-derived training data.

Another way to think about this: AI may reach human parity at today’s ML techniques and significantly accelerate AI progress for a bit. But once it has invented new ML techniques, it may fall back below human parity because there’s much less human-derived data for it to learn from.

You might object: “When AI invents a new technique, can’t it just write it up and throw the write-up into the next generation’s training data?”

Yes — but this helps much less than you’d think, because of AI’s weak sample efficiency. (Once AI training is roughly as sample efficient as human learning this objection goes through — but that won’t happen until late in the SIE.⁹) Today’s AI masters concepts like the Transformer not from the original paper, but from millions of blog posts, tutorials, Stack Overflow answers and codebases that use the concept in varied contexts. A handful of AI-written papers won’t replicate that. To teach a new concept properly, AI would have to generate a similarly broad and varied corpus showing the concept in use — possible, but a significant extra cost that today’s AI R&D doesn’t pay. And further, AI-generated synthetic data is often less effective for teaching AI than human-generated data. Moreover, with current methods, purely synthetic training data often underperforms mixtures that retain at least some human-generated data.

So this is a real problem. But AI companies can do work to address it. They can improve: the sample efficiency of training algorithms, techniques for AI-produced synthetic training data, or the flexibility and horizon-length of in-context learning. There are lots of tractable avenues here. I don’t expect this bottleneck to bring the SIE to a halt.

Still, this will slow down AI progress somewhat.¹⁰ The reduced relevance of human-generated data sources will reduce AI R&D capabilities and the pace of AI progress.

As AI pushes forward AI software R&D, human-derived training data becomes less relevant. This slows capabilities progress relative to what we’d have otherwise expected.

Here’s one way to think about it. Early in the SIE, introducing new ML concepts, paradigms, and coding languages comes with a “capabilities tax”: the new techniques aren’t included in the human-derived training data and so AI is less capable at using the new techniques compared to the old techniques. You can pay that tax via improved learning techniques. But paying the tax slows down progress (relative to a counterfactual where you didn’t have to pay it). This should reduce our estimate of the pace of AI progress, relative to our expectations before accounting for this dynamic.

To be clear, AI companies will only introduce a new technique when it’s worth it even after paying the tax. But that’s no comfort: relative to a world without the tax, they’ll either pay it (slowing progress) or steer away from promising new techniques altogether (restricting the search space — which also slows progress).

Now, once the tax has been fully paid, your learning techniques allow AI to master concepts not present in the human-derived data. From that point, this bottleneck no longer applies and wouldn’t stop AI progress from accelerating. So this bottleneck will slow the SIE once, but not prevent it from accelerating from that point onwards.

I want to clarify again: this is a pro tanto reason for AI progress to slow down. It may be outweighed by other factors. By analogy, limited high-quality internet data was a pro tanto reason for AI progress to slow down in 2025, and indeed progress was slower than in a counterfactual world with more internet data, but the overall pace of progress kept up because of new techniques like RL.

In our earlier bottleneck taxonomy, this is a data coverage bottleneck. As AI R&D advances, we lose data coverage of the AI R&D skills that matter.

We now turn to a data quality bottleneck to the software intelligence explosion.

When data quality is below the level of top humans, it’s easy to improve data quality; when data quality is above that level, improving it is harder

Many forecasts of the software intelligence explosion (including my own!) are based on extrapolating trends in LLM “algorithmic progress”. I use scare quotes, because it is quite plausible that a large fraction of the measured efficiency gains (getting the same capabilities for less training compute) actually come from improved data quality.

If data quality improvements will be much harder once the SIE starts than they are today, then our forecasts of an SIE have been too aggressive.

Why might this be the case?

Consider the highest quality data that could be easily derived from human expertise or from human-built artefacts. E.g. a human expert recording a step-by-step demonstration of how best to design an ML experiment, or an RL environment where AI must replicate some complex real-world software that was originally built by human experts.

Call this rough level of data quality the “human-quality ceiling”. Of course, this is a very vague concept that hides a lot of complexity,¹¹ but I think it’s meaningful enough for our purposes. Demonstrations from a skilled undergrad are below the ceiling; demonstrations from future superintelligence are above it.

Historical data quality improvements have only raised the quality of data up towards the human-quality ceiling. E.g. filtering techniques remove data far below this ceiling, and augmentation techniques increase the amount of data close to the ceiling.

As AI has improved, the field has increasingly pivoted towards producing data that’s very close to this human-quality ceiling. For example, OpenAI’s “Project Mercury” has reportedly paid former investment bankers to build financial models as training demonstrations.¹² Data-labelling companies like Scale AI have shifted from crowdsourced data towards credentialed domain experts who create high-quality demonstrations and RL environments.¹³ And startups such as Mechanize are constructing challenging RL environments out of real-world software systems.¹⁴

But, if data quality is to keep improving during an SIE, it must surpass the human-quality ceiling. And past this point, improving data quality may become harder.

Today, data quality efforts often focus on replicating skills and knowledge that already exist in the minds of humans or implicitly in artefacts like complex software. It’s essentially extracting high-quality data that is already latent in the world.

But to go above the human-quality ceiling, you cannot extract latent high-quality data — you have to construct the data yourself. Better filtering techniques for existing data won’t cut it. Rather than simply copying the demonstrations from existing human experts, AI must spend lots of time (and scarce compute!) thinking to construct higher-quality demonstrations than anything it has seen in training (this is the core idea of iterated distillation and amplification).¹⁵ Rather than building RL environments that teach the construction of real-world software artefacts, AI must construct RL environments that teach AI to make better software than anything that exists in the real world.

To be clear, I think this will be possible! It is easier to construct difficult challenges and verify their answers than it is to solve them. Already today, human experts build RL environments that produce data above the human-quality ceiling. Data quality will not hit a wall at the human ceiling.

But it will be harder to improve data quality once we’re above the ceiling. Creating high-quality data from scratch is harder than extracting it from human experts and human-built artefacts.

It’s harder to increase data quality when it is already above the level of top humans.

Concretely, this manifests as a one-time slowdown of the pace of AI progress, at the point at which we pass the ceiling. Once we’re above the ceiling though, this bottleneck doesn’t stop AI progress from accelerating over time (see explanation in footnote¹⁶).

An important caveat: data quality has already been approaching the human-quality ceiling over time in many areas. If AI software progress hasn’t slowed in these areas (which is plausible), that suggests the size of this effect is small.¹⁷

Data quantity bottleneck?

So we’ve discussed bottlenecks to a software intelligence explosion from data coverage and data quality. What about data quantity?

I don’t expect this to be an issue.

Forecasts of an SIE extrapolate historical improvements in the efficiency of training AI systems. That is, ways to achieve the same capabilities from less compute and less data. So the core engine of the SIE doesn’t rely on increasing the data quantity at all.¹⁸

Sceptic: Sure, but making those software improvements has relied on running a exponentially growing number of experiments, which in turn has relied on exponential increases in compute and data.

This just isn’t true. Yes, exponential increases in compute were needed, and I’ve previously discussed this potential compute bottleneck. But running more experiments doesn’t require exponential increases in the quantity of data. You can use the same data in each experiment.

(There is another nearby bottleneck that could stop the SIE from accelerating. Some “scale-dependent” algorithmic advances that work better if models are trained with more compute and more data. Perhaps algorithmic progress would be half as fast if training compute were held constant. If so, that would halve the estimate of the parameter r that determines whether AI progress will accelerate during an intelligence explosion. So this bottleneck could very plausibly block a software intelligence explosion! But I think of this as a compute bottleneck, not a data bottleneck. Its existence is a direct consequence of the fact that (by definition) compute grows slowly during a software intelligence explosion; it arises regardless of the situation with data.)

Ok, that concludes my discussion of bottlenecks to the SIE itself.

Let’s now turn to the final phase — after the SIE, will AI quickly learn the hugely varied jobs in the real-world economy?

Data bottlenecks to automating most real-world work

During the SIE, AI will be trained on data designed to elicit maximal AI R&D capabilities. Partly this will be specialised data for AI R&D tasks; partly data from similar tasks where there is strong transfer to AI R&D, e.g. cyber, maths and software engineering; and partly data to teach more general skills like problem solving, long-horizon coherence, and computer use.

But AI need not be trained on the specifics of most real-world tasks. So there is a potential data coverage bottleneck. The AI companies, after the SIE, will initially lack data for how to perform most real-world tasks.

How much will this delay the automation of real-world work?

We can break this question into two parts:

1. How sample efficient will AI learning techniques be at the end of the SIE? Specifically, how sample efficient on the distribution of real-world tasks.

2. How much real-world data will AI companies be able to access?

How sample efficient will AI learning techniques be at the end of the SIE?

I think AI sample efficiency will likely be as good as or better than humans.

There are four broad routes to this:

• From-scratch training algorithms. An SIE would compress many years of AI progress into one year. Humans are a proof-of-concept that human-level sample efficiency is possible algorithmically, and there are many ways in which AI training could be more sample efficient than humans.¹⁹

• Still, it’s unclear how much sample efficiency of training has increased over time for LLMs, so I’m overall unsure whether from-scratch training will match human-level sample efficiency.

• In-context learning. Already today in-context learning is much more sample efficient and flexible than from-scratch training. Currently it still has some pretty big weaknesses, but it would improve massively during an SIE. I expect this needs neuralese to work, so that AI can build up new and sophisticated internal representations while learning on the job. (Its “context” wouldn’t be a list of words, but a list of “thoughts” that it can flexibly draw on.) This seems likely to work.

• Synthetic data. The relevant metric for sample efficiency is how much real-world data AI needs to learn a job. So if AI can learn effectively from synthetic data — training data an AI produces based on real-world data or (more speculatively) high-fidelity simulations of real-world workflows — that increases sample efficiency. There are promising approaches here, see e.g. dreaming, EfficientZero, and on-policy self-distillation.

• New paradigms. Even if the current broad deep learning paradigm hits a strong sample efficiency wall, during an SIE AI companies could engage in massive parallel search for new approaches.

All these routes look tractable, and during an SIE there will be strong incentives to develop sample-efficient learning techniques. So I expect some combination of the above approaches to work.

In a fast SIE, these learning techniques will all be developed and refined on digital tasks — often simulated or synthetic ones. One might reasonably be sceptical that they will transfer to messy real-world economic tasks. After all, LLM generalisation is fairly weak.

But I’ll argue this won’t be a problem for two reasons:

1. Learning algorithms developed in one domain tend to transfer very well to new domains. (And we shouldn’t get confused by the fact that trained neural nets generalise poorly between domains.)

2. AI companies will be much better placed than evolution to find algorithms that transfer to real-world tasks.

Firstly, learning algorithms developed in one domain tend to transfer very well to new domains. A neural net trained on chess does not generalise to Go, but AlphaZero is a learning algorithm that can learn multiple games and transfers very well. Humans trained to be physicists do not generalise easily to being doctors, but the same learning algorithms that allow humans to master physics also allow them to master medicine. A Transformer trained to predict text cannot magically label images, but the Transformer architecture initially developed for language processing turned out to also work for images, audio, and robot control.

So if a learning algorithm is developed in one domain, and it doesn’t look highly specialised to that domain, we should expect it to transfer well to very different domains.

People often point out that neural nets generalise poorly. This is a strong argument against the literal neural net trained during an SIE immediately being a drop-in replacement for all work.

But it’s not an argument against the learning techniques generalising far. Which is the argument I’m making here.²⁰

Secondly, AI companies will be much better placed than evolution to find algorithms that generalise to real-world tasks. The human learning algorithm was “built” blindly by evolution to make humans better at hunting on the Savanna. No attempt whatsoever was made to make it transfer beyond that. But it turned out to transfer to reading and writing, abstract maths, and indeed all modern economic tasks. This is a very striking empirical fact!

AI companies, by contrast, will be deliberately optimising for digital-to-physical transfer, with the ability to measure transfer directly, and iterate. They are in a far, far stronger position than evolution.

The following strategy seems potent: 1) develop a learning technique by iterating on a subset of digital tasks, 2) test transfer to other digital tasks (and a few real-world tasks), 3) keep iterating steps 1 and 2 until you find something that transfers well.

For example, do sample-efficient learning techniques developed on coding and math problems transfer to learning broad computer use, World of Warcraft, highly specialised software for financial accounting, and byzantine government IT systems?

Strategy for finding learning algorithms that work on real-world tasks. First develop sample-efficient learning algorithms on a subset of digital tasks; then test generalisation to digital and some real-world tasks.

This is a powerful strategy because the range of tasks that can be performed and verified digitally is already vast. And abundant AI cognitive labour can make this range much broader still. If a learning technique transfers across the large gaps between very different digital tasks, that is strong evidence it will also transfer the one further step to real-world tasks.

Further, AIs will purposely create virtual tasks that closely resemble real-world tasks. They will have lots of information about real-world tasks — from the internet, from human experts paid to provide this data, and from companies deploying AI. Again, evolution didn’t have this.

So, AI sample efficiency will be as good or better than humans. But AI still needs real-world data to learn real-world tasks.

How long will it take to gather the data to learn real-world tasks?

If the whole world worked together to make this happen quickly, it would not take long. For each job, 1000 people could record their screens and wear video cameras on their heads, and within 4 months AI would have 300 years’ worth of experience. For skills that are highly specific to a particular role, it would be slower: AI would have to learn from the one human doing the job, “shadowing” them like human apprentices do today.

The bigger uncertainty for me here is whether organisations will simply refuse to share their data. The argument for refusal is simple. For many companies, proprietary data is the moat. If they hand it to an AI company, the resulting AI gets deployed across the whole economy — including to their competitors — and the thing that made them special is now available to anyone. So they refuse. This is the strong default today: the winning enterprise AI pitch right now is “we will never train on your data”.

I don’t ultimately find this convincing, for two reasons.

First, the company can sell the moat at a very high price. AI trained on its data (and perhaps also the data of other similar companies) can generate far more value across the economy than the company could ever capture alone, so the AI company can pay well above the company’s expected future profits and still come out ahead — paying, for example, with shares in the AI company itself. The surplus from combining the data with the AI cognitive labour is enormous, and so there’s plenty of room for both sides to come out ahead. Today the surplus is much smaller, so it’s not surprising that the data isn’t sold.

Still, CEOs might refuse to sell the moat simply because they want to run their own business. Or they might believe they could make more money by holding out and extracting rents from their control of a scarce resource (see Against the Coasean Singularity for a strong version of this pessimism and see footnote for discussion of the game theory²¹).

Second, and more convincingly, data-sharing deals can be structured to preserve the moat. The AI is fine-tuned on the company’s data under an exclusive arrangement — the resulting AI works in their workflows and nowhere else. Their moat is now embodied in an AI rather than in their employees. Deals like this leave both sides better off, so I expect them to happen. Of course, learning will be slower than if data were pooled across many firms.

Even this deal might fail though. People may not trust the AI company to keep to the agreement. Human employees may strongly resist their own replacement. Legal barriers may delay certain uses of customer data (though I’m sceptical, see footnote²²). And human-run companies may just be slow to adopt new technologies, as is common.

But in competitive industries this situation is fragile: once one company in the sector has adopted AI, it will be very hard for others to hold out.

Overall, I’d guess that, in competitive industries, AI will learn to automate most work within a year of the SIE.

Conclusion

Here’s a summary of my views:

So my overall view is:

• Data bottlenecks might significantly delay automating AI R&D — this depends on how well RL generalises as we scale it up and, if it generalises badly, how hard it is to gather lots of data specialised to AI R&D tasks.

• Data bottlenecks are unlikely to stop a software intelligence explosion from accelerating over time. But there will likely be some slowdowns from human-derived training data losing its relevance as AI software advances, and from it becoming harder to increase data quality once it reaches the human-quality ceiling. I expect these slowdowns to be fairly minor: I’d be surprised if they make the SIE take >2x as long.

• After the SIE, I expect AI sample efficiency on real-world tasks to match humans. And in competitive industries, I expect them to rapidly have access to the real-world data needed to automate the work.

Acknowledgements: thanks for comments from Herbie Bradley, Daniel Carey, Alan Chan, Josh Clymer, Owen Cotton-Barratt, Tom Cunningham, Daniel Eth, Lukas Finnveden, Ryan Greenblatt, Brendan Halstead, Anson Ho, Eli Lifland, Alex Mallen, Sam Manning, Sören Mindermann, Dwarkesh Patel, Carl Shulman, James Tillman, and Philip Trammell.

This article was created by Forethought. See the original on our website.

1. Suppose that absent a bottleneck the intelligence explosion would take 4 months to go from human-level to somewhat-superhuman AI, and 1 month from there to superintelligence.

If a bottleneck slows down the explosion then it might instead take 6 months from human-level to somewhat-superhuman and 1.5 months from there to superintelligence. Progress still accelerates, but the pace of change at each level of capabilities is slower.

2. Of course, these phases will blur into each other. Scaling will continue even after significant AI R&D automation, and AI will be deployed across the economy somewhat during a software intelligence explosion. And if there isn’t an SIE, the second and third stages will blur together much more.

3. I’ve tweaked the definition a little compared to the one Ajeya uses, which is about whether an AI company would choose to fire the AIs or the humans. The firing decision brings in factors other than capabilities at AI R&D, like alignment or good judgement about whether to pause progress.

4. This definition is ambiguous: what if you’d make faster progress initially with AIs but then get stuck? Let’s peg it to 1 year of progress (at the rate of progress in 2020–25). You reach AI-human parity when only-AIs could make a year’s worth of progress as fast as only-humans.

5. For example, AIs can get a huge amount done, but may exhibit critically flawed judgement if not directed by a human. Humans could hand off all execution to AIs, spending 10x more time on high-level prioritisation and planning. Consider how much more productive a research manager can be running a 30 person team compared with just doing the research themselves.

6. How much might it speed up? If cognitive labour inputs are 10x bigger by this point, and the share of R&D progress attributable to cognitive labour (rather than compute) is ~0.5, then this would be a ~3x speed-up in software progress, and a ~2x speed-up in overall AI progress. (Numbers very rough of course!) Probably a bit less than this to account for some low-hanging fruit being plucked along the way.

7. For example, today's LLMs learn from ~5–6 orders of magnitude more tokens than a human ever processes. In-context learning is also much less flexible than humans, and does not work over similarly long time horizons.

8. But won’t fully automating AI R&D require automating some parts of the AI R&D stack for which there is very little data? In which case AI will need to match human sample efficiency? At the point of AI-human parity, in practice humans will continue to do those low-data parts due to strong comparative advantage. Without humans, AI would hack together some way to avoid those parts going catastrophically wrong — constant checking and testing and huge amounts of inference-time thinking. They would be a major drag, but AI would still achieve parity through its large advantages elsewhere.

9. And eventually AI will be much better than humans at learning new paradigms (h/t Carl Shulman). When a field advances, humans must also laboriously learn the new concepts — and no human will ever have ten years' experience with a brand-new technique. AI, by contrast, will be able to read everything written on the new concept and do RL practice on it in parallel across thousands of copies.

10. At least, slow down progress relative to a world where this dynamic was not in play. In practice it might just result in AI progress speeding up less than it would have otherwise.

11. There’s a question of how much work is being done to “derive” the data from human expertise or human-built artefacts. For example, you could take a human-built artefact and then design an RL environment to build a more complicated variant. Or you could augment a human expert demonstration to improve its quality. I’m here including simple methods for ‘deriving’ the demonstrations or RL environments, but not very complicated ones.

What about a human expert building an RL environment for a task that they could not solve, but that they identify as testing an important skill and can verify? The resultant trajectories are above the ceiling, because no human could have produced them. And this generation-discrimination gap shows why data quality will continue to improve past the ceiling! My argument is just that progress will be harder past this point.

12. See e.g. reporting that OpenAI’s “Project Mercury” paid over 100 former investment bankers (around $150/hour) to build financial models as training data.

13. E.g. Mercor, a marketplace of credentialed domain experts (reported at roughly a 2-billion-dollar annualised gross-revenue run-rate by mid-2026); Scale AI's restructuring of its contractor operations towards credentialed experts; and Surge AI's hybrid human–AI pipeline (revenues reported around a billion dollars). Expert-built evaluation and data efforts such as FrontierMath, Humanity's Last Exam, and HealthBench's physician-written rubrics point the same way. Vendor gross revenues, though, considerably overstate the genuinely above-frontier expert-labour input to frontier training — platform margins, non-expert crowdwork, and non-lab customers all inflate them — so these figures show where the field is heading, not that expert data dominates lab inputs.

14. See Mechanize on building RL environments from real-world software.

15. To make the difference explicit: today humans do not need to be “amplified” by thinking for ages to create higher quality demonstrations, but in the future AI will need to be amplified to increase data quality.

16. Here’s why being above the ceiling slows progress once rather than preventing it from accelerating.

Suppose we’re already above the ceiling and improve AI by one “notch”; this takes some amount of time. Now we’re even further above the ceiling, and we improve AI by another notch, which also takes some amount of time. The question of acceleration is how long this second step takes compared to the first — which depends on how much harder the second set of improvements is than the first. Now, for neither step can we draw on the latent stock of high-quality human data. That makes each step harder than it would otherwise have been — but it doesn’t apply more to the second step than to the first. So while being above the ceiling makes each individual step take longer, it doesn’t prevent the second step from being faster than the first. That’s a one-time slowdown, not a brake on acceleration.

17. In fact, on some ways of modelling it, the effect is reversed. Suppose data quality becomes continuously harder to improve the closer you are to the human-quality ceiling. Then we are already dealing with a slight slowdown each year. Reaching the ceiling will be just one more speed bump that’s already baked into the trend of software progress over time. But then after reaching that ceiling, there are no more speed bumps! So we transition from a world of annual speed bumps to a world with no speed bumps. For every step of progress, there’s now a speed boost relative to our expectations. Progress accelerates more quickly than we’d have otherwise expected (or, for those familiar with the mathematical models behind the SIE, r becomes higher). So the sign of the effect depends on whether we model the difficulty as a dichotomous effect that kicks in after reaching the ceiling, or as a continuous effect that grows gradually as we approach the ceiling. I’m inclined towards the former: there’s something fairly dichotomous about extracting latent data vs constructing data from scratch.

18. Indeed, while historically data quantity has grown by orders of magnitude, this was never going to be possible during an SIE. The whole point of an SIE is that the quantity of compute stays roughly constant. You can’t continually increase data quantity by orders of magnitude while holding training compute fixed. Indeed, for an SIE to become very fast, AI companies will likely need to reduce the compute used in training.

Caveat: I do think data quantity could increase somewhat. Today RL data is very sparse — AI takes 100s or 1000s of steps before getting a reward. Rewarding each intermediate step of its work would provide much more data per task (this is the core idea of process-based feedback). This could be a big one-time increase in data quantity, but wouldn’t be an ongoing core driver of the SIE.

19. See “improvements to the brain algorithm” here.

20. In-context learning is a bit of an in-between case because the neural net is fixed. But rather than the weights encoding expertise at any specific skill (as they mostly do today!) I’m imagining the weights implementing a learning algorithm, and the activations representing the newly learned knowledge and skills. Again, I think that for this to work neuralese would be required, and plausibly also architectural changes as large as the Transformer.

21. How could a CEO expect to make more money by withholding their proprietary data? Given the large surplus from selling their moat, surely there’s some trade that makes both parties better off?

Firstly, win-win trade may not be made for similar reasons why nations go to war despite peace being better for both parties. The companies may disagree about how valuable the proprietary data is (just as nations disagree about who would win a war). Companies may systematically overvalue their own data, and are incentivised to do this as a means to getting a better deal. Further, company CEOs may be able to leverage their role for personal gain in a way that isn’t possible if they sell (just as leaders of nations go to war to advance their personal political ends).

Second, when there are more than two trading partners in total (the AI company and multiple companies with proprietary data), it’s possible that there’s no stable equilibrium where every company sells its moat to the AI company. Suppose that the sandwich industry currently has profits of $3/year. These profits are shared equally between three companies: Bread Corp, Butter Corp, and Cheese Corp each make $1/year. If all three companies sell their moats to the AI company, the industry’s profits would rise to $9/year. If just two companies sell their moats, industry profits rise to $5/year. But in this case, the company that hasn’t sold their moat AI is the scarce resource and captures $4/year profit. It’s not in the interests for any pair of two companies to sell their moats – they’re combined profits go down. And the equilibrium where everyone sells their moat isn’t stable because some company will make <=$3/year and could do better by being the scarce resource. Of course, if all companies could coordinate to sell their moats and split the gains equally, they’d all be better off, but this coordination might not happen. (I’m not sure how realistic this second situation is, but it is a coherent possibility.)

22. I don't think this legal barrier is that strong: companies already share sensitive data with third-party processors like cloud providers under standard legal agreements, fine-tuning can happen within the company's own environment so data never leaves its control, and AI itself will be far better than today's tools at reliably redacting protected material. But it could add real delay in heavily regulated industries.