|
 |
On one hand, I love AI technology. On the other hand, I do think there’s a substantial chance that AI will kill most people on Earth within the next decade or two, by designing superviruses. AI is already capable of designing viruses not found in nature, so this isn’t a sci-fi scenario.
Whether these superviruses would be designed and unleashed by nihilistic human individuals, doomsday cults, or rogue AI agents themselves might end up being a secondary question. We know we have nihilistic human individuals who might decide to destroy civilization in a fit of depression or pique. We know we have doomsday cults. The will to destroy humanity exists, and sufficiently capable AI will probably provide a way, if sufficient precautions are not taken. But right now, nobody really knows what precautions will be sufficient.
One idea — promoted by the big AI labs themselves! — is to intentionally slow down the development of AI capabilities. This could conceivably buy us time to take other precautions, such as improved security around bio-labs, better AI alignment, and so on. Intentionally slowing AI development is called “pacing”. The biggest question facing the “pacing” debate right now is whether to curb the use of AI to design better AI — often called “recusive self-improvement”, or “RSI” for short.
I haven’t waded into the pacing debate myself, but as a start, I thought it would be interesting to publish the thoughts of the good folks at the Institute for Progress, whose judgement I generally trust. Part 1 (today’s post) covers how seriously we should take this possibility of RSI, and whether it justifies slowing down frontier AI development. Part 2 will cover policy recommendations. If you work in US policy and would like to connect with the authors, you can reach Tim Fist at tim.fist@ifp.org and Saif Khan at saif@ifp.org.
Frontier AI companies are racing to automate the development of AI, but they seem increasingly worried about what will happen if they succeed.
More than 1,300 employees across every US frontier AI company recently signed an open letter calling for the government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” The official OpenAI and Anthropic accounts tweeted messages in support of the letter, and the same day Sam Altman told an interviewer “we may have to pace the rate of AI development.”
But before considering whether the letter is relevant for government policy, we have to answer two questions: What does “pacing” actually mean? And does the argument for it stand up to scrutiny?
In our view, the letter is implicitly arguing three things:
Frontier AI companies are close to fully automating AI R&D.
Automating AI R&D would pose serious risks.
Building the option to “pace” — i.e., somehow slow down progress toward fully automated AI R&D — is a good way to address those risks.
Slowing down AI progress should not be taken lightly: Advances in AI could unlock massive societal benefits, from new cures for diseases to abundant robotic labor.
Yet if the AI researchers and CEOs are right about automating AI R&D — both that they could do it and that it would be extremely risky — the right tradeoffs for policymakers might look very different when we get there. With the right preparation, we might be able to manage the risks of automated AI R&D while having AI’s capabilities progress faster and diffuse more broadly than they do today.
So, despite substantial uncertainty, we believe the US should take low-regret policy actions now to prepare for a possible future in which serious risks from automated AI R&D require some form of “pacing.”
Here we’ll explain why, including what makes us take the open letter’s claims seriously and our principles for choosing policies with minimal downside if the risks prove overblown. In the next post, we’ll provide a detailed list of specific policy recommendations.
Are frontier AI companies close to fully automating AI R&D?
Frontier AI companies are racing to automate AI R&D. Sam Altman, for example, stated last year that OpenAI aimed to have a “true automated researcher” by March 2028, and Anthropic’s leaders have made similar predictions. But how do those goals stack up to reality?
One way of answering is to look at how models are getting better over time at AI R&D. You can break down the skills required for an AI model to do AI R&D into two broad categories: software engineering, where the model writes code for research experiments and training runs, and research taste, where AI models decide which experiments are worth trying.
For software engineering, AI capabilities appear to be increasing exponentially. When AI models are evaluated against how long it would take humans to complete the same engineering tasks — so-called “time horizons” — their capabilities seem to be doubling every 7 months.
 |
These capability improvements apply to the software engineering tasks required for AI R&D. In a long-running experiment, researchers at Anthropic have found that their models now significantly outperform humans under a fixed time budget on an AI R&D task focused on speeding up AI model training.