Why we don’t need an IPCC for AI

N.b. I wrote an early version of this post in November 2023; I substantially revised it in March 2024.

The Intergovernmental Panel on Climate Change is a body of scientists charged by the UN to provide authoritative scientific reports on the status of climate change, especially to governments. A number of writers have recently called for an IPCC for AI (Suleyman et al., 2023, Mulgan et al. 2023, Maihe 2018). Full disclosure: I contributed to one of these calls. These calls tend to focus on AI safety, and their general premise is that:

1. AI poses a safety risk,
2. governments need to act,
3. to act, governments need a neutral, scientific assessment of the state of AI, and 
4. an IPCC-like solution can best offer that assessment.

What follows are some notes on why the IPCC is not the right institutional solution for assessing the state of AI. Effectively, IPCC : climate :: IPAIS : AI is an incomplete analogy and thus suggests an incomplete institutional response.

First off, I’m not here to discuss (1) whether AI poses a safety risk, i.e. the nature of the problem and the relative importance of AI safety, AI ethics, AI alignment, AI fairness, AI industrial organization, AI benefits, AI etc. etc. Nor do I want to dispute, defend, or dwell on (2), whether governments need to act. 

An IPCC for AI is better than nothing. So if that’s what’s on the table, then let’s go for it—though we should be aware that a ton is happening already on the observatory side, e.g. Stanford HAI, OECD, various incident and risk databases, etc. My real focus is on whether (3) governments need a neutral, scientific assessment of the state of AI and whether (4) an IPCC-like solution can best offer that assessment.

Climate and AI are pretty different as systems

This is obvious, but let’s list the ways.

1. Climate is a big, global system with a lot of observables. Hard to hide—though that’s with the benefit of building lots of efficient measurement devices and proxy methods.
2. AI is not physics. A lot of hidden variables in private companies or even in open source repos. AGI can (maybe) emerge from anywhere. Some stuff is not that easy to hide, e.g. buying 100M in compute. Other stuff, such as research, is easier to hide. The risk here is not just that some research will never be published, but that there is a risky delay between when things are developed and when things are made public.
3. AI is a research field. It’s a human system, not a natural system. The risk is a technology, i.e. a form of (human) knowledge, not a relatively public, shared state of the world. The rate of change is likely to be much faster than climate change, and harder to model. Not impossible, of course—things like scaling laws and biological anchors give us a shaky way of extrapolating future outcomes. But our social science for AI is not as developed as our climate science.
4. No one has really characterized the properties of AI research as a whole as a system. There are at least two methods for characterizing these properties. One method is to analyze the technical results of AI research as a whole, along the lines of how we analyze entire bodies of mathematics in the foundations of math or in category theory. For example, efforts to map out how iterative improvements in training techniques contribute to scaling laws. Another method is to consider AI research as a large, human, economic system—which it is, insofar as AI researchers are shaped by incentives. To a large part this is a question of AI industrial organization, which we need more of.

An IPCC-like entity is not a realistic assessment mechanism

Largely this is due to the incentives of private corporations and the decentralized nature of AI production (and AI risks).

IPCC is best understood as a giant peer review process for a bunch of climate scientists, backed by the UN. If you wanted IPCC for AI, you’d need to get a lot of AI researchers in the room participating in this long, bureaucratic peer review process. But you don’t really see the AI researchers themselves emitting these calls for an IPCC for AI. Why? First, there’s not as much agreement in the AI community about risks as there is in the climate community. Plus, the incentives aren’t there. It’s already hard to get good reviewers for normal conferences and journals in AI, and AI researchers have sharper economic incentives than climate scientists: you can’t really create companies with your climate skills or command 7 figure salaries, but you can with a sufficiently good background in AI (at least for now).

The biggest issue is that for an IPCC-like approach to work, you would need to get the major private companies conducting AI research—e.g. OpenAI, Microsoft, Google, etc.—to fully participate in this process. You would need to get them to publish and share capabilities research. But these are the same labs who are already stepping back from publishing capabilities research.

Something symptomatic of the IPCC approach: recently, I was on a panel where someone from the UN suggested, let’s create a panel to study this. Yes, let’s create another committee to study the issue! 🙃

Governments need to get used to staying in the dark

I don’t think a neutral, scientific, IPCC-like assessment of AI is possible. Not that you couldn’t try, but that once you did it you would find it’s full of holes that undermine the science, that it’s biased by the people who are incentivized to participate, that it was already 2+ years out-of-date, and that governments would not pay attention to it. But that doesn’t mean more assessments aren’t useful.

Here’s a lesson that I learned about from an old marine. In combat, the infantry digs in; they have something like an 80% read of the situation in their area. When the calvary (i.e. a tank unit) comes in, it should at best expect to know 20% of what’s going on. Effective cavalry action requires having infantry in place beforehand.

The government is the calvary in this analogy. And it needs infantry support with up-to-date, local information, not a global synthesis.

A global network of AI laboratories?

I don’t know how to resolve the incentive and informational issues with an IPCC-for-AI. Instead, I’ll try to illustrate the difficulty of the problem by dissecting an alternate proposal for shared AI laboratories, i.e. a network of affiliated private and public laboratories across industries and regions where each laboratory conducts their own AI research but also contributes / benefits from some shared infrastructure for AI research.

I came up with this proposal at some point in summer 2023, as part of discussions with my old GAIO collaborators. I’ve slightly updated it here in response to feedback, and to align it more closely with a similar proposal by Yoshua Bengio for a multilateral network of AI laboratories.

Here’s the premise of shared AI laboratories (SAILs):

1. AI poses a (global) safety risk
2. the entities doing AI research are the ones who need to act
3. people and companies need to be incentivized to act
4. a network architecture built on top of shared infrastructure is the best way to shape those incentives

The underlying aim of the proposal is to integrate a global assessment of the state of AI directly into the local practice of AI research.

But the shared AI labs proposal has problems too!

First off, why would labs want to be involved in this network, any more than they would want to be involved in an IPCC-for-AI? At least IPCC-for-AI has the notional prestige of bodies like the UN behind it. And people have tried networked approaches before, e.g. through the well-meaning but so-far ineffectual Partnership for AI.

To answer this question, the SAIL proposal leans heavily on the idea of shared infrastructure—certainly more heavily than Bengio’s proposal, which presumes public funding and some form of UN supervision. The proposal wants to delegate core informational and incentive problems onto some sort of magical technical infrastructure. But why would labs want to use this infrastructure?

Generally, private labs like OpenAI and Microsoft do not need or want shared infrastructure. They are not and will not be the primary consumers of public compute infrastructure like that being proposed through NAIRR or Calcompute. They might publish certain models through hubs like HuggingFace. But they don’t need any shared infrastructure to do core capabilities-based AI research.

I don’t know what this infrastructure will look like. The only thing I’d say is that we’re translating some of the institutional requirements that we observed in critiquing IPCC-for-AI (e.g. informational and incentive requirements, governance requirements, speed requirements) into a set of technical requirements for a particular technical infrastructure for AI development.

Maybe it’s just me, but I like the technical way of thinking about and representing the problem. People in AI have been building technical infrastructures for AI development for decades (neural networks being one example). The global, politico-economic aspect is new. But if we adapt and translate some of the new constraints (safety, ethics, bias, interpretability etc.) into a new regime of technical infrastructures, that could help us make progress in both more capable and safer AI. I’m reminded of what Joanna Bryson once told me: you don’t need to regulate engineers to do “safe” or “interpretable” things, just convince them that interpretable architectures promote good engineering.