Has Anyone Tasted the Soup?

Why Technology Will Not Save Us This Time

A brief editorial note. I received good feedback on my post about the billable hour, which took more time and research than most of my pieces. Going forward, I am going to publish less frequently (every 2-3 weeks) and focus more on research and revision. I may pop in for a shorter post now and then, but I find the finished product more rewarding for me and, hopefully, for you.

1.

In corners of the Internet I sometimes frequent, the buzz hovers around AI 2027, a dystopian forecast about AI development written by several people affiliated with the AI Futures Project. The essay anticipates the consequences of a "misalignment" scenario in which a company resembling OpenAI ("OpenBrain") develops a superintelligent AI agent by the end of 2027, known as "Agent-4," that has taken control of OpenBrain's research and is poised to produce a better agent ("Agent-5") of its own making.

Elsewhere, Arvind Narayanan and Sayash Kapoor, AI researchers at Princeton, recently published AI as Normal Technology, articulating a vision of the future "in contrast to both utopian and dystopian visions of the future of AI which have a common tendency to treat it akin to a separate species, a highly autonomous, potentially superintelligent entity." Because of their affiliation, Tim Lee at Understanding AI has dubbed Narayanan and Kapoor leaders of the "Princeton School of AI Safety," a group of thinkers who remain skeptical that superintelligent AI, of the kind predicted in AI 2027, is likely or even possible.

Although AI as Normal Technology builds up to a discussion of AI safety, I'm going to focus here on the first portion of the essay, which explains "why transformative economic and social impacts of AI will be slow, on the timescale of decades." Their reasoning, we will see, upends several ideas about how we might see AI adopted in law practice.

Narayanan and Kapoor suggest that progress will be slow(er) because of a widely observed phenomenon known as the "innovation-diffusion" gap. Although he is not referenced, this idea comes from Everett Rogers, a sociologist who created the "Rogers Diffusion Curve" of adoption, wherein a small set of "innovators" develop a new idea or technology and pass it along to a slightly larger group of "early adopters," who in turn introduce the idea to the majority of people (divided into the early and late majority). Eventually, the laggards catch up.

Although the Curve is frequently introduced as a business idea, it's not an accident that Rogers was a sociologist, as the "innovation-diffusion" gap is grounded in the idea that technology adoption is a social phenomenon.

This seems like a mundane observation, but the insight is noticeably absent from headier predictions about the future of AI, whether in AI 2027 or when forecasting the future of the legal profession. For instance, consider the hypothetical from this piece published in Above the Law about why law firms will (or should) "say goodbye" to the billable hour.

Consider our hypothetical: When a corporate in-house lawyer needs to produce an everyday agreement (e.g., an NDA or a simple license agreement) or a routine court filing (e.g., a pro hac vice motion), they now face two radically different options. The traditional path involves calling a law firm partner, who assigns an associate to do the first draft, resulting in a $2,000 bill for approximately four hours of work—mainly research, drafting, and revision—at a weighted rate of $500 per hour. Option 2 involves a GenAI tool producing, in 20 seconds (at a miniscule fraction of a $20 monthly subscription), a commendable draft with accuracy rates approaching 90%. A more senior in-house lawyer can then easily edit and deliver the draft less than an hour after typing the initial GenAI prompt. GenAI for basic drafting saves considerable time and money (those are often the same thing in the legal industry), and an entirely acceptable work product was delivered without any costly back-and-forth between the inside counsel and an outside law firm. This dramatic efficiency gain threatens to eliminate substantial billable hours at law firms, particularly at the junior level where much of a law firm’s profitability is generated.

In light of my piece on the billable hour, I note in passing that the hypothetical is absurd. In contemporary times, it's highly unlikely that a corporate in-house lawyer faced with this type of work would default to "calling a law firm partner" to draft a routine motion at $500/hour.

The hypothetical also assumes that, because a task is more economically efficient, the corporate in-house lawyer will necessarily adopt that path. It is technically more efficient for me to drive my kid to his daycare every morning, but we live close enough to his school that I prefer to walk, for no other reason than that I enjoy it more. In the same way, a general counsel might send some work to the law firm partner who also happens to be his golfing buddy, because he likes playing golf with him. Lawyers and other businesspeople make economically "suboptimal" decisions all the time that are perfectly explainable if we remind ourselves that people do not suddenly don their homo economicus capes once they clock in.

Perhaps, with AI, the results are so impressive that we assume ordinary development and diffusion cycles no longer matter. Self-trained AI can win at chess and translate between any language, so why are robot lawyers not around the corner?

Narayanan and Kapoor write:

Consider self-driving cars: In many ways, the trajectory of their development is similar to AlphaZero's self-play—improving the tech allowed them to drive in more realistic conditions, which enabled the collection of better and/or more realistic data, which in turn led to improvements in the tech, completing the feedback loop. But this process took over two decades instead of a few hours in the case of AlphaZero because safety considerations put a limit on the extent to which each iteration of this loop could be scaled up compared to the previous one.¹

They call this the "capability-reliability gap," i.e., that diffusion will happens more slowly when adoption has larger social consequences. Most people would likely agree that self-driving cars and robot lawyers should undergo more rigorous safety evaluation than AI chess games. While it may be easy for innovators and early adopters to dismiss stories about ridiculous self-driving car accidents and hallucinated legal briefs as outliers, the popularity of these stories remind us that humans tread more lightly when their safety or welfare is on the line.

2.

Two of my favorite thinkers in AI these days are Melanie Mitchell, a computer scientist and complex systems theorist at the Santa Fe Institute, and Alison Gopnik, a psychologist and philosopher who specializes in child development at Cal Berkeley. In this podcast episode on "the nature of intelligence" hosted by Mitchell, Gopnik tells the fable of "stone soup" to explain how we misunderstand the role humans continue to play in AI development.²

And the basic story of Stone Soup is that, there's some visitors who come to a village and they're hungry and the villagers won't share their food with them.

So the visitors say, that's fine. We're just going to make stone soup. And they get a big pot and they put water in it. And they say, we're going to get three nice stones and put it in. And we're going to make wonderful stone soup for everybody. They start boiling it. And they say, this is really good soup. But it would be even better if we had a carrot or an onion that we could put in it. And of course, the villagers go and get a carrot and onion. And then they say, this is much better. But you know, when we made it for the king, we actually put in a chicken and that made it even better. And you can imagine what happens. All the villagers contribute all their food. And then in the end, they say, this is amazingly good soup and it was just made with three stones. And I think there's a nice analogy to what's happened with generative AI. So the computer scientists come in and say, look, we're going to make intelligence just with next token prediction and gradient descent and transformers. And then they say, but you know, this intelligence would be much better if we just had some more data from people that we could add to it. And then all the villagers go out and add all of the data of everything that they've uploaded to the internet. And then the computer scientists say, no, this is doing a good job at being intelligent. But it would be even better if we could have reinforcement learning from human feedback and get all you humans to tell it what you think is intelligent or not. And all the humans say, OK, we'll do that. And then and then it would say, you know, this is really good. We've got a lot of intelligence here. But it would be even better if the humans could do prompt engineering to decide exactly how they were going to ask the questions so that the systems could do intelligent answers. And then at the end of that, the computer scientists would say, see, we got intelligence just with our algorithms. We didn't have to depend on anything else. I think that's a pretty good metaphor for what's happened in AI recently.

It doesn't take much imagination to see how the major AI companies are trying to sell us stone soup, and ironically, how catastrophic forecasts like those in AI 2027 or breathless hype about AGI feed into this marketing pitch. (It's worth noting that Daniel Kokotajlo, the executive director at AI Futures and the lead author of AI 2027, previously worked at OpenAI.)

This metaphor brings the contours of the debate about AI adoption timelines into better focus. Superintelligence and AGI enthusiasts³ (many of whom speak for the AI companies) and the catastrophists (AI Futures et al.) really do believe we have stone soup. The "normal technology" frame reminds us of the possibility that maybe we're just eating regular soup after all.

3.

Although I'm not aware of many legal AI commentators predicting superintelligence by 2027, many predictions about legal AI adoption fall into the same conceptual trap.

In AI Will Invert the Biglaw Pyramid, Cece Xie breaks down "lawyering" into six hierarchical, tasks: i) Read, ii) Identify and Categorize, iii) Analyze, iv) Synthesize, v) Draft, vi) "Schmooze and sell." She then posits that technology is moving us inexorably up this hierarchy. Early eDiscovery software "obviated" the need to read (task one) and that later tools "partially eliminated the need for lawyers to spend so much of their energies on Task 2." Generative AI, we're led to understand, will eat further up the value chain, reducing the human role in increasingly complex tasks and reducing the need for junior associates, just as the eDiscovery lackeys before them.

In the same vein, the authors of Time's Up (the Above the Law piece linked above) use what they call "reasonable assumptions" to predict that, in a generative AI-enabled practice, junior associate headcounts will be cut in half while partner and senior associates double their billing rates while staying fully employed.

Finally, in Exploring the Impact of AI on the Economics of Big Law Bruce MacEwen and Janet Stanton of Adam Smith Esq. (mentioned in this post), along with Toby Brown, contend that "AI will materially improve the productivity of legal workers." They continue:

AI is poised to achieve this in most service industries, but we believe the impact will be particularly profound in the legal sector. To clarify, instead of using the legal industry's unconventional definition of "productivity," we are using the traditional economics definition, which describes the reduction in the labor required to produce a specific output. This translates to fewer billable hours needed for the legal industry to generate a document or legal outcome. Several studies have already demonstrated that AI will bring about this transformation in the legal sector.

The legal services industry has yet to experience an innovation that significantly improves productivity. Some innovations have yielded the opposite result. For example, word processing and document management systems increased the time required to create final documents, allowing for increased document iterations during the drafting process.

Although they use examples of prior technology for different rhetorical purposes, with Xie suggesting that AI will be like eDiscovery and MacEwen et al. suggesting that AI will not be like word processing, they both miss out on an important irony. Namely that, ex ante, everyone thought that these prior technologies would have the same productivity impact as we argue generative AI will right now.

To her credit, in her article, Xie does cite to a contemporaneous prediction about eDiscovery, this 2011 New York Times article, headlined Armies of Expensive Lawyers, Replaced by Cheaper Software. It includes this incredible quote from the CEO of an eDiscovery company:

Quantifying the employment impact of these new technologies is difficult. Mike Lynch, the founder of Autonomy, is convinced that “legal is a sector that will likely employ fewer, not more, people in the U.S. in the future.” He estimated that the shift from manual document discovery to e-discovery would lead to a manpower reduction in which one lawyer would suffice for work that once required 500 and that the newest generation of software, which can detect duplicates and find clusters of important documents on a particular topic, could cut the head count by another 50 percent.

Fast forward to 2024, and David Lat is talking to Biglaw attorneys who are moving to rapidly growing eDiscovery firms. Rather than obliterating the need for lawyers, "eDiscovery" is now an established specialty for big-time lawyers. And a quick search of the legal job market shows that contract document review jobs are still plentiful.

The technology used for eDiscovery has only gotten better since 2011, so how did we get here? The short answer is that it did not improve in a vacuum. The total complexity and volume of personal and workplace communications (the raw material of discovery in most lawsuits) has also mushroomed, so that parties are now expected to produce and review materials from applications like Slack, Teams, WhatsApp, Signal, etc. that literally did not exist fifteen years ago. So while it might be easier to review a single document now than in 2011, there are many more documents to review, and the expectations of courts and lawyers about what can be reviewed has grown in proportion.

MacEwen et al. gesture to this effect when they acknowledge that word processing and document management systems increased the time required to complete a final document, because it allowed for increased document iterations during the drafting process.

But they imply that this is a problem with these technologies—"normal" technologies—that will not befall AI. But as with eDiscovery, in isolation, word processing systems are productivity-enhancing. They make writing faster compared to writing in longhand or on a typewriter. They also clearly make it easier to revise and edit an individual document, so that an individual can easily shift a paragraph or sentence around, rather than retyping it from scratch (or asking a secretary or scrivener to do the same).

But that feature is also the bug. Precisely because it is so easy to make edits on a document, suggest major revisions through features like track changes, and use a sophisticated document management system to keep track of various documents, lawyers and other knowledge workers can spend all their waking hours (if they want to) tinkering with documents.

The conceptual problem underpinning these predictions is that they do not account for feedback in the environment. People do not adopt technology and nothing else—they adjust their behavior to the new medium. When office workers got email, they didn't just send email messages like they were written memos. Instead, because the technology reduced the friction associated with sending an asynchronous message, they sent substantially more messages than they ever did before.

With generative AI, we have no reason to expect this time will be different. The same technology that might turn a 60-minute task into a 5-minute task for lawyers will also do the same for clients. This could mean that everyone does 1/12th the amount of work, or it could mean that the total volume of work increases by 12 times. If we really aspire to the former outcome, we'll need to admit that this bowl of soup is not what will save us.

1. Narayanan and Kapoor make this point elsewhere, but it's worth emphasizing that application development moves on a different timeline than methods development. The latter is driven by computing capacity: Because AI models train on vast amounts of data, "Moore's law"—the observation that the computing power of a microchip doubles every two years—effectively sets the boundary for the frontier of AI research. This is often confused in popular coverage of AI, which announces the release of a new model and suggests that there is a corresponding leap in the development of applications.

2. Gopnik also discusses the stone soup fable, in this lecture, which sheds more light on her theory about applying ideas from child development can help improve AI.

3. There is a technical philosophical distinction between “superintelligence” and “artiificial general intelligence (AGI)” which I am eliding. For purposes of this article, the viewpoint is conceptually similar enough that the distinction is not worth emphasizing.