Building the Knowledge Base, Pt. 1
In search of the Stack Overflow for lawyers
A programming note: In a couple of weeks I am submitting a short paper for an AI & Access to Justice workshop, about building useful “knowledge bases” for legal aid lawyers.The post below and next week’s post will be “working drafts” of that article. Feedback is always welcome, but especially so in this case.
Since large language models first entered the public consciousness 2.5 years ago, the most compelling practical application so far has been in coding. I personally use language learning models more frequently to code than for other "knowledge work" type tasks, and my behavior is consistent with labor market studies such as this one from Anthropic. And, to date, the startups that appear to be generating the most revenue (i.e., signing up actually paying customers) have been coding assistants.
This shouldn’t be that remarkable; noting that "people in tech are the first people to adopt a tech product" has a certain man-bites-dog quality. Coders, as a group, are more likely to adopt a technology early, and the people who make language models are coders themselves or live and work in a technology-oriented culture. Nor is this evidence that AI applications outside of coding are necessarily bound to fail. The average person probably did not see the use of the early 90s Internet when it was mostly comprised of nerds making HTML pages.
1.
This symbiosis between coders and the Internet means the latter is littered with programming content, and this is the content the big models have been trained on. Even if you indiscriminately scraped this data and ran an unsupervised machine learning algorithm over it (i.e., what OpenAI et al. have done), your resulting model would do pretty well when faced with a coding question. Chatbot users figured this out early on, and the big AI companies have presumably done plenty of work behind the scenes to enhance this feature. I only started using language models for my own coding last summer, and in that time I've noticed significant improvement. And public AI researchers—again, who are almost always coders themselves—have published plenty of open research exploring how to use the Stack Overflow data more effectively in model training.
Over the past 10-15 years, the go-to resource for programming questions on the Internet was Stack Overflow. Fundamentally, Stack Overflow is a discussion forum where users can ask programming questions and get them answered. But it was successful because it included several crowd-sourcing features—such as upvoting—that made it easy for a later user to find the best answer. Thus, by the early part of this decade, the easiest way to get "unstuck" on a programming problem was to simply google "stack overflow + [my problem]" and then browse Stack Overflow threads until you found what you were looking for. Not coincidentally, sites like Reddit that adopted the same features had very active Subreddits for help with programming issues. And it wasn't just amateurs; people who program for a living would frequently post on Stack Overflow and look up posts when dealing with a work problem.
But programmers are outliers. Lawyers and most knowledge workers do not have anything resembling Stack Overflow. Just because there is not a "Stack Overflow for lawyers" does not mean that lawyers do not talk to each other. Instead, information is either stored within the Company or shared through professional networks. But because most of that conversation does not happen in view of "the Internet," we cannot expect general-purpose chatbots to be particularly useful tools for lawyers.
2.
I am slowly burnishing a conviction that the correct way to think about language models is as "cultural" technologies, as argued in this essay, of which the primary benefit is that they "allow[]" humans to take advantage of information other humans have accumulated." In contrast to overblown debates about "AGI," the authors argue
large models combine the features of cultural and social technologies in a new way. They generate summaries of unmanageably large and complex bodies of human-generated information. But these systems do not merely summarize this information, like library catalogs, internet search, and Wikipedia. They also can reorganize and reconstruct representations or “simulations” of this information at scale and in new ways, like markets, states, and bureaucracies. Just as market prices are lossy representations of the underlying allocations and uses of resources, and government statistics and bureaucratic categories imperfectly represent the characteristics of underlying populations, so too are large models “lossy JPEGs” of the data corpora on which they have been trained.
While this is certainly less exciting than an argument that our jobs will be replaced in X number of years by superintelligent agents, one may still argue (as do the authors) that the impact of language models will be profound. This is true especially in knowledge work, where most people's jobs (lawyers included) involves making sense of large amounts of information and repackaging it in a format that has business value. In these specific environments, introducing a tool that theoretically makes all an organization's information accessible through natural language inputs is a very big deal!
But doing this is very hard, and the achievement is going to look very boring to an average consumer. As Henry Farrell, one of the co-authors on the cultural technologies essay, argues elsewhere:
That means that LLMs are comparable in kind with previous innovative technological leaps such as the filing cabinet and the inter-office memo, though considerably more complex and sophisticated. Those were both transformative technologies in their time. That we do not see them as such - that we think of them as boring and mundane - is a mark of how profoundly our understanding of organizations has been transformed.
By the same token, Ben Thompson of Stratechery argues that AI adoption will take us "back to the future." By this he means that—because of the personal computing and "software as a service" revolution of the past 30 years—we are conditioned to think that AI adoption will follow a similar adoption pattern, where the key challenge to make a tool that is useful for an individual. But, Thompson argues that if we look at the longer history of technology and enterprise, we see that the adoption is done by the business itself, and the personal adoption follows.
[T]he relationship of most employees to AI is like the relationship of most corporate employees to PCs in the 1980s; sure, they’ll use it if they have to, but they don’t want to transform how they work. That will fall on the next generation.
Executives, however, want the benefit of AI now, and I think that benefit will, like the first wave of computing, come from replacing humans, not making them more efficient. And that, by extension, will mean top-down years-long initiatives that are justified by the massive business results that will follow. That also means that go-to-market motions and business models will change: instead of reactive sales from organic growth, successful AI companies will need to go in from the top. And, instead of per-seat licenses, we may end up with something more akin to “seat-replacement” licenses (Salesforce, notably, will charge $2 per call completed by one of its agents). Services and integration teams will also make a comeback. It’s notable that this has been a consistent criticism of Palantir’s model, but I think that comes from a viewpoint colored by SaaS; the idea of years-long engagements would be much more familiar to tech executives and investors from forty years ago.
3.
If this view is right, teaching "AI skills" to lawyers is unlikely to drive adoption. The key for adoption is to figure what information sources lawyers currently use, how to make those sources available to a language model, and design the user experience so that it fits the existing information flow.
In a large law firm, the organization itself likely contains sufficient information. In a large firm with many practice groups containing dozens and dozens of attorneys, enough work is churned out on a daily basis that the “knowledge” being sought through the language model will likely exist somewhere within the firm already. It may not be organized well or easily identified, but if someone has a question, it's a good bet that the answer exists somewhere within the firm's knowledge base. The technical challenge going forward is to figure out how best link the users to that information, through the language model.
For solo practices, small law firms, and legal aid offices, this assumption likely does not hold. Instead, in my experience working mostly at small law firms and nonprofits, lawyers create online communities for education and information-sharing, often in the form of email listservs. For me, the paradigm of an active lawyer listerv is the "Housing Justice Network" listserv moderated by the National Housing Law Project. It's primary users are legal aid and small-time attorneys representing tenants in eviction and other housing matters.
It is extremely active: On an average day, there are usually a dozen or so original posts asking a question about some fine point of housing law, often related to the very complicated world of federally subsidized housing. But it's also useful: Attorneys at the National Housing Law Project monitor the listserv and provide very detailed and helpful answers, and the many other attorneys on the listserv will also provide helpful information.
I do not think HJN is an outlier; I've been part of many other active listservs for different topics. Indeed—confirming my view that there is a Law Review Article for Everything—there is a law review article discussing the listserv for the Bankruptcy Law Section of the State Bar of Texas. The author, Josiah M. Daniel III, a retired bankruptcy lawyer in Texas, uses a descriptive discussion of the listserv to validate the notion of "listserv lawyering" as a worthwhile approach to practicing law.
His first point in favor of listservs is that the community of solo and small firm attorneys serves the same information-gathering function that would be filled by an attorney's colleagues at a large law firm.
First, the listserv's communication of the multiple types of questions, answers, notifications, and other messages is the sort of back-up and mutual assistance that is characteristic of lawyers practicing in "multi-staffed legal organizations"—law firms that are large enough to house more than one lawyer in the same practice area. But the SBOT listserv is obviously not a law firm, and many, if not most, of its members are solo or small firm practitioners. Such lawyers often lack easy access to pricey resources such as Collier on Bankruptcy, the leading bankruptcy encyclopedia, and specialized treatises such as Collier Consumer Bankruptcy Practice Guide, Collier on Bankruptcy Taxation, Bloomberg Law: Bankruptcy Treatise, and HeinOnline for access to bankruptcy-topical journals and articles within the broad universe of law-journal literature.
In this way, Stack Overflow is instructive by analogy; even though small-firm lawyers are not posting their practice tips on public forums, they are sharing this information somewhere. Next week I'll look more closely at how Stack Overflow content helps language models work better, and sketch out how we might use listservs in the same way to improve the usefulness of language models for solo, small firm, and legal aid lawyers.