Hallucinations, Pt. 2
I'm worried about the little guys
A few weeks ago, I examined the circumstances in the now infamous Mata v. Avianca case, where lawyer Steven A. Schwartz—believing he had access to a "super search engine"—used ChatGPT to write a brief that included many hallucinated citations. From this, I asked: Why did Mr. Schwartz feel compelled to turn to ChatGPT in the first place?
I concluded that, perhaps, the choice pointed to a broader issue about information disparity. As a scrappy plaintiff's personal injury lawyer, Mr. Schwartz and his team had bare-bones legal research access, leaving them ill-equipped for federal court and the vagaries of the Montreal Convention and the U.S. Bankruptcy Code. I ended the article discussing a 2013 article, Bad Briefs, Bad Law, Bad Markets by law professor Scott A. Moss.
The Case for Settlement Mills
For Part 2, we'll start by returning to Bad Briefs. As a refresher, Moss tries to answer the question: Why do (so many) plaintiff's lawyers submit (so many) bad briefs? To study this empirically, he designs an interesting natural experiment where he looks at summary judgment briefings in Chicago and Manhattan federal employment discrimination cases that raise the "same-actor" defense under federal employment discrimination law. The experiment is useful because Chicago and Manhattan sit within federal circuits where, at the time, there is an "intra-circuit split" on the viability of this defense. In essence, both the plaintiff and defendant in the cases studied by Moss could submit "winning" briefs by finding good law to match their facts. The failure to do so provides a clear signal of poor underlying quality. Unfortunately, the vast majority of plaintiff's briefs studied by Moss--around 75 percent--are "bad" briefs that result in losses for the plaintiff.
Moss tests several hypotheses for why this is. The one we'll explore here is what he calls the "lazy lawyering" approach, wherein Moss describes how plaintiff's attorneys—who typically get paid through contingency fees of 33-40%—have an economic incentive to minimize the hours worked on any one case. There is some math involved, but the basic idea is that even though a diligent attorney might obtain more revenue, the lazy approach can be more profitable because (with a 33% contingency fee) the increase in overall recovery needs to be $2 for the lawyer to justify an additional $1 spend in cost.
Referencing a paper written by law professor Nora Freeman Engstrom, Moss calls the law firms that exploit this strategy "settlement mills" and finds that many of the bad briefs in his sample were submitted by settlement mill practitioners; in Moss's analysis, these are general injury attorneys who tend to advertise heavily and often do not market themselves as employment lawyers. In contrast, many of the good briefs—and none of the bad briefs—are submitted by "expert plaintiff-side employment litigators who publish in the field, speak at major conferences, and serve as bar association leaders."
Because he can only research this question by looking at the attorney's online profiles, Moss cannot analyze it systematically. He moves on to other explanations, but in the context of Mata I found this hypothesis compelling and worth digging into.
Moss’s reference point is Engstrom's 2009 law review article, Run-of-the-Mill Justice. And although Moss doesn't cite it, Engstrom wrote a follow-up article in 2011, Sunlight and Settlement Mills. Based on extensive research, including interviews with dozens of former attorneys and employees at "settlement mill" practices, Engstrom provides a more nuanced and sympathetic picture these mills than Moss gives credit for.
Notably, not all personal injury firms are "settlement mills." But, there is no bright line that differentiates a mill from a more ordinary practice. Engstrom offers the following guidelines:
Ten characteristics help to distinguish settlement mills from more conventional personal injury law firms. Four of these factors are necessary (meaning a law firm that does not exhibit each characteristic cannot be considered a settlement mill), and six represent traits that are probative. Settlement mills necessarily (1) are high-volume personal injury practices that (2) engage in aggressive advertising from which they obtain a high proportion of their clients, (3) epitomize "entrepreneurial legal practices," and (4) take few—if any—cases to trial. In addition, settlement mills generally (5) charge tiered contingency fees; (6) do not engage in rigorous case screening and thus primarily represent victims with low-dollar claims; (7) do not prioritize meaningful attorney-client interaction; (8) incentivize settlements via mandatory quotas or by offering negotiators awards or fee-based compensation; (9) resolve cases quickly, usually within two-to-eight months of the accident; and (10) rarely file lawsuits. Each of these factors is considered below.
As one might expect from the above, high-quality legal research is not a defining characteristic. As Engstrom writes about one practice she profiled:
The firm's tenuous link to the formal law is further illustrated by how little traditional "lawyering" was done and the infrequency and absence of trials. As for the former, one lawyer recalled that "no one did any research," while another bemoaned the fact that, while there, he "wasn't practicing law."
Although many attorneys, including Moss, seem to detest settlement mills, Engstrom argues that these firms actually provide a valuable service within the constraints of a rigid system. The goal of that system (known for personal injury claims as "tort" law) is to "make the plaintiff whole," meaning that plaintiff may not only recover for any physical injuries but also for emotional damages such as pain and suffering. To obtain these damages, the plaintiff must show that the defendant is at fault for the injury, typically by showing that the defendant behaved in a negligent manner.
While this sounds theoretically nice, in practice it's very costly to determine fault and evaluate every plaintiff's unique damages. Partly for this reason, we made a large carve-out for workplace injuries in the early 20th century. Today, in every state except Texas, employers are required to participate in a no-fault liability system known worker's compensation: A "grand bargain" wherein employers must pay workers for their injuries at the workplace, regardless of whether the employer is to blame. The flipside is that the injured worker receives payments according to a predetermined "schedule," based on their injury; they are not individually "made whole" as they might be if they had the chance to pursue a tort claim, but the case resolves quickly and the worker is sure to receive some money.
Outside of the workplace, most personal injuries are caused by cars, and at one point there were serious efforts (made by, among others, Michael Dukakis before he ran for President) to implement a national no-fault liability system for auto accidents. This never happened, but Engstrom finds that "settlement mills" have effectively operationalized no-fault liability by establishing "going rates" through repeat negotiations with insurance companies. Engstrom describes the negotiations as follows:
[G]oing rates reflect well-established legal rules and entitlements and bear some relation to past trial verdicts. What is distinctive is that the relationship between going rates and trial verdicts is muted, and going rates are relatively unaffected by the many merit- and non-merit-based factors that would serve to increase or decrease a claim's value in a court of law. In some ways, this comes as no surprise. A victim's unique personal attributes are less likely to affect settlement values when the negotiator (or the attorney fixing the settlement parameters) has never seen or spoken to the client. It is hard for witness credibility to play a prominent role when witnesses are seldom interviewed. And it would be unusual for the negotiation to focus on fine-grained legal considerations, since settlement mill negotiators are frequently non-lawyers. A former Garnett attorney perhaps said it best: "Adjusters don't know the people. We don't know them. So this kind of neck sprain would tend to go for $5,000, $7,500, like that. Generic kinds of injuries, generic kinds of price." In practice, rather than resembling the dominant model of settlement, as Samuel Issacharoff and John Witt have observed, the system more closely resembles a private, under-the-table, ultra-flexible workers' compensation scheme. Indeed, the system is, in the words of Sledge, "a grid." Instead of an individualized and fact-intensive analysis of each case's strengths and weaknesses alongside a careful study of case law and comparable jury verdicts, settlement mill negotiators and insurance claims adjusters assign values to claims with little regard to fault based on agreed-upon formulas, keyed off lost work, type and length of treatment, property damage, and/or medical bills, which in turn relate to the severity of the injury. And, like the grand bargain which undergirds the workers' compensation scheme, as we will see below, participants in the settlement mill system appear to trade the possibility of a significant verdict in favor of greater assurance of some recovery.
Ultimately, despite the decidedly unlegal nature of the work, Engstrom argues that settlement mills provide a meaningful legal service to clients that should not be overlooked. The main problem, in Engstrom's view, is not the representation itself but that mills do not provide adequate informed consent about the scope of representation: Clients don't know that they are actually signing up for an "under-the-table, ultra-flexible workers’ compensation scheme." This deception is unethical, to put it mildly, and it does materially harm clients who may in fact have strong individualized claims. But there is reason to believe that many clients appreciate having the option to hire someone who gets them money quickly, and Engstrom suggests that attorney regulators have failed these clients by not adapting to business realities and providing space for settlement mills to operate ethically.
Returning to the saga of Steven A. Schwartz and his firm, Levidow, Levidow & Oberman, the economics of the "settlement mill" may help explain the firm's lack of attention to its legal research platform choices. This exchange from the court hearing, where Mr. Schwartz apologizes, is perhaps also telling.
[MR. SCHWARTZ: ] Again, I couldn't more deeply apologize to the Court.
Thank you.
THE COURT: Thank you, Mr. Schwartz.
Mr. Schwartz, I realize this is a difficult moment for you, a difficult moment for Mr. LoDuca, and it's easy for something to slip one's mind. But I noticed that among those who you apologized to, one name was missing and that was Roberto Mata, your client.
MR. SCHWARTZ: I was remiss -- I do apologize to my client.
A “Better” Settlement Mill?
As I was preparing this post, news broke of another law firm, Morgan & Morgan, submitting a hallucinated brief in a Wyoming case. David Lat, who covers the legal profession at his blog Original Jurisdiction, wrote about the filing in detail here:
We’re all familiar with the infamous tale of the lawyers who filed a brief full of nonexistent cases—courtesy of ChatGPT, the AI tool that made up aka “hallucinated” the fake citations. In the end, Judge Kevin Castel (S.D.N.Y.) sanctioned the attorneys, to the tune of $5,000—but the national notoriety was surely far worse.
The offending lawyers, Steven Schwartz and Peter LoDuca, worked at a tiny New York law firm by the name of Levidow, Levidow & Oberman. And it seems that their screw-up stemmed in part from resource constraints, which small firms frequently struggle with. As they explained to Judge Castel at the sanctions hearing, at the time their firm did not have access to Westlaw or LexisNexis—which are, as we all know, extremely expensive—and the type of subscription they had to Fastcase did not provide them with full access to federal cases.
But what about lawyers who work for one of the nation’s largest law firms? They shouldn’t have any excuse, right?
[...]
As you can see from the signatures on the offending motion in limine, Taly Goody works at Goody Law Group, a California-based firm that appears to have three lawyers. But Rudwin Ayala and Michael Morgan work at the giant Morgan and Morgan, which describes itself on its website as “America’s largest injury law firm.” According to The American Lawyer, Morgan and Morgan boasts more than 1,000 lawyers, making it the #42 firm in the country based on headcount.
Moral of the story: lawyers at large firms can misuse ChatGPT as well as anyone. And although Morgan and Morgan is a plaintiff’s firm—which might cause snobby attorneys at big defense firms to say, with a touch of hauteur, “Of course it is”—I think it’s only a matter of time before a defense-side, Am Law 100 firm makes a similar misstep in a public filing.
Lat seems to assume here that "resources" are correlated with attorney headcount, but he underplays the difference in business model between Morgan & Morgan and comparably-sized "Am Law 100" (i.e. prestigious corporate law) firms. Although I have not been able to find much good reporting about their business practices, Morgan & Morgan, which started in Florida, has become a national personal-injury phenomenon. What I find interesting is that they have the market cornered on big-ticket personal injury cases like the 2023 train derailment in East Palestine, Ohio while also pursuing small-time cases around the country through aggressive digital advertising tailored to the local market.1 The case with the hallucinated brief appears to fall in the latter category: It's an individual lawsuit against Wal-Mart in Wyoming over a defective hoverboard.
In 2025, I'm not surprised that we're seeing a personal injury firm acquire a national presence. My gut feeling is that the past 10 years of developments in technology (the "Web 2.0"/"big data"/social media/smartphone era) have driven extreme national consolidation and standardization of basic consumer goods and services that were once dominated by regional companies. (e.g., despite the local branding, at this point most seemingly "regional" grocery stores are owned by either Kroger or Albertsons.) Although there are many plausible non-technological causal explanations for these changes, I think a part of the story is one of cheaper, more pervasive, and highly optimized digital advertising.
In his piece on "Painkillers vs. Vitamins," Alex Su, a former attorney turned sales executive who thinks very insightfully about law as a business, instructs lawyers (or sales professionals in legal) to understand the difference between selling a "vitamin" or a "painkiller." Classic vitamin examples are services like estate planning or tax advice, where there is no immediate urgency to buy, while painkillers are services like criminal defense and—although he doesn't mention it—personal injury, where the problem is pressing and urgent. As he writes, clients seek out these attorneys in the same manner as "when you're looking for a plumber when you've got a burst pipe in your house."
Although he is providing marketing advice and not making a global analysis, Su explains very well why "painkiller markets" tend towards consolidation:
Painkiller sellers zone of influence is mostly in the awareness stage. Which means advertising or brand marketing are really the only things you can do to acquire sales. Everything else you do, has a much more limited impact. All of those detailed articles you wrote to demonstrate you’re the best expert end up having a tiny impact. Buyers don’t need the best, they just need something that’s probably good enough to fix the problem.
As a result, this is what ends up happening in painkiller markets:
Incumbents with mindshare among buyers, through years of branding end up with a huge unfair advantage.
For smaller competitors, the cost of brand awareness spikes because it becomes super expensive to advertise.
Just look at how expensive specific keywords for Google ads have gotten. Google ads are great for aggregating buyers who are on the cusp of making a buying decision. Think about all the times you were ready to buy something to fix a problem you had, and needed some quick options. You probably went to Google. And relevant products probably showed up through paid ads at the top of your results.
These types of ads are highly effective, so they end up being really expensive.
In the personal injury context, the "burst pipe" for injured plaintiffs is medical expenses. As Engstrom notes in her 2009 article, settlement mill attorneys pay very little attention to trial verdicts, which in theory set the "market" for personal injury claims. Rather, pricing is set around medical expenses.
At the Sledge firm, "[adjusters] would pay medical bills, drug bills, lost income (if the doctor said you couldn't work), and a thousand dollars a month." At the Dupayne firm, claims typically settled for three-to-four times medical bills. At Jeffers, one attorney used a settlement metric of two-to-three times medical bills. And at Jones, one attorney recalled that, in most instances, he would ask for "three times the meds and hope to get two."
Writing in 2009, Engstrom profiles three regional personal injury law firms (in Georgia, Louisiana, and Texas), which all rose to prominence in the late 1990s through aggressive advertising in their home states. Morgan & Morgan has pushed this business model even further, and although no one has yet profiled them in the same level of detail as Engstrom did 15 years ago, it doesn't take much digging on Reddit to find the M-word appear in reference to Morgan & Morgan.
[Morgan & Morgan] ALONE filed like 20 thousand cases before the new statute kicked in. Created a huge cluster of cases and some were automatically stayed for six to eight months. It’s created headaches for us on defense, but it’s fucked them since they have to actually you know work the case as plaintiffs.
Recently co-counsel in a case told me he has over a 100 files in litigation, and that’s probably the norm for them.
It’s very very eat what you kill. Some of my classmates work there and seem to enjoy it. But they seem to be the minority.
This is all a long way to say they are a mill. A very large and very successful one, but a mill none the less.
Contrary to Lat, I do not think the fact that a "big" firm like Morgan & Morgan filed a hallucinated brief changes my forecast about whether we’ll see one submitted by a corporate law firm. As Lat acknowledges, it's very easy to not submit a hallucinated case, and big law firms have all layers of citation checking process to weed out much finer-grained errors than "this case doesn't exist." These firms compete in a market where success turns on their ability to schmooze (more or less) with highly sophisticated executives and lawyers at the biggest companies in the world. In that environment, there is huge downside risk to becoming the "ChatGPT" firm.
In the world of settlement mills, however, I think the bad press coverage will hardly matter to Morgan & Morgan. The financial incentive to cut down time spent on each case is strong, and they will continue to grow by sucking up mindshare at the expense of small firms like Levidow even if they occasionally cut corners with ChatGPT.
Whither the "Level Playing Field"?
We ended last time with this quote from Moss:
Failing to find and cite available caselaw has become especially inexcusable with online research costs having substantially decreased. From the late 1990s to present, basic Lexis or Westlaw for a solo practitioner has cost as little as $100-$175 a month. Caselaw is also free online, if in less easily searchable form.
His cite for the "late 1990s" price is Jesse Richardson, who at the time was a solo practitioner in Winchester, Virginia (he is now a law professor at West Virginia University). In his 1998 article, How the Sole Practitioner Uses the "Electronic Office" to Maintain a Competitive Law Practice, Richardson was relentlessly optimistic about the potential of technology, and promised
[H]ighlight the areas in which technology can make the small law firm (especially the solo practitioner) more competitive with large law firms and will assert that technology 'levels the playing field' for the solo practitioner and small firm. 'Every lawyer, even a solo practitioner in a small town in New Mexico, can have the same law library as a lawyer in the largest firm in America."
It seems we are a far cry from the level playing field that Jesse Richardson envisioned in 1998, and I think it would be hard to argue that solo practitioners are better off now than they were then.
What’s interesting is that Richardson wasn’t necessarily wrong—access to information has increased. It’s just that "having the same library as the largest law firm in America" is not helpful to the solo practitioner if they cannot justify taking the time to search through it. Richardson simply didn't foresee that more information would power the business models of social media and search engines, thus providing a platform for savvy marketers like Morgan & Morgan to expand.
For solo practitioners in highly competitive "painkiller" markets like personal injury law, this ability to optimize only increases competition, and the winners are those who bring in more clients and reducing the costs associated with each representation. In that context, it makes complete sense why Mr. Schwartz, competing at a small firm in a rapidly consolidating market, would be inclined to seek out a "super search engine" like ChatGPT.
For example, my spouse is from Lexington, Kentucky—home of the Kentucky Wildcats (and their men’s basketball team)—where my in-laws still live. We visit often and a year or so ago, I noticed that Morgan & Morgan had taken out a prominent billboard near campus with a photo of John Morgan standing next to the Wildcats mascot, which was still there last time we checked.