Notes: Yonathan Arbel and David Hoffman are legal scholars who have done significant empirical work. In a sense, this article is two legal empiricists taking a new toy out for a test drive to see what it is capable of. That new toy comes from the other major branch of AI, which uses statistical methods to inductively learn patterns in data.
Questions:
Compare and contrast generative interpretation with the formal methods discussed in the previous class.
In Snell v. United States Specialty Insurance Co., 102 F.4th 1208 (4th Cir. 2024), a child was injured on an in-ground trampoline and her family sued the landscaping contractor who installed it. The contractor’s insurance policy covered it from liability that “arises from” “operations” in which “[the] Insured performs landscaping.” The contractor and the insured disputed whether installing a trampoline was “landscaping.” In a concurrence, Judge Kevin Newsom suggested that the issue might be a good one for LLMs. Having read Arbel and Hoffman, how would you actually test this out? When you get an answer from the LLM, do you need to confirm that it is correct? If so, how?
Arbel and Hoffman focus on contract interpretation. Could the same approach work for other kinds of legal interpretation and reasoning, such as statutory interpretation, constitutional interpretation, or applying caselaw?
What kinds of empirical validation would LLM interpretation need to be considered reliable in general? In a specific case?
How would you feel about having a lawsuit in which you are party resolved by using LLM interpretation? Does it matter what kind of case it is and how high the stakes are?
What is similar about LLM interpretation to human interpretation? What is different?
Additional Resources:
Judge Newsom’s entire concurrence in Snell is well worth reading. So is his follow-up in United States v. Deleon, 116 F.4th 1260 (4th Cir. 2024). Another interesting case in which the majority and dissent debated the use of LLMs is Ross v. United States, No. 23-CM-1067 (D.C. Ct. App. Feb. 20, 2025).
Christoph Engel, Experimental Comparative Law 2.0? Large Language Models as a Novel Empirical Tool (draft 2024), finds that “merely giving the LLM the otherwise identical vignette in different languages leads to strongly different results.” (Is this a disturbing finding that LLMs are influenced by arbitrary and irrelevant factors, or an encouraging finding that LLMs have learned the different legal contexts in which speakers of different languages typically work?)