Office hours used to be a staple part of the law school experience. I’m sure you remember what it was like: you’d walk in with questions you had about a reading or with hopes that you can figure out what will be tested on the exam, the professor says something that is as elucidating as it is cryptic, then you walk out a little more confused than you entered. Companies targeting law students for AI boast that the programs can help them organize their notes or summarize cases, but they rarely go so far to say that you’d be better off asking them your questions than going to office hours. Surprisingly, data compiled from law professors assessing how well LLMs answered student questions suggests that’s a selling point these programs should be leaning on. Stanford has coverage:
A groundbreaking study led by Stanford Law School Professor Julian Nyarko reveals that law professors overwhelmingly prefer AI-generated answers to student questions over responses written by their fellow instructors—a finding that could reshape how legal education is delivered.
The study, titled “Law Professors Prefer AI Over Peer Answers,” was conducted with 16 law professors across U.S. law schools and tested whether large language models could serve as effective tutors for contract law courses.
…Participants created 40 representative contracts law questions that students might ask after class or during office hours, wrote their own answers, and then evaluated responses without knowing whether they came from AI or other participating professors. The AI systems performed comparably to the best human instructor in the study. Perhaps most striking: professors flagged AI responses as pedagogically harmful only 3.5% of the time, compared to 12% for peer-written answers.
In times of questionable job security, the threat of AI replacing jobs is usually met with some kind of exceptionalism. “Sure, AI and robots may take the burger flipping jobs, but the gigs requiring higher order thinking and analysis like mine are safe!” Stories about how obviously bad AI is at replacing said jobs are usually tacked on for extra measure. It’s one of the times I wish people were more familiar with the toupee fallacy. The name may be a little dated — toupees get called “Man Units” now — but the idea is that people have a tendency to be overconfident in their assessment of how well things work because the few times they notice them are when they’re done poorly.
AI Is Reshaping Legal Practice—But Tools Aren’t The Real Differentiator.
Explore the mindset, cultural shifts, and training strategies that define the AI‑savvy lawyer, revealing why human judgment, standardized competence, and integrated learning—not technology alone—will shape the future of the profession.
Basically, this:
Is a lot easier to notice than this:
Protégé™ In CourtLink® Explains The Whole Case Faster
Designed to reduce manual docket work by prioritizing what litigators need most: on-demand full docket summarization that explains the whole case to date, followed by on-demand document summaries for filing triage, and AI-powered natural language searching for faster search and retrieval.
An AI-generated response to a student-generated question may have trouble beating the slop allegations if there are obvious tells like a watermark or a lazy “Would you like help with anything else?” left at the bottom, but seeing that the responses were graded to be better than what experts in the field were offering in a double blind study is huge. How far off are we from having an LLM trained on decades of contract lectures teaching 1Ls the four corners rule? We already know that they can answer student questions on par with the best of ’em.
There are, of course, other points of interest. Was the sample size big enough? What sort of soft skill interactions would students miss out on if they swapped out office hours for asking ChatGPT follow-up questions? While I’m sure whatever letters of recommendation ChatGPT could write would make you look like a swell candidate, potential employers will probably prefer the vetting being done by your professor and office hours are a great way for them to know more about you.
While we still need law school professors, students can rely on them a little less. Especially because LLMs are a lot less likely to cold call you into answering your own questions when you need to figure out if there was consideration.
AI Outperforms Law Professors in Stanford Law Study [Stanford]
Chris Williams became a social media manager and assistant editor for Above the Law in June 2021. Prior to joining the staff, he moonlighted as a minor Memelord™ in the Facebook group Law School Memes for Edgy T14s . He endured Missouri long enough to graduate from Washington University in St. Louis School of Law. He is a former boat builder who is learning to swim and is interested in rhetoric, Spinozists and humor. Getting back in to cycling wouldn’t hurt either. You can reach him by email at [email protected] and by Tweet/Bluesky at @WritesForRent.