NCBE Chief On Possibility Of Serious Evaluation Of Online Bar Debacle: 'I Think So... I Don't Know.'
The test happened... and that's all that matters.
Credit where credit’s due. Back when a botched online exam was only a glimmer in the eyes of bar examiners around the country, National Conference of Bar Examiners chief Judith Gundersen quipped of the impending tire fire, “Why don’t you interview me Oct. 6?” Only a very tiny sliver of the legal landscape was holding onto that receipt and if she just ghosted on this pledge like she was bar exam tech support there would be few if any consequences. But she actually went through with it!
Gundersen’s follow-up interview with Karen Sloan dropped yesterday afternoon. Rather than forge a comprehensive take on her answers, let’s just follow along as I read it in real-time. Join me on this journey:
You said the online exam was a success. How do you define a success? What factors are you looking at? If you think about why we did this in the first place, our goal was to give candidates, or examinees, the ability to safely take the bar and have the opportunity to become licensed. And to give jurisdictions that didn’t feel they could give an in-person bar exam the option to do that. By that account, it’s certainly is a success. We did that.
Early Adopters Of Legal AI Gaining Competitive Edge In Marketplace
Indeed. If you set expectations at “it happened,” then it was certainly a success.
For the true fans out there, check out how she describes the online exam like it was her idea instead of a proposal from individual states that the NCBE tried to vigorously undermine and only begrudgingly got on board. They commissioned material last week to use in their next lobbying blitz against online exams.
I’m hearing skepticism from examinees about these early, good numbers that states have put out. What explains this discrepancy between how examinees seem to feel the test went and then these early numbers? We’ll see. The proof will be when the results come out, when examinees’ files are uploaded and their scores are being graded. I’m not saying there weren’t tech issues—I understand that there were—but most of the tech calls were resolved quickly.
“Quickly” is in the eye of the beholder. It does seem as though most tech calls took less than an hour, but not much less than an hour. The balance of the reports seems to run from half and hour to 90 minutes. And these figures don’t tell the whole story as many people claim tech support often required them to engage in multiple shorter 10-15 minute calls that are assuredly not being added up in this cheery debrief. This also counts just the people who got connected even though many of the reports describe calling and giving up after 30-40 minutes.
Sponsored
Legal AI: 3 Steps Law Firms Should Take Now
The Business Case For AI At Your Law Firm
Is The Future Of Law Distributed? Lessons From The Tech Adoption Curve
Navigating Financial Success by Avoiding Common Pitfalls and Maximizing Firm Performance
Will there be some analysis of the tech issues people had—even if those problems didn’t prevent them from actually completing the exam and uploading answers? Is there a way to gauge that? I think so.
She thinks so. There is not, at present, a plan to conduct a post mortem on the tech issues. This answer can’t be stressed enough — the people charged with ensuring that professional licensing is handled effectively isn’t sure if they’re going to make sure the exam they just gave worked. And lest you think that’s just Gundersen being colloquial:
Are individual jurisdictions going to analyze their own data, or will that happen at the NCBE level? We’re meeting with all jurisdictions, so it will probably be a combination of both—between ExamSoft, the jurisdictions and the NCBE as well.
No, there really isn’t a strategy here. It’s all by the seat of their pants. And it all goes back to the very first answer: the definition of success is that the test happened, not whether or not it was the best or even an adequate exam. They don’t care if data was compromised, computers broken, or passwords hacked. The exam… happened. The bar exam industrial complex lives on. Huzzah!
Sloan keeps trying, in a very steady, understated manner, to get Gundersen to acknowledge the gravity of what just transpired. She asked about applicants who dropped out before the exam even happened or who withdrew mid-exam — even citing “demographics” in an effort to subtly call attention to the widespread reports that people of color withdrew in disproportionate numbers because the facial recognition algorithm wouldn’t recognize them. Gundersen said any evaluation of these problems would be left up to individual jurisdictions.
Sponsored
Navigating Financial Success by Avoiding Common Pitfalls and Maximizing Firm Performance
Early Adopters Of Legal AI Gaining Competitive Edge In Marketplace
Do you have a sense that that’s an analysis jurisdictions plan to do? I don’t know.
Just really galling.
I have to ask you about people urinating on themselves during the exam. I heard from more than one person—and these are credible accounts—that they peed in their seats during a session because the rules prohibited leaving the view of their cameras. What do you make of that? The first I heard of this was in the story you wrote yesterday. For me to comment on it, I think would be speculative. I don’t know the facts of the individual cases. I can assure you that we will be looking at what jurisdictions learn about any issues like this that may have arisen. There were breaks every 90 minutes. For people who needed accommodations, there were shorter sessions.
A couple of takeaways. First, I guess Gundersen isn’t reading Above the Law. Second, if true, this amounts to a disqualifying failure in planning from national leadership. “How will this system deal with bathroom breaks?” should have come up at some point in this process. It may not be a day 1 concern, but over the course of the months that went into planning this, a responsible organization would have flagged it. They already got to see it happen in the UK!
And let’s talk accommodations. Gundersen waves it around like a silver bullet, but it’s existence in theory doesn’t erase the reality. States place onerous restrictions on accommodations and people with confirmed medical conditions get rejected for accommodations all the time. One state asked applicants to stop being diabetic for the sake of the test! Accommodations aren’t a solution in practice. Moreover, this isn’t even the sort of thing accommodations are intended to address. Sometimes people have to go to the bathroom. We also had someone withdraw because she got her period during the test — that’s not a situation that accommodations can fix either.
For her part, Sloan is clearly taken aback by this answer:
But for you as a bar examiner, what was your reaction when you read that? Like I said, it was the first I had heard about it, so I was surprised. We want people to have an optimal testing experience.
Remember, that back in April the NCBE thought it was optimal that all states keep holding in-person exams so it seems “optimal” may be on a sliding scale based on whatever protects the NCBE’s interests at the moment.
Another issue that a lot of people have complained about is the MBE [multiple-choice] questions. Many people said they felt this week’s questions weren’t comparable to those in their study materials or previous exams. “Poorly written” was a phrase I heard quite often. That is 100% false. The questions we used on this exam were fully developed, fully vetted questions that were prepared by our drafting committee members—as always—and that were administered in the past under standard testing conditions as scored items.
“We can’t have written bad questions because we wrote them.”
In fairness, the questions on this exam are always shitty. It flows from the fact that the bar exam is not, despite its PR copy, a test of minimum competence, but rather an intentionally confusing test designed to guarantee a certain percentage of failures. That these questions have appeared on other tests isn’t dispositive — the test almost always breaks down with about 40 percent straightforward material, 40 percent more difficult but gettable questions and then 20 percent complete left-field curveballs where multiple answers are technically correct and examinees are forced to choose more or less at random between them. Applicants this week reported a test skewed wildly toward the last category and the testimonials of those who have already passed bar exams and were merely sitting in a new jurisdiction lend those claims a lot of credence.
Having talked to quite a few examinees this week, my sense is that the process leading up the bar and the online test itself has really left a bad taste in the mouths of many. They are really unhappy with how this unfolded. What would you like to say to them? We’re certainly sympathetic to anyone who experienced an issue during the remote exam. With respect to the entire bar exam cycle, there were a lot of people working night and day to try to make licensing options available. I understand it’s a very hard time for applicants—to have spent their second semester online, and to be confronted with the pandemic and the uncertainty. I completely understand the anxiety and the resentment and anger about it. But a lot of people were working hard to give them opportunities to get a license.
No, they weren’t.
These folks weren’t “working hard to give [applicants] opportunities to get a license” or they would have advocated granting licenses the way Gundersen herself got her license… without an exam. Or they may have advocated for open book exams that more accurately track the actual practice of law like they had in Indiana, or pushed for an apprenticeship track as an alternative, or designed an enhanced CLE program with iterative quizzing for this class.
The NCBE and the state examiners behind the online tests weren’t working hard to give applicants opportunities to get a license, they were working hard to ensure the bar exam is maintained in its current form. They were working hard to protect a monopoly. They were indeed working hard to keep a lid on licensing opportunities.
And so far all that hard work has paid off… for them.
Earlier: NCBE President Gives Trainwreck Of An Interview
Like COVID-19, Online Bar Exam Is A Disaster And Was Entirely Preventable
The Online Bar Exam Amounted To Two Days Of Cruel Vindictiveness
Joe Patrice is a senior editor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, questions, or comments. Follow him on Twitter if you’re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search.
