The Story of Moneyball
2002 was a tough year for the Oakland Athletic’s. General Manager Billy Beane was frustrated by losing three key players—and with his general inability to compete with larger market teams with the smallest budget in the league for player salaries that year.
He enlisted Paul DePodesta, a statistician from Harvard, to find players that were undervalued by the market.
Together, they looked at statistics of in-game activity by players (aka sabermetrics) to reveal that a player’s on-base percentage was a better indicator of winning baseball games than batting average or home runs. Players who scored high here were undervalued by the market.
Beane began recruiting these overlooked players to deep skepticism from intuition-driven scouts.
The A’s started winning against bigger budget teams, becoming the first American League team in over 100 years to win 20 consecutive games. Today, sabermetricians are routinely hired by every major league front office.
Can Law Firms Gain a Similar Statistical Edge?
Like sabermetricians, Industrial and Organizational (I/O) Psychologists study human behavior in organizations. They use the five factor model of personality characteristics aka “the big five” to assess an individual’s personality traits. Studies show that strengths of certain personality traits can predict professional success. Generally, candidates are likely to be top performers across industries and jobs when scoring high on these traits:
- Conscientiousness—a person’s ability to be organized, disciplined, achievement-oriented, and dependable.
- Emotional resilience—a person’s ability to remain calm under pressure and adapt to stressful situations and people.
Candidates who aren’t conscientiousness rock stars can still succeed and there will be variations on what traits predict success across professions and even within subsets of the same profession. Various roles in the legal industry have their own unique combinations of traits that predict success—traits that make for a successful litigator, may not make a successful corporate transactional attorney.
Measuring Attorney Soft Skills
So how can you measure a legal professional’s fit for a role?
- Analysis of the Knowledge, Skills and Abilities (KSAs): Identify what requirements are essential to the job to know what to look for in a candidate.
- Candidate Workstyle Assessments[1]: Candidate-completed questionnaire assessing traits linked to important competencies for the role.
- Structured Behavioral Interviews[2] : Interview questions tailored to tap into how candidates have dealt with specific situations in their work experience that relate to competencies required for the job. All candidates are asked the same questions in order under similar conditions to limit interviewer bias and increase predictability of the assessment.
- Feedback from References and Peer Reviews: This should also be structured to be uniform and to assess candidate’s competencies in relation to job KSAs.
- Triangulation of Data: The above tools are more powerful, accurate and less likely to be “gamed” when used together.
- Large Data Set: Having a substantial pool of candidates and data to compare against each other is crucial to making accurate comparisons.
Playing Moneyball
An exact prediction of future human behavior is impossible; these methods have no absolutes or guarantees. But like Moneyball, we can see correlations, and offer better chances of finding the best talent.
And while machine learning recruiting technologies have been recently shown to perpetuate patterns of bias as Amazon recently discovered, psychometric methods can reduce bias with algorithms based on core competencies for a job versus a machine making assumptions based on past biased patterns.
Still, psychometric methods should be put through rigorous statistical analyses to ensure that all groups (e.g. race and genders) are treated equally throughout the recruiting process. Each assessment question and sets of questions should be compared to make sure that no group has a significant advantage over another via comparing outcomes of both scale scores and real word performance to each pairing of race and gender collected in the recruitment process.
At Hire an Esquire we’ve used psychometric assessment methods that have been rigorously analyzed—and have seen our candidate incident rate fall from 4% to 1%, the rate of clients requesting the same temporary candidate again jumping 67% and our temp to perm conversion rate tripling across the board. At the same time, women and minority hires were increased by 15% versus traditional recruiting methods.
As we approach a world where careers are becoming more transient and on-demand, psychometrics can provide a key competitive edge in practice and recruiting while also making operations smoother via higher productivity and lower turnover.
Will your practice play moneyball?
[1] Barrick, Murray R., and Michael K. Mount. “The big five personality dimensions and job performance: a meta‐analysis.” Personnel psychology 44.1 (1991): 1-26.
[2] Campion, Michael A., David K. Palmer, and James E. Campion. “A review of structure in the selection interview.” Personnel Psychology 50.3 (1997): 655-702.
References and further reading:
Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: a meta‐analysis. Personnel psychology, 44(1), 1-26.
Judge, T. A., & Zapata, C. P. (2015). The person–situation debate revisited: Effect of situation strength and trait activation on the validity of the Big Five personality traits in predicting job performance. Academy of Management Journal, 58(4), 1149-1179.
Shultz, M. M., & Zedeck, S. (2012). Admission to law school: New measures. Educational Psychologist, 47(1), 51-65.
Eric Fox is the Director of People Analytics at Hire an Esquire. He is an Industrial/Organizational psychology practitioner with expertise in employee selection, psychometrics, and data analysis. He completed his graduate degree at Baruch College under Harold Goldstein and Charles Scherbaum- the creators of the NFL player assessment or as its known as “Moneyball for football.”
