Back In The Race: 5 Questions To Ask Before Taking A Lawyer Survey Seriously
Not all surveys are created equal, as columnist Shannon Achimalbe explains.
Sometimes we wonder how we are doing compared to our colleagues and everyone else. When I have a good year financially, I wonder if my colleagues and competitors have experienced the same good fortune. If everyone else did just as well, then I must admit that my accomplishments feel less special. But since most of us have only a limited number of colleagues and try to avoid talking about personal matters, we turn to the next best thing: surveys.
I see surveys about lawyers on a regular basis. Most of them involve how much money they make. And some involve things that are almost impossible to quantify – such as “happiness levels.”
But I am generally skeptical of surveys about solos and small firms. The main reason for this is that solos and small firms have many different business models based on specialties, regions, the managing attorneys, and the types of clients. And most of the time, the results of these surveys are nowhere near what I have experienced. So when I read a survey, I secretly ask myself five questions before deciding whether to take it seriously.
Generative AI In Legal Work — What’s Fact And What’s Fiction?
Who conducted the survey? One of the first things I check is who conducted the survey. This is because some people and organizations have an agenda, so any survey results from them should be questioned or even disregarded. In politics, most people are familiar with Fox News, MSNBC, and CNN (Clinton News Network), and their respective preferences.
I tend to trust surveys conducted by the government and bar associations because they are simply gathering information to set policy. Bar associations generally conduct surveys in order to help their members.
Why are they conducting this survey? Most of the time, authors conduct surveys in order to gather information for group members, such as salaries, management techniques, what practice management software is being used, etc. Or they can be used to see if a problem exists, how big the problem is, and how it has been dealt with. I see this a lot when it comes to depression.
But survey results may be used for questionable purposes. I’m a bit wary of surveys done by businesses. This is because most of the time, they promote the results to try to sell you something.
Sponsored
The Business Case For AI At Your Law Firm
Navigating Financial Success by Avoiding Common Pitfalls and Maximizing Firm Performance
Generative AI In Legal Work — What’s Fact And What’s Fiction?
Navigating Financial Success by Avoiding Common Pitfalls and Maximizing Firm Performance
What was the methodology? Did you ever hear about the survey that concludes that there are two lawyers for every American? Now I get that lawyers are bad with numbers. But Occam’s Razor should apply. A survey’s methodology should be transparent, easy to understand, and easy to duplicate. So easy that even a lawyer can understand it.
One thing to look at is the sample size and demographics. I cannot take seriously a survey that simply asks the World Wide Web, “Which one of you makes over $100,000 per year?” First of all, $100,000 per year (or even $250,000 per year) in Manhattan does not get you as much as $100,000 in the Midwest. I will also need to know a lot more information to determine whether the survey is relevant to me. For example, the number of years the respondents have been in practice, and his or her specialty (if any).
Another thing to look at is how the respondents were selected. If they were invited to participate in an online survey, it is more likely than not that only those who are successful will respond. Also, without means of verification, the online survey can be tainted by anonymous trolls, which leads me to the next issue.
Are the respondents being honest? This is the biggest issue when it comes to surveys. A survey works only if people provide honest answers. The problem is that some people are reluctant to discuss and disclose sensitive matters such as their finances or their mental state (or lack of either). While most people try not blatantly lie, they are likely to finesse or exaggerate their answers, depending on who is asking the questions and what kind of questions they are asking. For example, a business owner may report a certain income to the IRS while presenting a much higher number to a loan officer.
One way to incentivize honest disclosure is to give respondents anonymity. But that can be a double-edged sword. Some people may be willing to come forward and provide embarrassingly frank answers anonymously. But anonymous people may still not tell the truth. Also, some trolls may deliberately try to sabotage the poll. Why would they do this? For the lulz.
Sponsored
Is The Future Of Law Distributed? Lessons From The Tech Adoption Curve
Legal AI: 3 Steps Law Firms Should Take Now
Another way to get more honest answers is to ask very specific questions. Leave nothing open to interpretation and puffing. But this can make the survey cumbersome and time-consuming.
How are the survey results interpreted? After the authors present the survey results, they try to explain what happened and offer their conclusions. But I noticed that many times, they do not disclose their limitations and if they do, they do not adequately address them. Always ask yourself, are the authors trying to sell me something? Or do they want me to vote or think a certain way?
No survey is perfect. They can be useful for helping solos and small firms set future goals and make changes as needed. But if the results of a survey do not coincide with your reality, it does not necessarily mean you are doing something wrong or that the results are correct.
Shannon Achimalbe was a former solo practitioner for five years before deciding to sell out and get back on the corporate ladder. Shannon can be reached by email at sachimalbe@excite.com and via Twitter: @ShanonAchimalbe.
