Thursday, December 8, 2011

Levels of Expertise

An attempt to improve the objectivity of skill self-ratings.

Contents

Discussion

We are often asked to rate things on a scale, typically 1 to 5 or 1 to 10. Rarely is there an attempt to define what those different numbers mean. From a statistician's point of view, this makes the values useful for the sole purpose of comparing a single individual's ratings against other ratings of that individual. In particular, without a good definition of what the various levels mean, I don't see how there can be any effective communication from one person to another of the meaning of such a rating.

When my doctor asks me to tell him how much something hurts on a scale of 1 to 10, I have no idea what information he expects to get when I say "3" or "7".

I once asked an acquaintance to rate, on a scale of 1 (bad) to 10 (good), a movie he had just seen. He said it was a 9. I was suspicious of this answer, so I asked him how he would rate Star Wars, which I knew to be his all-time favorite movie, on the same 1-to-10 scale. He said 12.

I personally consider it an aspect of innumeracy, but people often try to emphasize something by using numbers that are outside of the valid range. We may chuckle when Nigel says he likes his amp better because it goes to 11, but how often have you heard someone talking in all seriousness about putting in a "110% effort"? What does that actually mean? How would you know if someone were putting in 110% versus 100%? If 110% is a valid number, then presumably so is 120%, so anyone suggesting a mere 110% is clearly not asking for enough effort.

People tend to overestimate how good they are at all sorts of things, including cognitive, social and physical skills. If we all overrate ourselves by the same amount, I suppose that could all cancel out and you could still compare people's ratings - but without knowing a priori what their ratings should be, we don't know how much they might be overrating themselves.

When people consider their own expertise, it is common for those with less expertise to overvalue themselves more than people with more expertise. With more expertise comes more awareness of what one could do better. Einstein said, "As our circle of knowledge expands, so does the circumference of darkness surrounding it." Relative beginners easily fall into the Sophomore Illusion of thinking they know a lot because the circumference of their knowledge is not yet large enough for them to recognize the size of the surrounding darkness.

In 1989, psychologist John Hayes at Carnegie Mellon University identified what is now called the "ten-year rule" (although there are earlier commenters, including Herbert Simon, who was also at CMU). As Leonard Mlodinow says in "The Drunkard's Walk", "Experts often speak of the 'ten-year rule,' meaning that it takes at least a decade of hard work, patience and striving to become highly successful in most endeavors." (links mine) The ten-year rule is related to the idea that it takes about 10,000 hours of practice at something to become an expert; with 5 hours of practice per business day and 200 business days per year, it would take ten years to rack up that many hours. If you find yourself thinking how wonderfully expert you are in something that you have practiced for only a few years, perhaps you should consider the ten-year rule and temper your evaluation.

Given that people are so bad at these ratings, it seems to me that the only way to get any useful information from someone when asking this kind of self-rating question is to have an objective definition of what each level means.

One way to think about a scale is by how many people fall into each level. There are currently 7 billion people in the world, or almost 10 to the 10th power. This conveniently maps to a logarithmic scale from 0 to 10, allowing us to define eleven levels starting with level 0 containing all approximately 10 billion people in the world and with each higher level having one tenth the number of people as the level just below it. If the descriptions of a level are hard to interpret, perhaps the size of that level will help give an indication of whether a person should be rated there.

Years ago, during a job interview, I was asked to rate my level of expertise in various subjects, such as programming languages and development tools. This was not an unusual question, I had been asked this question before and have been asked it since. What was different that time was that the interviewer included a scale with some relatively objective descriptions for determining level of expertise. I rather liked the scale, so although I don't recall the exact definition of his levels, I have tried to reproduce that concept here, using descriptions somewhat similar to those given by that interviewer. Unfortunately, I don't remember who introduced that scale to me, so I am unable to give credit.

There are many reasons one might want a scale of expertise, including rating potential employees or creating a summary of the amount of expertise within a company. The scale I present here is intended to be very general; given its logarithmic nature that can include the entire world population, it is capable of allowing comparison of expertise across everyone in the world. You might think that would make it suboptimal for rating (potential) employee expertise, but I think there are enough levels to make it useful for that purpose.

Scale

The scale below includes the following columns:
  • Level: a number for the level, from 0 to 10, with 10 being the highest level of expertise.
  • Name: a name for the level. These are taken from a set of expertise level names proposed by the Traveling School of Life. My use of them probably doesn't quite match their intent, but I liked the names and thought the ten words matched my levels pretty well, so I applied them to my levels and added "ignorant" for level 0.
  • Description: a brief description of the level. The descriptions are worded as if for a technical tool; for application to other areas or concepts, modify accordingly. Comments referring to companies assume a large company (10,000+ people) with large divisions (1000+ people); being a company-wide guru in a company with 100 people might not get you past level 6.
  • Size: the approximate number of people expected to be at that level worldwide. As mentioned above, this is a simple logarithmic scale. The number of people in a level is 1010-L where L is the level number.
  • Practice: the approximate amount of practice that could be required to reach that level of expertise. Putting in that many hours does not guarantee reaching that level, and reaching that level does not necessarily require putting in that many hours. The conversion factors are 1,000 hours per year or 5 hours per day.
All of these different factors are rough estimates, not intended as absolutes but merely as guidelines to help people rank themselves in a way that allows for more meaningful results. I don't have any research to show how well my guesses about Description, Size and Practice correlate; if anyone knows of something along those lines, that would be interesting.

Level Name Description Size Practice
0 ignorant I have never heard of it. 10,000,000,000 none
1 interested I have heard a little about it, but don't know much. 1,000,000,000 1 hour
2 pursuing I have read an article or two about it and understand the basics of what it is, but nothing in depth. 100,000,000 1 day (5 hours)
3 beginner I have read an in-depth article, primer, or how-to book, and/or have played with it a bit. 10,000,000 1 week (25 hours)
4 apprentice I have used it for at least a few months and have successfully completed a small project using it. 1,000,000 3 months (250 hours)
5 intermediate I have used it for a year or more on a daily or regular basis, and am comfortable using it in moderately complex projects. 100,000 1 year (1,000 hours)
6 advanced I have been using it for many years, know all of the basic aspects, and am comfortable using it as a key element in complex projects. People in my group come to me with their questions. 10,000 5 years (5,000 hours)
7 accomplished I am a local expert, with ten or more years of solid experience. People in my division come to me with their questions. 1,000 10 years (10,000 hours)
8 master I am a company-wide guru with twenty or more years of experience; people from other divisions come to me with their questions. 100 20 years (20,000 hours)
9 grandmaster I am a recognized international authority on it. 10 30 years (30,000 hours)
10 great-grandmaster I created it, and am the number 1 expert in the world. 1 50 years (50,000 hours)

References

Other scales of expertise: Other articles:

11 comments:

Anonymous said...

Thanks for this post! It was critical in helping me put together a rationale for calling someone a "pro." See http://proswrite.com/2012/09/13/who-deserves-to-be-called-a-pro/ if you're interested.

Jeanne Sutherland said...

Hi Dr. McBeath,
I like your post. I would like to email you and ask you additional questions. First one is, do you play golf? I would like to use something like this for a self-assessment. The problem is as you outlined, the better the golfer, the more knowledge they have of their flaws. I once had an average player tell me he was a better wedge player than Tiger. He actually believed it, too.

Unknown said...

Many thanks for sharing your insight with the world. I was in the midst of putting together a skills matrix and found my scale to be lacking. I found your article to be extremely insightful and helpful to my task, I'm very grateful that you have shared your work and would like to continue to follow your work.

Laura Brown said...

Thank you for sharing this. I am using for self-assessment of expertise among a group of physicians that lead quality improvement at a large academic medical center.

Unknown said...

Thank you! Working on a qualitative dissertation and should probably follow your blog.

Unknown said...

I like your idea of trying to create a scale and also like how you use orders of magnitude to size populations. Your notions are good ones and you are off to a very good start. As you say, correlations are a different matter. So are tying different scales together. While your multiplier (divisor) for Population is 10x (x/10), you have done some real sliding around with your practice scales, which go 5x,5x,10x,4x,5x,2x,2x,2x,2,5x.

I found this while looking at something else and wanted to offer a comment more than a criticism. It seems like a very interesting idea and something fun to work on. My guess would be that a more accurate model would not use a scale with 11 increments, but something closer to 5 or 6.

On the low end, four levels below apprentice seems like a lot of distinction for not much difference in knowledge or ability. On the high end, three levels of Master seems a lot. A category with only one person in it doesn't have much utility.

Jim McBeath said...

Unknown: Yes, there are some arbitrary choices in my scale. Using an equally-spaced logarithmic scale is a simple choice, but I don't have data on the distribution of how much practice people have.

The usefulness of the ends of the scale depends on what you are trying to do. The low end may not be interesting for hiring, but could be useful to specify how familiar a large population is with a particular technology.

Regarding categories with just one person, it seems to me that many people are very keen on knowing who the best in the world is for any given challenging skill.

Thanks for your comments, I'm happy to have people contribute their relevant ideas and constructive criticism.

JackBlanchart said...

Hi Jim
I like very much your expertise rating scale. Simple, readable, ptacticable.
I'm a retired engineer still working on low-cost, diy instrumentation and I'm trying to explain whick skills are need to perform the needed tasks (development and manufacturing of low-cost scientific instrumentation).
Are there any publications with your expertise scale ?
I'm writing an article about "diy instrumentation". How can I reference this article ?
Thanks by advance
Jacques
Sorry for my poor french-english dialect.

Jim McBeath said...

Jacques: I am not aware of any publications referencing my scale. If you would like to reference this article, you can use the URL, https://jim-mcbeath.blogspot.com/2011/12/levels-of-expertise.html

JackBlanchart said...

Many thanks, Jim. I'll do that way.
It will be a kind of "technical note" that I will post on ResearchGate. I'll let you know when it will be posted (within one month, probably).

Unknown said...

Hi Jim,

I like your scale and am planning to use it to assess levels of expertise regarding various aspects of information related to the COVID-19 pandemic.

Great work, thanks for sharing.

Dan