Levels of Expertise

An attempt to improve the objectivity of skill self-ratings.

Contents

• Discussion
• Scale
• References

Discussion

We are often asked to rate things on a scale, typically 1 to 5 or 1 to 10. Rarely is there an attempt to define what those different numbers mean. From a statistician's point of view, this makes the values useful for the sole purpose of comparing a single individual's ratings against other ratings of that individual. In particular, without a good definition of what the various levels mean, I don't see how there can be any effective communication from one person to another of the meaning of such a rating.

When my doctor asks me to tell him how much something hurts on a scale of 1 to 10, I have no idea what information he expects to get when I say "3" or "7".

I once asked an acquaintance to rate, on a scale of 1 (bad) to 10 (good), a movie he had just seen. He said it was a 9. I was suspicious of this answer, so I asked him how he would rate Star Wars, which I knew to be his all-time favorite movie, on the same 1-to-10 scale. He said 12.

I personally consider it an aspect of innumeracy, but people often try to emphasize something by using numbers that are outside of the valid range. We may chuckle when Nigel says he likes his amp better because it goes to 11, but how often have you heard someone talking in all seriousness about putting in a "110% effort"? What does that actually mean? How would you know if someone were putting in 110% versus 100%? If 110% is a valid number, then presumably so is 120%, so anyone suggesting a mere 110% is clearly not asking for enough effort.

People tend to overestimate how good they are at all sorts of things, including cognitive, social and physical skills. If we all overrate ourselves by the same amount, I suppose that could all cancel out and you could still compare people's ratings - but without knowing a priori what their ratings should be, we don't know how much they might be overrating themselves.

When people consider their own expertise, it is common for those with less expertise to overvalue themselves more than people with more expertise. With more expertise comes more awareness of what one could do better. Einstein said, "As our circle of knowledge expands, so does the circumference of darkness surrounding it." Relative beginners easily fall into the Sophomore Illusion of thinking they know a lot because the circumference of their knowledge is not yet large enough for them to recognize the size of the surrounding darkness.

In 1989, psychologist John Hayes at Carnegie Mellon University identified what is now called the "ten-year rule" (although there are earlier commenters, including Herbert Simon, who was also at CMU). As Leonard Mlodinow says in "The Drunkard's Walk", "Experts often speak of the 'ten-year rule,' meaning that it takes at least a decade of hard work, patience and striving to become highly successful in most endeavors." (links mine) The ten-year rule is related to the idea that it takes about 10,000 hours of practice at something to become an expert; with 5 hours of practice per business day and 200 business days per year, it would take ten years to rack up that many hours. If you find yourself thinking how wonderfully expert you are in something that you have practiced for only a few years, perhaps you should consider the ten-year rule and temper your evaluation.

Given that people are so bad at these ratings, it seems to me that the only way to get any useful information from someone when asking this kind of self-rating question is to have an objective definition of what each level means.

One way to think about a scale is by how many people fall into each level. There are currently 7 billion people in the world, or almost 10 to the 10th power. This conveniently maps to a logarithmic scale from 0 to 10, allowing us to define eleven levels starting with level 0 containing all approximately 10 billion people in the world and with each higher level having one tenth the number of people as the level just below it. If the descriptions of a level are hard to interpret, perhaps the size of that level will help give an indication of whether a person should be rated there.

Years ago, during a job interview, I was asked to rate my level of expertise in various subjects, such as programming languages and development tools. This was not an unusual question, I had been asked this question before and have been asked it since. What was different that time was that the interviewer included a scale with some relatively objective descriptions for determining level of expertise. I rather liked the scale, so although I don't recall the exact definition of his levels, I have tried to reproduce that concept here, using descriptions somewhat similar to those given by that interviewer. Unfortunately, I don't remember who introduced that scale to me, so I am unable to give credit.

There are many reasons one might want a scale of expertise, including rating potential employees or creating a summary of the amount of expertise within a company. The scale I present here is intended to be very general; given its logarithmic nature that can include the entire world population, it is capable of allowing comparison of expertise across everyone in the world. You might think that would make it suboptimal for rating (potential) employee expertise, but I think there are enough levels to make it useful for that purpose.

Scale

The scale below includes the following columns:

• Level: a number for the level, from 0 to 10, with 10 being the highest level of expertise.
• Name: a name for the level. These are taken from a set of expertise level names proposed by the Traveling School of Life. My use of them probably doesn't quite match their intent, but I liked the names and thought the ten words matched my levels pretty well, so I applied them to my levels and added "ignorant" for level 0.
• Description: a brief description of the level. The descriptions are worded as if for a technical tool; for application to other areas or concepts, modify accordingly. Comments referring to companies assume a large company (10,000+ people) with large divisions (1000+ people); being a company-wide guru in a company with 100 people might not get you past level 6.
• Size: the approximate number of people expected to be at that level worldwide. As mentioned above, this is a simple logarithmic scale. The number of people in a level is 10^10-L where L is the level number.
• Practice: the approximate amount of practice that could be required to reach that level of expertise. Putting in that many hours does not guarantee reaching that level, and reaching that level does not necessarily require putting in that many hours. The conversion factors are 1,000 hours per year or 5 hours per day.

All of these different factors are rough estimates, not intended as absolutes but merely as guidelines to help people rank themselves in a way that allows for more meaningful results. I don't have any research to show how well my guesses about Description, Size and Practice correlate; if anyone knows of something along those lines, that would be interesting.

Level	Name	Description	Size	Practice
0	ignorant	I have never heard of it.	10,000,000,000	none
1	interested	I have heard a little about it, but don't know much.	1,000,000,000	1 hour
2	pursuing	I have read an article or two about it and understand the basics of what it is, but nothing in depth.	100,000,000	1 day (5 hours)
3	beginner	I have read an in-depth article, primer, or how-to book, and/or have played with it a bit.	10,000,000	1 week (25 hours)
4	apprentice	I have used it for at least a few months and have successfully completed a small project using it.	1,000,000	3 months (250 hours)
5	intermediate	I have used it for a year or more on a daily or regular basis, and am comfortable using it in moderately complex projects.	100,000	1 year (1,000 hours)
6	advanced	I have been using it for many years, know all of the basic aspects, and am comfortable using it as a key element in complex projects. People in my group come to me with their questions.	10,000	5 years (5,000 hours)
7	accomplished	I am a local expert, with ten or more years of solid experience. People in my division come to me with their questions.	1,000	10 years (10,000 hours)
8	master	I am a company-wide guru with twenty or more years of experience; people from other divisions come to me with their questions.	100	20 years (20,000 hours)
9	grandmaster	I am a recognized international authority on it.	10	30 years (30,000 hours)
10	great-grandmaster	I created it, and am the number 1 expert in the world.	1	50 years (50,000 hours)

References

Other scales of expertise:

• Ted Neward describes four levels in his post of August 16: Apprentice, Journeyman, Master, Adept.
• The Dreyfus model of skill acquisition: Novice, (Advanced) Beginner, Competent, Proficient, Expert.
• Paul Schempp's take on Dreyfus's five levels, with "Capable" rather than "Advanced Beginner".
• The Four Stages of Competence of Thomas Gordon: Unconscious Incompetence, Conscious Incompetence, Conscious Competence, Unconscious Competence. And how they might apply to programming.

Other articles:

• Teach Yourself Programming in Ten Years, by Peter Norvig, 2001.
• From Novice to Expert: Harnessing the Stages of Expertise Development in the Online World, by Douglas A Kranach, in the 2009 ASCUE Proceedings.
• A College Student's Guide to Computers in Education, by Dave Moursund; Chapter 3, "Expertise and Problem Solving", page 25, with a discussion of expertise as related to hours of study and practice.
• An 2006 excerpt from Jonah Lehrer on the importance of practice for Mozart and Tiger Woods.