Assessing ability

I’ve recently had a few conversations with recruitment agencies, where they’ve asked me to rate my IT skills from 1-5. However, I’m reluctant to answer, because I think the concept is flawed.

Firstly, what’s being measured? Is it the amount I know, or is it my relative ranking?

For instance, I consider myself to be fluent in English (since it’s my native language), and I like to think that I have a fairly wide vocabulary. In theory, it should be possible to quantify my vocabulary, e.g. by saying that I know 40% of the words in the dictionary. However, I can’t tell you how many words I actually know; I also don’t know how many words are in the dictionary without looking it up. So, I can’t calculate that percentage.

Putting it another way, if I could quantify my vocabulary, 40% would give me a score of 2 out of 5. However, if that put me in the top 20% of the population, should my score then be 5 out of 5? Or should most people be graded 1, with nobody above a 3?

Taking a technical equivalent, look at the Java class library. According to Wikipedia, this “has grown from a few hundred classes in JDK 1.0 to over three thousand in J2SE 5”. So, in the early days, it was feasible for a Java developer to have a poster on their wall showing the entire library, and possibly be familiar with all the classes. Nowadays, I doubt that anyone knows them all, even the people who work for Oracle. So, how should you rate your Java skills?

If you make it a relative scale instead, that also has problems. How can I compare myself to everyone else, bearing in mind that I’ve only met a tiny fraction of IT workers? Arguably, this is something that the agency should do, since they spend all day looking at CVs so they’ll have a better idea of how I compare to other candidates.

Some people argue for a qualitative approach (aka “make something up”). However, that has drawbacks. For instance, Ola Svenson published a paper in 1981: “Are We All Less Risky and More Skillful than our Fellow Drivers?” According to that, “In the US sample 93% believed themselves to be more skillful drivers than the median driver and 69% of the Swedish drivers shared this belief in relation to their comparison group.”

To be fair to the participants, this isn’t simply a case of being boastful or modest. As another example, think about Maths. If you’d asked me to rate myself when I was at primary school, I would have given myself 5 out of 5. I could count up to any number, add and subtract, I knew my times tables, and I’d done long division; I even knew about negative numbers! However, I didn’t come across trigonometry or calculus until secondary school. It’s not simply that I didn’t know those skills at primary school: I didn’t know that I didn’t know them.

At university, one of my lecturers described his balloon theory of knowledge. The air inside the balloon is what you know, the air outside is what you don’t know, and the surface of the balloon is your awareness of the knowledge gap. So, when you start out, the balloon is very small. There’s not much air inside it (because you don’t know much), but the surface is also small so it looks as if you’ve accumulated roughly half of the world’s total knowledge. The more you learn, the bigger the balloon gets, and the more you come to realise how much you don’t know yet. This is related to the Dunning-Kruger effect: if someone has never heard of IPv6, they won’t be aware of this gap in their knowledge, and they might think that they’re an expert on network protocols.

So, what’s the solution? I think it’s better to look at qualifications, where people have been assessed against a specific syllabus by a neutral (objective) third party. For instance:

  • If I have GCSE Maths, you can assume that I know how to calculate the area of a circle.
  • If I have CompTIA Network+, you can assume that I know how to configure a DHCP server.

Ideally, a recruitment agency should be familiar with all the common certifications, so that they can decide which ones are relevant to a particular role.

There might be cases where people have the qualification without the skill, but then they’d presumably lie on the “1-5” question so you’re no worse off. There might be other cases where people have the skill without the qualification (IT exams aren’t cheap!), but in that case it would be better to ask specific questions, e.g. “Have you ever configured clustered servers?” Either way, I return to what I said at the start: asking people to rate themselves from 1-5 isn’t helpful, so please don’t do it.

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.