Metrics and Incentives
November 17, 2006
There’s a great discussion going on in the security blogosphere about the importance of metrics – Mogull, Amrit and Rothman are talking about them.
I’m going to fall on the same side as Amrit on this one, as metrics have always been one of my favorite things to play with. The problem, as Mike points out is that the wrong metric is always worse than no metrics at all. Anybody who has known me a while has heard me rant about the over-simplistic metrics that we use in technology, and the importance of finding metrics that can really help solve problems. And, as is evident in the articles that started this whole discussion over at Joel on Software (here and here), metrics have been given a really bad name in technology. From the articles:
“Software organizations tend to reward programmers who (a) write lots of code and (b) fix lots of bugs. The best way to get ahead in an organization like this is to check in lots of buggy code and fix it all, rather than taking the extra time to get it right in the first place.”
And:
“The whole fraud is only possible because performance metrics in knowledge organizations are completely trivial to game.”
Joel seems to think that the reason that this is the case is because there’s some inherent flaw in what you’re trying to measure. I would argue that the problem isn’t the measurement, but the incentives that the metric creates that are the real problem. Each thing you measure and reward (or discourage) creates a behavioral incentive – in the case above, measuring only the creation of volume of code (without quality) and remediation of bad code (without proactively avoiding bad code in the first place) creates a situation where the metric is, as Joel points out, “trivial to game”.
This is why metric development has to end up as an iterative process. What most companies do when they realize that they’ve incentivised bad behavior is the same thing that Joel suggests: dump the metrics altogether. Unfortunately, that’s actually the wrong approach – the point of creating metrics is to measure and incentivize, and proving that people are gaming the system proves something quite simple:
Measuring things works to modify behavior. If it didn’t, people wouldn’t be gaming the system.
The problem at this point is to ask yourself the hard question: what is it that I want to incent my people to do? Where are we now? And how can we alter the metrics to incentivize the correct behavior? How can we correct the metric?
Let’s take the rather simple example Joel gave us: what if we took the metrics a) and b) above, and added c) create few bugs? And created a composite metric something like the following:
Performance = ( amount of code + bugs fixed ) / (bugs created)
How would you game that? Let’s iterate through this process in the comments – let me know how you’d game it, and I’ll modify the metric (in case it’s not obvious, I’ve done this iteration before)
![[del.icio.us]](http://images.del.icio.us/static/img/delicious.small.gif){kind=small}
The Sham of Job Interviews
November 17, 2006
One of my favorite things about managing a team was interviewing candidates. Anyone who ever interviewed for a job for any of my teams got to know my penchant for asking strange questions and doing strange things – the main purpose was always to overcome the fact that most interviews are a complete and total sham.
For most interviews, the decision on whether to make the hire or not is made within the first 30 seconds of the interview, and has absolutely nothing to do with any of the rational parts of the decision-making process. While we may think that we’re completely rational, what research has shown is that the interview process is mostly a matter of the interviewer having a first impression, and then spending however long the interview lasts gathering evidence to back up that first impression.
I remember a particular story from my past where this became all too clear to me. Early in my career, I was on a team that was interviewing for a trainer who would be responsible for all customer-facing training. We saw a few mediocre candidates, and then we interviewed a relatively attractive woman. The woman had no training experience, and was absolutely terrible in the group interview. When the interview ended, we all re-grouped around my then boss’s office while he saw her out and started discussing. It was clear that we all thought she wasn’t a fit – we all thought she was too much trying to rely on her looks and had very few skills.
That was when my boss returned, and started extolling her virtues. I argued with him for a few minutes, until he uttered the line that those who were on the team make jokes about to this day: “She will be our trainer.” Suffice it to say, she didn’t exactly set the world on fire with her performance after she was hired. And, after making some pretty big mistakes and generally being a disruption at times, she was eventually let go.
What was most interesting to me that day was that it was as though my boss (who was normally quite sharp and a decent senior manager) was in a completely different interview than the rest of us. Gladwell reported on this research in the New Yorker, and again in his book Blink
“She took fifteen seconds of videotape showing the applicant as he or she knocks on the door, comes in, shakes the hand of the interviewer, sits down, and the interviewer welcomes the person,” Bernieri explained. Then… Prickett got a series of strangers to rate the applicants based on the handshake clip, using the same criteria that the interviewers had used. Once more, against all expectations, the ratings were very similar to those of the interviewers. “On nine out of the eleven traits the applicants were being judged on, the observers significantly predicted the outcome of the interview,” Bernieri says. “The strength of the correlations was extraordinary.”
The difficulty with interviews is that we end up having our unconscious mind hijack the interview process – we end up in rapport with the person, and our unconscious tells us that we should like them and that they’re worth hiring because of it. Or, we don’t develop rapport (or even actively dislike the person) and choose not to hire them because of that.
Almost everyone who has ever interviewed for a job can think of a time when they were completely qualified and the opportunity was absolutely perfect, but they didn’t get the job. And almost everyone who has ever interviewed someone has been hijacked by their unconscious this way.
The only way around this is to set up a process that ensures that your conscious and unconscious mind can work together on the process of interviewing.