Metrics and Incentives

November 17, 2006

There’s a great discussion going on in the security blogosphere about the importance of metrics – Mogull, Amrit and Rothman are talking about them.

I’m going to fall on the same side as Amrit on this one, as metrics have always been one of my favorite things to play with. The problem, as Mike points out is that the wrong metric is always worse than no metrics at all. Anybody who has known me a while has heard me rant about the over-simplistic metrics that we use in technology, and the importance of finding metrics that can really help solve problems. And, as is evident in the articles that started this whole discussion over at Joel on Software (here and here), metrics have been given a really bad name in technology. From the articles:

Software organizations tend to reward programmers who (a) write lots of code and (b) fix lots of bugs. The best way to get ahead in an organization like this is to check in lots of buggy code and fix it all, rather than taking the extra time to get it right in the first place.

And:

The whole fraud is only possible because performance metrics in knowledge organizations are completely trivial to game.

Joel seems to think that the reason that this is the case is because there’s some inherent flaw in what you’re trying to measure. I would argue that the problem isn’t the measurement, but the incentives that the metric creates that are the real problem. Each thing you measure and reward (or discourage) creates a behavioral incentive – in the case above, measuring only the creation of volume of code (without quality) and remediation of bad code (without proactively avoiding bad code in the first place) creates a situation where the metric is, as Joel points out, “trivial to game”.

This is why metric development has to end up as an iterative process. What most companies do when they realize that they’ve incentivised bad behavior is the same thing that Joel suggests: dump the metrics altogether. Unfortunately, that’s actually the wrong approach – the point of creating metrics is to measure and incentivize, and proving that people are gaming the system proves something quite simple:

Measuring things works to modify behavior. If it didn’t, people wouldn’t be gaming the system.

The problem at this point is to ask yourself the hard question: what is it that I want to incent my people to do? Where are we now? And how can we alter the metrics to incentivize the correct behavior? How can we correct the metric?

Let’s take the rather simple example Joel gave us: what if we took the metrics a) and b) above, and added c) create few bugs? And created a composite metric something like the following:

Performance = ( amount of code + bugs fixed ) / (bugs created)

How would you game that? Let’s iterate through this process in the comments – let me know how you’d game it, and I’ll modify the metric (in case it’s not obvious, I’ve done this iteration before)

[Slashdot] [Digg] [Reddit] [del.icio.us] [Facebook] [Technorati] [Google] [StumbleUpon]

Comments

2 Responses to “Metrics and Incentives”

  1. Linda Ferguson on November 18th, 2006 10:46 am

    offering a complete outsider’s eye. . .

    Why is the metric about bugs instead of about measuring whatever it is one is trying to accomplish? If one measure of code is elegance, how does lines of code reflect that?

    In the measure you propose, it looks to me like bugs created and bugs fixed could cancel each other out – leaving lines of code as your metric (assuming people fix their own bugs and that they find all their own bugs – I assume if everyone found all their own bugs, we would not have end products with bugs in them! but you could use those bugs in the metric when and if you found them.

    What about other criteria? I know someone who has worked with young programmers who would rather reinvent the wheel than work with a template – inevitably, that’s not productive since the template was constructed for reasons the young programmer didn’t take the time or have the experience to evaluate.

    How do you measure how much time coding errors cost other people in the organization?

  2. Amrit on November 18th, 2006 11:10 am

    Lines of code is a poor metric for quality, performance or the most important adherence to requirements. it incentivizes the wrong behavior. Anyone who uses amount of code to track performance is asking for a world of trouble.

    As for the forumula that Murray describes it relies on the defect tracking system, and the process behind it, to be working well. One of the reasons I introduced the concept of RTQA was to avoid developers that would “game” thhe system. Essentially they could hide bugs by marking them deferred or resolved and then putting them in a state where they would go unnoticed among the general fog of battle. They would popup at the end of a cycle and it woudl turn out that a critical flaw had been deferred by a developer, hidden from the normal QA routine and a decision had to be made as to how it would be dealt with since it would cause a release delay or worse.

    Back in the late 90′s I came up with a concept called return to QA, essentially QA was responsible for moving all defects to closed or verified. All developers could do was mark a defect RTQA; RTQA/fixed, RTQA/Duplicate, RTQA/Defer, etc – they became assigned to QA and they were now accountable for them. This avoided the issue of developers “hiding” defects or moving them into a state they should not be in. RTQA forced QA to rerview the defects and developers to be more diligent in how they proceeded with resolution.

    It was easy to look at one set of metrics as defects assigned vs. closed. new, ppen to dev, RTQA/* vs. verified or closed. Monitoring the release against these metrics in almost all cases gave a strong indicator as to how the team was doing.

Forget the Parachute, Let Me Fly the Plane

Learn the same processes that the most successful people follow to find the job of their dreams!

Get your copy now!

Read more...

Frustrated sending in your resume and getting no response? Sign up today for “10 Hidden Secrets of Successful Resumes”!