Tuesday, May 24, 2011

Velocity as A KPI In Scrum

A key performance indicator (KPI) is a measure of performance. Such measures are commonly used to help a teams or organizations define and evaluate how successful they are, typically in terms of making progress towards its long-term goals. Velocity for a Scrum team is a key planning tool and describes the quantity of work a team can be expected to deliver at the end of a sprint. From this velocity, the product owner can do release planning to predict when a feature or multiple features can be delivered to within, say, a month. Jeff Sutherland said in his blog, "Story Points: Why are they better than hours?":

"The management metric for project delivery needs to be a unit of production. ... The metric of important is the number of story points the team can deliver per unit of calendar time. The points per sprint is the velocity."

It makes sense that a product owner would use velocity as a means to predict the amount of work a team can deliver in a sprint, assuming the measure (i.e. story point) remains constant and core development team doesn't change. However, velocity is not and should not be the only metric used to measure success of a Scrum team. Some other measures might include:

  • Cycle Time (or Lead Time) - The time from when a customer request comes into a process to a time when it has actually delivered to a customer. Good way to measure the whole process, not only development efficiency. With this, you can check how good your flow is.
  • Rework - Things which are coming back to the process after they have declared done and delivered i.e. was reviewed during sprint review. Because those things are defects like bugs and misunderstandings in customer requirements, this is a measurement for quality and communication.
  • Customer Satisfaction - This may be from the company's customer satisfaction surveys but should also include internal sources e.g. Product Support, Sales, Marketing, and Product Management.

Starting Off With Velocity

When a new Scrum team first starts working together or when a seasoned Scrum team begins a new project, velocity will most likely oscillate, sometimes wildly. This can be traced to the many 'unknowns' that exist. For a new Scrum team, team members will take some time to acclimate themselves to Scrum, user stories, understanding each other, new technologies, and accepting the reality that they can't do nor are they expected to do precision estimates. For an experienced Scrum team on a new project, the unknowns are usually related to technology and the vagueness of the customer requirements. In either of these cases, using velocity as an indicator of when the team becomes 'predictable' can be very useful, especially to the product owner but also to the team when it comes time to commit doing specific user stories in a sprint.

After a few sprints the team's velocity should stabilize and trend upward over time. But what if it doesn't stabilize? Is is because the user stories are 'bad'? The team members are changing? Or is there some other 'smell' getting in the way? If the velocity of the Scrum team is fluctuating beyond that expected for the particular team, I would suggest no more than 15% after 3-5 sprints, the team should use their sprint retrospective to do some analysis to figure out the root cause and identify actionable improvement measures that it can implement in the next sprint.

I would think a flat velocity is better than one trending down but unless there’s no room for improvement, i.e. the team is perfect in every way possible, then I think the velocity trend should be up over time and this is propably how the velocity KPI should be implemented.

The Scrum team should be managing their velocity to ensure its accuracy throughout the project. The Scrum team, product owner, developers, testers, scrum master, might use Mike Cohn's Velocity Calculator Tool which calculates a confidence interval around the median using the Binomial distribution to determine a 90% confidence interval for 'n' sprint velocity values. The target a Scrum team may wish to grow toward is to have the deviation from the mean, average, sprint velocity not to exceed 10%. This is well in line with question 6 of the Nokia Test: Estimate error < 10%.


Resistance to using velocity as a KPI

As with any changes to the development team and product owners, expect some resistance to having Scrum team KPIs including one on velocity. Team members may feel having a KPI that is based on team and not individual performance puts their current situation at risk. The scrum master and managers need to ensure the Scrum team understands that the team as a whole is greater than the sum of its parts and team KPIs measure the strength of the team. Some things that can help the Scrum team accept Team KPIs and a velocity KPI in particular are:
  • Allow the Scrum team to set the velocity goals. Usually no specific target velocity is mentioned although there may be words to the effect that the velocity will gradually increase over time. By allowing the Scrum team to set a velocity goal, they'll feel more empowered and in control. For example, a Scrum team doing 18 +/- 10% story points per sprint may target 20 story points per sprint.
  • Allow the Scrum team to remove early sprint velocities. When a team first begins using Scrum or when a new project is introduced to the team, sprint velocities may fluctuate greatly. The Scrum team should have the opportunity to remove aberrations to help 'smooth' out their velocity chart.
  • Make it clear to each Scrum team that velocities are not comparable between teams. Velocity is very much a localized measure to a specific Scrum team. In addition to different team members with different team 'personalities', projects typically possess unique characteristics in terms of estimating techniques, detail process, technology, customer involvement, etc. As a result, this can make organization-wide analysis very inaccurate.

When is using velocity as a team KPI not advised?

You might want to reconsider using velocity as a KPI in the following circumstances:
  • XP: Unlike Extreme Programing (XP), Scrum teams do not allow changes into their sprints; XP welcomes changes to the current iteration which makes using velocity as a team KPI in XP impractical. XP teams would use velocity at the project or release level but not as a metric from iteration to iteration.
  • When the Scrum team is doing maintenance work (bug fixing): Unless the Scrum team is quite small, the team will rarely work one bug at a time. Teams doing full time maintenance work should probably look at Kanban rather than Scrum although the daily stand-up meeting should still happen.
  • When dissimilar projects are very short: If the Scrum team is working on disimilar projects, i.e. projects where technology or product domain are not related, and are only 1 - 3 sprints long, it's not likely that the velocity would ever stabilize or the the velocity established on the previous project would be applicable. However, very short projects might indicate that the product owner or company doesn't have any long term objectives, especially in highly volatile markets. If this is the case, maybe XP or even Kanban is a better choice.
  • When not everyone in the Scrum team is measured: If the scrum master, product owner, developers and testers all on one team are not measured the same then this can lead to either perceived or real conflicts of interest. Product owners write the user stories, developers and testers implement and test against the Scrum teams definition of done, and the scrum master is responsible to teach the Scrum team by coaching and by leading it to be more productive and produce higher quality products. All these factors affect velocity and these people need to be equally accountable.

4 comments:

  1. As you mentioned near the start, care needs to be taken so that Velocity (or production in general) isn't the only KPI - or the team will produce rubbish to meet/exceed their KPI. KPIs need to be balanced so that pulling one pushes another, resulting in the team working towards what the business actually wants (eg more productivity AND higher quality) rather than just the KPI.

    Using velocity as a Team KPI when the projects are very short can still be realistic. If they're individual projects in a domain the team is familiar with - and can continue using the same reference story - then project size should be immaterial.

    Using KPIs makes some people start acting with questionable ethics. Committing to conservative story point counts for a sprint to improve estimate accuracy (but therefore affecting the velocity KPI), or not wanting to pull in more stories when they complete early (and therefore under-estimated) so as not to affect their estimate accuracy. This is just like a kid not wanting to accept the consequences of a mistake they've made.

    I think so long as KPIs are correctly balanced to engender the required behaviours, teams will be, or become over time, honest with themselves and work towards project goals, and realise the KPIs will sort themselves out so long as they follow Agile/Scrum correctly.
    But having the wrong, or imbalanced, KPIs could be detrimental.

    ReplyDelete
  2. I agree with your blog and with Tony's comments. I will just add that the velocity KPI should measure an increase in velocity over time rather than just the attainment or maintenance of a certain velocity value. The point is for the teams to continuously improve rather than to reach any prescribed level.

    ReplyDelete
  3. Thank you for sharing. With iReliability you can track and trend your facility’s Maintenance KPIs over the web anytime and anywhere.

    ReplyDelete

Note: Only a member of this blog may post a comment.