Velocity as A KPI In Scrum

A key performance indicator (KPI) is a measure of performance. Such measures are commonly used to help a teams or organizations define and evaluate how successful they are, typically in terms of making progress towards its long-term goals. Velocity for a Scrum team is a key planning tool and describes the quantity of work a team can be expected to deliver at the end of a sprint. From this velocity, the product owner can do release planning to predict when a feature or multiple features can be delivered to within, say, a month. Jeff Sutherland said in his blog, "Story Points: Why are they better than hours?":

"The management metric for project delivery needs to be a unit of production. ... The metric of important is the number of story points the team can deliver per unit of calendar time. The points per sprint is the velocity."

It makes sense that a product owner would use velocity as a means to predict the amount of work a team can deliver in a sprint, assuming the measure (i.e. story point) remains constant and core development team doesn't change. However, velocity is not and should not be the only metric used to measure success of a Scrum team. Some other measures might include:

• Cycle Time (or Lead Time) - The time from when a customer request comes into a process to a time when it has actually delivered to a customer. Good way to measure the whole process, not only development efficiency. With this, you can check how good your flow is.
• Rework - Things which are coming back to the process after they have declared done and delivered i.e. was reviewed during sprint review. Because those things are defects like bugs and misunderstandings in customer requirements, this is a measurement for quality and communication.
• Customer Satisfaction - This may be from the company's customer satisfaction surveys but should also include internal sources e.g. Product Support, Sales, Marketing, and Product Management.

Starting Off With Velocity

When a new Scrum team first starts working together or when a seasoned Scrum team begins a new project, velocity will most likely oscillate, sometimes wildly. This can be traced to the many 'unknowns' that exist. For a new Scrum team, team members will take some time to acclimate themselves to Scrum, user stories, understanding each other, new technologies, and accepting the reality that they can't do nor are they expected to do precision estimates. For an experienced Scrum team on a new project, the unknowns are usually related to technology and the vagueness of the customer requirements. In either of these cases, using velocity as an indicator of when the team becomes 'predictable' can be very useful, especially to the product owner but also to the team when it comes time to commit doing specific user stories in a sprint.

After a few sprints the team's velocity should stabilize and trend upward over time. But what if it doesn't stabilize? Is is because the user stories are 'bad'? The team members are changing? Or is there some other 'smell' getting in the way? If the velocity of the Scrum team is fluctuating beyond that expected for the particular team, I would suggest no more than 15% after 3-5 sprints, the team should use their sprint retrospective to do some analysis to figure out the root cause and identify actionable improvement measures that it can implement in the next sprint.

I would think a flat velocity is better than one trending down but unless there’s no room for improvement, i.e. the team is perfect in every way possible, then I think the velocity trend should be up over time and this is propably how the velocity KPI should be implemented.

The Scrum team should be managing their velocity to ensure its accuracy throughout the project. The Scrum team, product owner, developers, testers, scrum master, might use Mike Cohn's Velocity Calculator Tool which calculates a confidence interval around the median using the Binomial distribution to determine a 90% confidence interval for 'n' sprint velocity values. The target a Scrum team may wish to grow toward is to have the deviation from the mean, average, sprint velocity not to exceed 10%. This is well in line with question 6 of the Nokia Test: Estimate error < 10%.


Resistance to using velocity as a KPI

As with any changes to the development team and product owners, expect some resistance to having Scrum team KPIs including one on velocity. Team members may feel having a KPI that is based on team and not individual performance puts their current situation at risk. The scrum master and managers need to ensure the Scrum team understands that the team as a whole is greater than the sum of its parts and team KPIs measure the strength of the team. Some things that can help the Scrum team accept Team KPIs and a velocity KPI in particular are:

• Allow the Scrum team to set the velocity goals. Usually no specific target velocity is mentioned although there may be words to the effect that the velocity will gradually increase over time. By allowing the Scrum team to set a velocity goal, they'll feel more empowered and in control. For example, a Scrum team doing 18 +/- 10% story points per sprint may target 20 story points per sprint.
• Allow the Scrum team to remove early sprint velocities. When a team first begins using Scrum or when a new project is introduced to the team, sprint velocities may fluctuate greatly. The Scrum team should have the opportunity to remove aberrations to help 'smooth' out their velocity chart.
• Make it clear to each Scrum team that velocities are not comparable between teams. Velocity is very much a localized measure to a specific Scrum team. In addition to different team members with different team 'personalities', projects typically possess unique characteristics in terms of estimating techniques, detail process, technology, customer involvement, etc. As a result, this can make organization-wide analysis very inaccurate.

When is using velocity as a team KPI not advised?

You might want to reconsider using velocity as a KPI in the following circumstances:

• XP: Unlike Extreme Programing (XP), Scrum teams do not allow changes into their sprints; XP welcomes changes to the current iteration which makes using velocity as a team KPI in XP impractical. XP teams would use velocity at the project or release level but not as a metric from iteration to iteration.
• When the Scrum team is doing maintenance work (bug fixing): Unless the Scrum team is quite small, the team will rarely work one bug at a time. Teams doing full time maintenance work should probably look at Kanban rather than Scrum although the daily stand-up meeting should still happen.
• When dissimilar projects are very short: If the Scrum team is working on disimilar projects, i.e. projects where technology or product domain are not related, and are only 1 - 3 sprints long, it's not likely that the velocity would ever stabilize or the the velocity established on the previous project would be applicable. However, very short projects might indicate that the product owner or company doesn't have any long term objectives, especially in highly volatile markets. If this is the case, maybe XP or even Kanban is a better choice.
• When not everyone in the Scrum team is measured: If the scrum master, product owner, developers and testers all on one team are not measured the same then this can lead to either perceived or real conflicts of interest. Product owners write the user stories, developers and testers implement and test against the Scrum teams definition of done, and the scrum master is responsible to teach the Scrum team by coaching and by leading it to be more productive and produce higher quality products. All these factors affect velocity and these people need to be equally accountable.

Tweet

Acceptance Tests in Scrum

"Acceptance tests are the tests that define the business value each story must deliver."

-       Lisa Crispen & Janet Gregory, "Agile Testing: A Practical Guide For Testers and Agile Teams"

 "Acceptance testing is the process of verifying that stories were developed such that each works exactly the way the customer team expected it to work."

-       Mike Cohn, "User Stories Applied For Agile Software Development"

 Acceptance tests, derived from the User Stories’ acceptance criteria, serve three purposes:

1.    Acceptance tests are used by the product owner to know that the acceptance criteria have been met and the User Story is “Done” to their satisfaction.

2.    Acceptance tests help drive development in an “Acceptance Test Driven Development” environment.

3.    Acceptance tests serve as the source of product regression tests.

Acceptance criteria are a list of the specified indicators or measures used in assessing the ability of the product to perform its intended function and add real business value to the product. Acceptance tests are written based on the acceptance criteria and validate that the product has the expected business value i.e., functionality, intended by the User Story. 
Acceptance test cases further serve as a source of product regression tests.  It’s to the company's advantage to have in-place a collection of tests covering all functionality and acceptance tests are the means to getting that coverage.  Once User Stories are "Done" (acceptance testing being a significant contributor), the User Story is delivered to the "always releasable" branch of the configuration management system.  Along with the delivery of the User Story, the acceptance tests are “delivered” to the configuration management system.

Some of the issues that may arise when adopting acceptance testing as a measure of “Done” are:

1.    Developers on the Scrum Team feel it’s “the testers” responsibility to write and run functional tests.

One of my favourite comments early in the company’s adoption of Scrum occurred when the ScrumMaster was saying unit testing wasn’t enough and the Scrum Team needed to validate the new functionality by running tests from the end-user’s perspective. One of the team members suggested that the company “hire more testers” if functional testing was required. This is not an uncommon reaction from developers especially when, prior to going Agile, developers were not held directly responsible for quality testing and the test team had been a separate entity in the organization.

To help the team recognize their testing responsibilities, the ScrumMaster began to specifically ask during the Daily Scrum meeting if functional testing had been done by the developer and one other member of the team before a requirement (the team wasn’t doing user stories yet) was considered “Done”. This made the Daily Scrum meeting slightly longer but it quickly sank into the team that they were responsible for functional testing. It wasn’t long before a new team member was prevented from completing a requirement because the team was concerned that the appropriate testing had not been carried out.

2.    When there is a “tester” in the Scrum Team, the tester is the only person writing and running Acceptance Tests.

In this scenario the Scrum Team has more or less continued the pre-agile practice of having a separate test team. This practice virtually guaranteed that functional testing would occur very late in the iteration resulting more often than not with User Stories not being “Done” at the end of the sprint.

The behaviour Agile encourages is the development team, as a whole, to feel responsible for the quality of the product and for meeting the User Stories’ acceptance criteria. Having a professional tester in the team when first adopting Scrum can prove too tempting for the team to heap all functional testing upon this individual. When I was in defence, test was a separate entity from development and this worked quite well owing to the detail of the well documented software requirements. Agile doesn’t share the same enthusiasm for documenting every detail but rather prefers face to face communication. The best scenario is for the tester to share their test philosophy with the developers and help build their competencies for functional testing.

Eventually all professional testers were pulled out the Scrum Teams to focus their efforts on testing the releases and creating automated functional tests for regression testing. The immediate result was the Scrum Team no longer felt they had “dedicated testers” on the team. However, during the sprint planning meeting the Scrum Team (developers and product owner) spent more time discussing how they would test the User Stories. During these discussions, acceptance criteria might be added, modified, or removed but in the end, the Scrum Team had a much clearer understanding of the User Stories.

Jeff Sutherland had this to say about the Nokia Test question on dedicated testers:

“Testing within a sprint - there must be testers on the team that complete testing at the feature level during the sprint, i.e. the acceptance tests defined by the Product Owner and enhanced by the testers are completed. Only stories with completed tests are DONE and count towards team velocity. On some teams developers test (but not the same developer that wrote the code). For other teams, it is essential to have professional testers when the consequences of failure are high - a production system for treating patients in a hospital, for example.”

3.    The User Stories are written at an implementation level e.g. As a product owner, I want the SQL database to have a new table with fields x and y.

The thing we did short of having the product owners going on a User Story writing course was to have the development teams set the expectations using the INVEST qualities of a User Story – Independent, Negotiable, Valuable, Estimable, Small, Testable (I believe these were first defined by Bill Wake). And it wasn’t only the product owners put on the spot; the development team’s also needed to actively contribute to bettering the quality of User Stories.

4.    The Acceptance Tests are written after the code is completed and validate the result rather than the intent.

Following a “Test Last” approach is generally not a good choice. There are a couple of potential problems with testing last including:

·         Dropping acceptance test during crunch time. This could occur any time but is more likely to happen in a test last scenario.

·         The tests may be biased toward the solution rather testing the requirements.

The easiest way to avoid these pitfalls is to:

1.    Adopt a test first approach – write acceptance tests based upon the acceptance criteria before any code is written.  This will

·         ensure the acceptance tests  are written,

·         serve as an aid to design for the development team,

·         automated tests can be run daily with the daily build i.e. there would failures at first but when the tests all pass, the User Story is complete, and

·         give the product owner plenty of opportunity to review and comment on the test early in the User Story implementation.

2.    Get the product owner to review and approve the acceptance tests before writing any code. 

5.    Acceptance Tests are not automated.

It is sometimes difficult to convince the development team and product owner that automating acceptance tests (feature, function, or user tests) are worth the effort. This has been a common position taken by the Scrum Teams and although there is evidence showing that running regression tests would have caught problems later caught by customers, the Scrum Teams have remained unconvinced. The most common excuse I’ve heard is that it is “too much work”.  The reality is that teams do not re-run the manual product acceptance tests to ensure that previously implemented functionality still works so maybe the thinking goes, why automate tests that will never be run again.

Automated acceptance tests help increase quality by catching errors early in the development cycle. The product is improved by creating a "safety net" of tests which can be run during the daily build. The value of this is obvious, but what is not as obvious is that automated acceptance tests also reduce the time to market and cost of development by shortening the development time. This is accomplished by decreasing a developer's time in debugging loops by catching errors in the safety net of tests.

There are a couple of measures that could be taken to encourage Scrum Teams to develop automated acceptance tests and include:

·         Having a KPI that measures the percentage of automated tests per iteration.

·         Having the development team sit on the product support desk for two or three months doing root cause analysis on the customer issues. This may help them see where additional automated testing would be most helpful.

Once, when a delivery was made to a customer site and some fundamental functionality didn’t work, root cause analysis found that the problem had slipped through code reviews, unit tests and acceptance tests. The new functionality tested fine but in the process of implementing the new functionality, previously working functionality had unknowingly ceased working. Because any regression testing would have been manual and time to release was critical, no regression testing was identified. Had functional tests been automated as part of the daily build, no one would have been required to make the decision to skip regression testing and the problem would have been caught prior to release. 

 Summary

Acceptance testing is a critical component to the definition of “Done” for Agile User Stories and is the product owners’ best means to ensure the desired functionality of the User Story is the development team’s primary focus. Acceptance testing, or more accurately, the writing of acceptance tests, can help the development team understand the software requirements of the User Story before any code is written. Acceptance tests also provide a legacy of regression tests that can be used to ensure the product hasn’t regressed in other areas when new User Stories are implemented.

Tweet