Agile Product Development Metrics
1. Team productivity and efficiency metrics
These metrics help you discover whether the forecasted delivery of value tousers is taking place efficiently and predictably. These metrics delvespecifically into the productivity and efficiency of the team’s developmentprocess.Here is a list of metrics to track the speed of development and delivery ofvalue in the application development process. * Lead time: This is the time elapsed between the identification of a requirement and its fulfillment. It is the duration between the formulation of a user story and its subsequent release. * Cycle time: This is the time from when the actual work is started by the agile team until the line item goes from in-progress to completed. Each team has its own definition of complete, so the definition of done should be determined right at the beginning. Since the acceptance criteria vary from person to person, defining them at the start is essential to avoid confusion. * Sprint velocity: Velocity is the amount of work completed by the agile team in a given period of time. Sprint velocity is basically the rate at which the conditions mentioned in the software requirements specifications get converted into lines of tested code, or the number of story points covered by the team on every sprint. * Flow efficiency: Lead time includes both working time and wait time. The ratio of the actual work time measured against the total wait time gives us the flow efficiency. * Stories committed vs. completed: The completion against commitment metric (CaC) is the percentage of user stories completed in the sprint against the total committed during the planning meetings. * Sprint burndown: Also known as the burndown chart, this is a graphical representation of estimated tasks planned and the actual tasks completed. * Cumulative flow diagram: This is a visual representation of the stages all the different tasks are in at any given period of time. The vertical axis denotes the various story points, while the horizontal axis is the time. * Control chart: This denotes the cycle time of issues to go from in-progress to the done stage. Agile teams with consistently shorter cycle times have greater throughput, and the control chart helps visualize any changes on an immediate basis.
2. Development quality metrics
Quality metrics deal with the quality of the application developed, namely thenumber of defects discovered either during production or as a part of userfeedback.In agile, the QA mindset is embedded into the development process. The entireteam takes ownership of the quality of the application being developed andorganic improvement in the quality of software being developed.Here are a few quality-focused agile metrics you can choose from to ascertainthe quality of the application being developed. * Escaped defects: This is the number of bugs the agile team encounters after a build, after a sprint completes, or when the release enters into production. * Defect density: This is the number of defects detected in the software module within a particular duration, divided by the size of the software being developed. * Code churn: Churn measures the lines of code added, removed, or altered. By visualization of the fluctuations in the churn rate, the quality of the software being developed can be predicted. Agile teams should aim at churn close to zero as releases approach. * Defects deferred: Bugs entering the deferred state implies that the bugs discovered are scheduled to be fixed in the next release. * Automated test coverage: Test coverage is the amount of testing performed on a product. Automated test coverage calculates the degree to which the application code is covered by test cases and what portion of it takes place in an automated manner. * Failed deployments:This is the total number of deployments released into the testing and production environments that fail. It helps in determining the reliability of these environments and whether your agile teams are developing working software.
Testing Metrics Before Agile
In the old waterfall model for software development, QA was a separateactivity from development, performed by a separate team. The waterfall modeltakes a non-iterative approach to development where each stage needs to becompleted before the next stage begins. In this model, an independent qualityassurance team defines test cases which determine whether the software meetsthe initial requirements.SourceIn the waterfall model, the QA dashboard focused on the software applicationas a whole, and measured four key dimensions: * Product quality: The number of defects and the rate of defects was measured to gauge software quality. * Test effectiveness: Code coverage was checked for insight into test effectiveness. Additionally, QA also focused on requirements-based testing and functional tests. * Test status: The number of tests run, passed, or blocked would check the status of testing. * Test resources: The time taken to test software and the cost of that testing. The modern Agile development environment relies on the collaborative effort ofcross-functional teams. Thus, the metrics that took on such importance in theold independent waterfall model are less relevant today—testing is now anintegrated part of the entire development process.With QA teams becoming part of a cross-functional Agile effort, new metricsemerge that reflect this integrated environment. New goals and expectationslead to new metrics that can help the whole team from a united perspective.There are two types of Agile testing metrics: 1. General Agile metrics that are also relevant for software tests. 2. Specific test metrics applicable to an Agile development environment.
Number of Working Tested Features/Running Tested Features
The Running Tested Features (RTF) metric tells you how many software featuresare fully developed and passing all acceptance tests, thus becomingimplemented in the integrated product.The RTF metric for the project on the left shows more fully developed featuresas the sprint progresses, making for a healthy RTF growth. The project on theright appears to have issues, which may arise from factors including defects,failed tests, and changing requirements.Relevance to testing: * Since RTF metrics measure features that have undergone comprehensive tests, all features included in the metric have passed all of their tests. * More features shipped to the customer means more parts of the software have been tested.
The Missing Testing Metric—Software Quality
There’s been one thing missing in all the scrum metrics we covered—softwarequality. Some metrics like escaped defects, defect categories and defect cycletimes shed some light about quality. But they don’t hit the nail on the headand tell you “how good” your release really is.When you release, after all the effort invested in tests and bug fixes, areyou really confident that the new release will work well in production? Canyou be sure you have identified and covered all the important quality risks?Few agile teams can say that.To learn more about the missing metric in agile testing, and how to getcomplete visibility of your release quality, read our white paper, Quality:The Missing Agile MetricRequest A DemoHow Agile Metrics can Boost your Product Development ProcessA report by the Harvard Business Review states that companies that adopt Agileprocesses, experience 60% in revenue and profit growth. The statisticssurrounding Agile Methodology thus points to the simple fact that thismethodology has grown legs. In simpler words, the adoption of agilemethodology has become popular owing to the fact that it is the secret saucefor ensuring efficient product development. That said, businesses need to beacquainted with an aspect of Agile called the agile metrics to effectivelyreap the benefits. What exactly is Agile Metrics?Metrics could be defined as a system or a standard of measurement. Therefore,Agile metrics help development teams to measure the various aspects of thesoftware development cycle, such as the product’s quality and delivery, andthe productivity of the team. While quantitatively measuring work, thesemetrics also enable teams to become self-aware. The agile metrics can bebroadly classified into three families. They are as follows: * Lean Metrics: Helps control and observe development processes to facilitate continuous quality improvement by eliminating unnecessary activities. Eg: Cycle Time, Lead Time. * Scrum Metrics: Widely used metrics that focus on the effective delivery of the working software to the clients. Eg: Burndown Chart. * Kanban Metrics: Focuses on measuring the workflow and enables to predict the time required by the team to complete the project. Eg: Cumulative Flow.Mark Twain once remarked, “If the metrics you are looking at aren’t useful inoptimizing your strategy- stop looking at them.” However, the process ofdifferentiating between useful metrics and the ones that aren’t is easier saidthan done. Nevertheless, as a result of our experience of working with Agile,we have compiled a list of Agile metrics that could help boost your productdevelopment process. We have grouped them according to the results deliveredby these metrics. Let us take a look at some of the most effective metrics inthe world of Agile.
Agile Quality Metrics
Release Net Promoter Score: Also called as NPS, Net Promoter Score helps tomeasure customer satisfaction while also predicting the business growth.Customer satisfaction is an important factor that determines success, andthus, it becomes important to gauge the customer experience. This agile metricmeasures the likelihood of the customers recommending or not recommending thesoftware to others. The NPS index ranges from -100 to 100 and helps understandthe customer trends while constantly improving the product quality. Therespondents are classified into the following categories: * Promoters: Customers who are likely to contribute to the business growth by buying the software while also recommending it to others. * Passives: Although these customers are satisfied with the product, they are unenthusiastic and more prone to lean towards competitive offerings. * Detractors: As the name suggests, these are unsatisfied customers who can detract business growth.Let us take a look at an illustration of the above-mentioned classificationfor better understanding.Defects Leakage: This is a metric that helps to identify the number of defectsafter the build has been released to the customer for UAT or production. Insome cases, this metric is also used to track those bugs that had not beenaddressed in the same sprint and had flowed over to the next sprint. Thismetric proves to be highly relevant and useful as it helps maintain andmeasure the deployed software quality.Failed Deployments: It helps measure the number of deployments while measuringthe reliability of the testing and production environments. It is a highlyessential quality metric that helps understand if the sprints are production-ready or not.
Agile Velocity: Also known as ‘Velocity,’ this metric measures the number ofcompleted story points on an average over the past sprints. It’s a powerfulresult metric that helps measure the team’s productivity and also predicts itsefficiency in the upcoming sprints. It is important to understand thatvelocity keeps evolving with respect to the optimized work processes. Hencethe teams have to monitor its evolution through time. A decrease in velocitywould suggest inefficiencies in some parts of the development process. Take alook at a sample velocity chart.Sprint Burndown Chart: Similar to Agile Velocity, this metric helps visualizethe number of completed and remaining story points. It enables theScrum/Kanban master to track the completion of story points in real-time andhelps predict if the sprint scope will be completed on time. It is animportant productivity metric as it helps measure the value delivered by thesprint while showing how efficient and agile the team is.Given below is a sample sprint burndown chart.Epic and Release Burndown: This metric tracks the development progress over alarger body of work as compared to the sprint burndown chart. It guides thedevelopment for both scrum and kanban teams and focuses on tracking theprogress of existing epics and versions. The epic and release burndown chartensures that the entire team is kept aware of the workflow in the epic andversion.
Agile Product Development Metrics
With the world of Agile becoming highly competitive, one has to keep up withthe fast-paced development and changes in the scenario. At this point, we canaccept that the hype around Agile is not a fad that is going to be short-lived. With that in mind, it is important to focus on its implementation, andfor that, one must include the above-mentioned list of metrics as a part ofthe workflow. Also, one must measure the various aspects of the SDLC forcontinuous improvement in your business processes and for the business toscale new heights.By, Amritha Nampalat Marketing Executive, RapidValueReferences: Scrum Institute, Kanbanize.10 Powerful Agile Performance Metrics
The Importance of Agile Testing Metrics
Agile methodologies place a special emphasis on quality because the end goalis delivering working software to users – buggy or unusable software is notworking software. Quality is also manifested in internal aspects that are notdirectly visible to customers, such as code quality, maintainability andtechnical debt.Agile testing metrics can help teams measure and visualize the effort spent insoftware quality, and to a certain extent, the results of this effort. Forexample, the escaped defects metric measures, across versions, sprints orproduct lines, how many bugs were discovered in production – whereas ideallybugs should be discovered and fixed during the development stage.
How to Use Testing Metrics in an Agile Environment
The idea behind the Agile approach to project management is to encouragecollaboration, transparency, and responsiveness to feedback among anintegrated team. Agile software development means using the set of principlesoutlined in the Agile manifesto to develop high-quality working software,frequently.The Agile methodology emphasizes fast-paced software development, meaningsoftware testing must also be performed at speed while remaining thoroughenough to ensure high quality.It is vital for Agile teams to find a way to evaluate and improve theirtesting efforts. Test metrics are useful for providing basic measurements ofthe effectiveness of any software testing effort in Agile teams.This post outlines what exactly testing is in Agile development by comparingit to traditional testing in the old waterfall framework for softwaredevelopment. You’ll also find out about Agile test plans and get the low-downon some useful Agile test metrics.We focus on six key metrics that are relevant to testing in an Agile team. SeeSeaLights’ agile testing metrics learning section for a wider list ofrecommended metrics.After reading this post, you will better understand how to measure the testingefforts of your software development teams and improve on them, leading tohigher-quality software and more productive development. In other words,you’ll be better placed to achieve the goals of Agile development.
Running Tested Features
Running tested features (RTF) is an Agile metric that measures the amount ofcustomer-defined software features verified as functioning by software tests.This metric is helpful because it essentially makes teams more agile by: * focusing on features instead of design or infrastructure, and * verifying each feature as working, producing ready-to-go software at each iteration.By measuring the RTF growth for a given project, teams can easily analyzewhether there are issues with the software coding or the tests used to verifyfeatures as functional. The data is visually represented as a line graph basedon a count of running tested features, providing easy verification on whetherthe quantity of running tested features grows with each iteration (asexpected).
Common Agile Testing Issues
Even with the many different metrics to measure, testing is, in itself, aproblem for Agile teams. It is clearly essential to test software thoroughlybefore releasing it, but testing tends to slow down the time to market forsoftware. Therefore, the main Agile testing issues revolve around implementingsolutions for improved efficiency and productivity. Some of the main Agiletesting challenges are: * Lack of test coverage. The pressure to get software out quickly can cause teams to write too few tests for user stories. It’s important to have visibility on all code changes to write enough tests to cover the code in a given user story. * Broken Code. The more frequently a team delivers builds, the higher the chances of breaking existing code. Daily regression testing is impractical with manual test runs. In addition, as the use of microservices becomes more common, where each microservice runs in its own pipeline to production, it is essential to verify that all the moving parts are functioning and are properly integrated. * Catching Defects Too Late. Defects found late in the development cycle cost much more to fix than those found earlier. This rule applies regardless of your project’s framework. The challenge is to figure out the best way of identifying defects as early as possible in an Agile framework. There is a need to shift left, meaning conducting software tests as early as possible in the development cycle. * Performance Bottlenecks. Agile teams need to understand how best to monitor software performance so that additional features do not cause significant system slowdowns.There are also a number of problems associated with tracking certain testmetrics. Such metrics create problems because they can either cause confusion,go against Agile principles, or otherwise provide little value. For example: * Tracking individual metrics. This goes against the Agile ethos because it encourages excessive competition among members of the same team. For example, measuring productivity by counting the number of tests written. Too much competition compromises teamwork and creates quality testing issues. * Tracking meaningless metrics. Meaningless metrics are those that don’t tell you anything about testing productivity. For example, comparing two Agile Scrum teams on their respective velocities is a poor metric, because velocity is unique to each team, as it depends on estimations unique to each team. Comparing velocities between teams encourages teams to fudge their estimates, leading to poor planning for sprints.The only way to overcome the potential for using problematic test metrics orusing test metrics incorrectly is to promote increased awareness of whatconstitutes a useful test metric in an Agile team among both team members andproject managers.