Software Engineering Services Blog

The Challenge of Measuring Software Development Effectiveness

  

 Email Article  

Tweet  

  

  

 delicious  

  

  

by Neil Fox

Is software development really different from any other business function?

Think about it for a moment. Companies continually work to improve sales effectiveness, manufacturing effectiveness, operational effectiveness, etc., and as a result nearly every aspect of an organization can measure just how effective it is at performing its function. So why not software development?

Why we don’t measure development effectiveness

There are three primary reasons that our industry has been unable to address this challenge:

1. Effectiveness depends on business context. There are times when companies must focus on getting their product out the door (time to market). As they mature, they become more focused on feature development or product enhancement. Later in their lifecycle, priorities shift to maintaining legacy applications and driving innovation. All of these activities are inherently different and require different approaches to be performed effectively but we tend to measure them with the same yardstick: Did we meet the objective or not? The alternative here would be defining effectiveness for each business context or phase in the lifecycle and then measuring performance accordingly.
2. Lack of a standard measure of output. It is quite difficult to understand how much work a team has produced if there is no real way to measure their output. For a brief period of time, organizations used KLOC or lines of code produced. It wasn’t long before people realized that this encouraged poor development practices and really bad code. Again, I recommend finding a reasonably objective and implementable standard that applies to the work at hand. Depending on the nature of the work, you might consider using a standardized scale of complexity points, story points, ideal hours, function points, or even the number of defects fixed. Ultimately, to measure effectiveness in a meaningful way, you need a standardized unit of work that is relevant to the goal.
3. Measurement drives behavior. It’s been demonstrated again and again that people will align their behavior to factors that are measured. If you measure the number of story points delivered, then your team will knock out story points, nearly to the exclusion of all else. The same holds true if you measure defects, estimation accuracy, or anything else. The trick to measuring effectiveness is choosing metrics that will encourage the right behavior but not distract from the ultimate outcome: a successful, timely, and bug-free release. After all, the metrics are a guide and an early warning system, not the goal of the development process.

The goal of software development

Generally, we can agree that the goal of software development is to create new, high quality products and features for end users (though sometimes those end users are other systems). To achieve that goal most businesses want to understand how quickly they can deliver a set of functionality to their target market with the highest total quality. Measuring effectiveness in a relevant way means focusing on those activities that have the greatest impact on time to market and quality.

Software quality may be judged on a range of attributes. To ensure and promote effectiveness, the attributes of quality we want to measure should include, but not be limited to: technical defects; functional defects; security; reliability; performance; scalability; usability; and even something as elusive as “innovation.” Of course, to serve as standards of measure, each of these attributes needs to be translated back into the practices and activities that influence them.

And making the connection between the desired outcome (quality software) and the activities most likely to produce it is where the real work comes in.

We’ll dig deeper into the details of what to measure and how in the next entry.

Neil Fox heads the strategic consulting unit at Ness. His team partners with clients to maximize return on technology investments. He was an early member of Macromedia’s team, a pioneer of Internet technology while at TRW in the 90’s (now part of Northrop Grumman), and  has led large software development efforts for most of his 25+ year career. He can be contacted at Neil.Fox@ness.com.