Many companies have increased productivity, improved quality, established better control of their projects, reduced cycle time and saved costs by using the SEI CMM® as a guide for process improvement. Organizations typically report productivity improvements of 10-50% and a return on investment of 4:1 (cost savings realized vs. process improvements dollars invested). The following table is extracted from the SEI report Performance Results of CMMI®-Based Process Improvement, CMU/SEI-2006-TR-004, which can be downloaded from http://www.sei.cmu.edu/publications/documents/06.reports/06tr004.html.
Another good source of results data is the March 2007 issue of Software Tech published by DACS, the Data & Analysis Center for Software. That issue can be accessed at https://www.softwaretechnews.com/stn_view.php?stn_id=41.
| Performance Category | Median Improvement |
|---|---|
| Cost | 34% |
| Schedule | 50% |
| Productivity | 61% |
| Quality | 48% |
| Customer Satisfaction | 14% |
| Return on Investment | 4.0 : 1 |
Other data from previous process improvement activities is presented below.
General Dynamics Decision Systems (reported in Crosstalk, March 2002) noted the improvement trends shown in the table below. Return on Investment was calculated at: Level 2-Level 3: 167%; Level 3-Level 4: 109%; Level 5: 14%.
| SEI CMM® Level | Percent Growth | Phase Containment Effectiveness | CRUD Density per KSLOC | Productivity (relative) |
|---|---|---|---|---|
| 2 | 23.2% | 25.5% | 3.2 | 1x |
| 3 | 14.3% | 41.5% | 0.9 | 2x |
| 4 | 9.5% | 62.3% | 0.22 | 1.9x |
| 5 | 6.8% | 87.3% | 0.19 | 2.9x |
Master Systems, Inc. (data reported by Giga Information Group, May 28, 1999) For a typical 200,000 LOC development project (based on data from 1200 completed projects):
| CMM® Level | Calendar Months | Level of Effort person-months | Defects Shipped | Median Cost | Lowest Cost | Highest Cost |
|---|---|---|---|---|---|---|
| Level 1 | 30 | 600 | 61 | $5.5 million | $1.8 million | $100 million+ |
| Level 2 | 18.5 | 143 | 12 | $1.3 million | $960,000 | $1.7 million |
| Level 3 | 15 | 80 | 7 | $728,000 | $518,000 | $933,000 |
Effects of Maturity on Software Effort (Bradford Clark, Univ. of Southern California - reported at 1998 SEPG Conference) Clark created a formula and used regression analysis to determine how much effort is reduced as companies progress up the SEI CMM® maturity ladder. He collected data from 112 projects. Using his mathematical models, he concluded that Software Process Maturity was a significant factor (95% confidence level) affecting software development effort. After normalizing for the effects of other influences on effort, a one level increment change in maturity level resulted in a 15% - 21% reduction in effort. An interesting side note is that Personal Capacity (people) has an even stronger influence than effort. This supports the notion that you can have great processes, but if you do not have teams that work well together little benefit is derived. Improvement in the People CMM® capabilities will greatly enhance the benefits derived by improving in the Software CMM®.
Boeing (SEPG Conference 1998) Defect program added 4% to project cost, saved 31% in rework, for ROI of 7.75 to 1. Benefits of Software Process Improvement were small at Level 2, more at Level 3, major benefits at Levels 4 and 5.
Bellcore (SEPG Conference 1998) Results (over a 5 year period) of their process improvement efforts were a reduction in defects from 48 per 1000 Function Points (a measure of software size) to approximately 8 per 1000 Function Points, an increase in customer satisfaction from 60% to 91%, and a dramatic decrease in maintenance costs.
Litton PRC (SEPG Conference 1998) Some of the results of their quality and process improvement efforts were: CMM® level 1 to level 3 in 3 years, 63% reduction in employee turnover and a 51% improvement on incident closures. Organization has a 4% attrition rate, down from 40%.
Motorola Government Electronics Division (reported in IEEE Software, Sep/Oct 1997) Improvement trends for quality, cycle time and productivity by SEI level (based on project internal self-Appraisals).
| SEI CMM® Level | Number of Projects | In-Process Defects/Million Assy-Equiv. Lines of Code | Cycle Time (x factor)* | Productivity (relative) |
|---|---|---|---|---|
| 1 | 3 | N/A | 1.0 | N/A |
| 2 | 9 | 890 | 3.2 | 1.0 |
| 3 | 5 | 411 | 2.7 | 0.8 |
| 4 | 8 | 205 | 5.0 | 2.3 |
| 5 | 9 | 126 | 7.8 | 2.8 |
Raytheon Equipment Division: (1996)
Improved Project Control: General Industry Performance (also reported in CrossTalk, Sep 1995) Flowe and Thordahl, while at the Air Force Institute of Technology, did a correlational study of CMM® ratings and software development performance on DOD contracts. They reviewed 52 projects which had received SEI ratings. Their data showed improvements in both cost and schedule performance for higher maturity organizations. Specifically, higher maturity organizations were less likely to have cost overruns or schedule overruns; further, the Level 3 organizations had a greater degree of predictability of cost and schedule.
Go to http://www.sei.cmu.edu/cmmi/results.html for recent results data.
New products and services from the SEI include the new certification for SCAMPISM Lead Appraisers and the CMMI constellation for services (CMMI-SVC) which was released in March 2009.