I was reading an article, on the Embedded Systems Design web site, titled “Big Code”. The author, Jack Ganssle, talks about the estimated number of lines of code (LOC) in the upcoming Microsoft Windows Vista as being around 50 million. At a 5% bug rate, this would yield a total bug count of 2.5 million bugs. A CMMI level 5 organization should average a bug rate of one bug per one thousand lines of code (KLOC) (Capers Jones Software Assessments, Benchmarks, and Best Practices, Addison Wesley, 2000). If Microsoft was a CMMI level 5 rated company, this would mean Windows Vista could ship with 50,000 bugs.
Where does the bugs per KLOC statistic come from? The Software Engineering Institute (SEI) tracks objective evidence from companies and projects to demonstrate the costs and benefits of CMMI. Performance results for CMMI teams are measured in several categories: Cost, Schedule, Productivity, Quality, Customer Satisfaction, and Return on Investment. Capers Jones book listed the quality measurement category, the defects per KLOC for each maturity level as
Microsoft spends a lot of internal resources for testing. They also leverage an “ocean“ of partners, developers and customers to help with testing and compatiblity checks. How many of you have joined their “experience improvement program“ where you can be more active in testing and reporting problems? How many of you have sent in crash bug reports to Microsoft? Instrumenting software so that the users can help gather information to track down nasty bugs is something that we have also been putting into our products. Microsoft has also been releasing preview versions of the operating system to gain additional feedback and test/bug reports. We are helping Microsoft by testing our products for compatibility. We are also working with Microsoft to ensure that our future products will work well with Vista when it comes out. It will be interesting to see how much quality we can “ensure” when Windows Vista finally ships.
Online community projects like our BDN QualityCentral are just one way to help improve quality in software products. While it is not a substitute for the automated and human testing that we do ourselves, it is another tool in our quality efforts.
You can find more CMM and CMMI information at the SEI site. A good place to start is the CMMI Overview presentation. Additional information and links are available on the Wikipedia CMM entry.
{ 6 } Comments
CMMI, in my humble opinion, is just a certificate that says, "hey! we are a matured company". CMMI is about process and process in no way will directly influence the reduction of bugs (Murphy’s Law). Optimizing companies tend to be vastly smaller than MS. Products of these companies are also vastly smaller than any of MS offerings, hence, lesser bugs.
How many lines is Delphi? And can you see where this question is leading? Do Borland keep track of metrics like this for their own internal usage?
This calculation makes no sense, a lot of code is generated, maybe 10% is human-written.
[quote]This calculation makes no sense, a lot of code is generated, maybe 10% is human-written. [/quote]
My question then is, who wrote the generator?
At one time or another it all comes down to 100% human input.
bert, Where did you get your information that only 10% of Vista is hand-written code?
This calculation assumes that the estimated 50+ million LOC are being written from scratch, which is not the case in Vista, as it will benefit from a large Windows code-base which has been worked on and debugged years after years.
To me, this calculation makes no sense.
Regards,
/CL
Post a Comment