How many bugs will be in Microsoft’s Windows Vista?
I was reading an article, on the Embedded Systems Design web site, titled “Big Code”. The author, Jack Ganssle, talks about the estimated number of lines of code (LOC) in the upcoming Microsoft Windows Vista as being around 50 million. At a 5% bug rate, this would yield a total bug count of 2.5 million bugs. A CMMI level 5 organization should average a bug rate of one bug per one thousand lines of code (KLOC) (Capers Jones Software Assessments, Benchmarks, and Best Practices, Addison Wesley, 2000). If Microsoft was a CMMI level 5 rated company, this would mean Windows Vista could ship with 50,000 bugs.
Where does the bugs per KLOC statistic come from? The Software Engineering Institute (SEI) tracks objective evidence from companies and projects to demonstrate the costs and benefits of CMMI. Performance results for CMMI teams are measured in several categories: Cost, Schedule, Productivity, Quality, Customer Satisfaction, and Return on Investment. Capers Jones book listed the quality measurement category, the defects per KLOC for each maturity level as
Microsoft spends a lot of internal resources for testing. They also leverage an “ocean“ of partners, developers and customers to help with testing and compatiblity checks. How many of you have joined their “experience improvement program“ where you can be more active in testing and reporting problems? How many of you have sent in crash bug reports to Microsoft? Instrumenting software so that the users can help gather information to track down nasty bugs is something that we have also been putting into our products. Microsoft has also been releasing preview versions of the operating system to gain additional feedback and test/bug reports. We are helping Microsoft by testing our products for compatibility. We are also working with Microsoft to ensure that our future products will work well with Vista when it comes out. It will be interesting to see how much quality we can “ensure” when Windows Vista finally ships.
Online community projects like our BDN QualityCentral are just one way to help improve quality in software products. While it is not a substitute for the automated and human testing that we do ourselves, it is another tool in our quality efforts.
You can find more CMM and CMMI information at the SEI site. A good place to start is the CMMI Overview presentation. Additional information and links are available on the Wikipedia CMM entry.
October 9th, 2005 at 8:28 pm
CMMI, in my humble opinion, is just a certificate that says, "hey! we are a matured company". CMMI is about process and process in no way will directly influence the reduction of bugs (Murphy’s Law). Optimizing companies tend to be vastly smaller than MS. Products of these companies are also vastly smaller than any of MS offerings, hence, lesser bugs.
October 10th, 2005 at 3:08 am
How many lines is Delphi? And can you see where this question is leading? Do Borland keep track of metrics like this for their own internal usage?
October 10th, 2005 at 3:22 am
This calculation makes no sense, a lot of code is generated, maybe 10% is human-written.
October 10th, 2005 at 7:41 am
[quote]This calculation makes no sense, a lot of code is generated, maybe 10% is human-written. [/quote]
My question then is, who wrote the generator?
At one time or another it all comes down to 100% human input.
October 11th, 2005 at 9:39 am
bert, Where did you get your information that only 10% of Vista is hand-written code?
October 11th, 2005 at 11:20 pm
This calculation assumes that the estimated 50+ million LOC are being written from scratch, which is not the case in Vista, as it will benefit from a large Windows code-base which has been worked on and debugged years after years.
To me, this calculation makes no sense.
Regards,
/CL