CodeGear, and focus on Quality.
Quality. Quality. Quality!
I’d be happy to do a rant A-La-Steve Ballmer on quality, since it’s one of those difficult to prove and define - yet everyone KNOWS if a product is high quality or not. And word travels fast!
The team always wants to ship a high quality product. In the past, we’ve shipped too early after significant code changes, and with testing underway to determine the state of the final product. Hindsight is always 20/20, and Delphi 7 was ‘fairly solid’, where Delphi 8 was not. The community determines the quality of a product, though the internal QA team really should have a clear grasp on what we’re shipping. It’s especially tough when the QA team says the product is not ready, and then it ‘has’ to ship for business reasons, something I won’t go into too much detail - but anytime the development group (combination of QA and R&D) have not given joint thumbs up on a product release- it’s been tough times.
How does that happen, and how to stop it? In the past 6 years, I’ve watched an automated test system fall into various levels of disfunction. There were ‘reasons’ behind this - work on the Linux platform meant existing Windows based testing would not operate on Linux, and new frameworks such as .NET which meant our testing tools had to be greatly extended to connect to and drive .NET based controls. This is exactly what occured after Delphi 7 into Delphi 8, and meant that by Delphi 8 and BDS 2005, we had far less automated testing coverage then in the past. And the results were dramatic - the product quality of the shipping product, was MUCH less then expected.
The good news is, the team is on the path to recovery. And making great progress. We’re close (some would say well past since we include .NET and C++ - but I’ll be conservative) to Delphi 7 levels of automated testing, and even working on having our tests run on multiple OS, and in multiple cultures (English, Japanese, French, German, etc… ). We also have considerable more testing on the check-in and build side, where we run CruiseControl that builds and runs a number of basic acceptance tests as part of the build process. Then after a build passes those tests, it’s sent to integration to make a ‘customer’ build, which additional tests are run by the QA group - first a smoke test, then additional functional tests. And all of this is completely automated.
I’d like to give kudos to VMWare for having great technology to helping make this happen - we use a mix of VMWare workstation, ESX and GSX servers to drive this testing system. A side benefit is after tests are run, a clean virtual machine is available for QA in the morning, so we can start testing immediately on the latest build - AND already have a set of visible results from tests to compare to.
What does this mean for our customers?
The short version is - product quality will not stray far from ‘good’ as in the past. We know VERY rapidly if a new failure is introduced, and we can fix it almost immediately since we have a record of that exact check-in. Late found defects make for unstable products and code-base. Also, defects found early means code is fresh in developments mind and they can rapidly change/tweak code as well as catch it before any new dependancies are introduced.
Testing early, and often, is the right thing to do. And having repeatable, visible and auditable testing plus a revamped test harness can only result in a win/win for both the development team, and our customers. It’s been a true team effort to get things on track for Highlander, and we’re still mid-project. I’m really quite excited to see how the end result will come out!
Go CodeGear! Go Team!
July 1st, 2007 at 12:32 am
Good opinion