June 21, 2007

Pushing the Quality Central ‘clean up’

Filed under: Quality Central, Uncategorized — Chris Pattinson @ 2:43 pm

Hard numbers - a search this morning on Quality Central resulted in 3980 Reported issues for Delphi.  This should include ’everything’ - all versions, all locales, feature requests, the works.

What’s interesting is the C++ number - 144. David Dean (was an active field tester, now a member of my team), and Leo Siefert seem to be the main two keeping that product clean in QC. Special kudos to those two since that was a HUGE amount of work. It also proves the actions of a few can really matter.

Knowing the problem you have, is half the problem in solving it. I’ve directed the QA team to look at at least 5 a day, putting a quick note in weekly status reports their findings, and looking at ways to find the nastiest bugs we need to fix NOW, fast.

A few of the items today:

 1)

The IDE is sending Delphi AIR’s (Automated Incident Reports) into the Delphi.NET project. (QC 44970) - this absolutely has to get fixed.

2) 

From a QA engineer on my team came this email:

—–

I’ve just searched QC, and I see at LEAST 200 (I limited my search to 200)   automated incident reports with the phrase:

"Cannot change Visible in OnShow or OnHide"

the stack trace has : 

Forms.Forms.TCustomForm.CMShowingChanged

DeskForm.DeskForm.TDeesktopForm.CMShowingChanged

on top.  

—–

Good example of data mining to find a trend of an otherwise difficult to reproduce report.

We’ve passed this information on to R&D and will investigate to understand and reproduce as much as possible.

3)

Today I put a Post-It wall pad from 3M next to my office, to track daily the number of QC Reported bugs. A little visibility is a good thing. 

June 14, 2007

Delphi/C++ Team, Quality Central and Bug Prioritization

Filed under: Uncategorized — Chris Pattinson @ 6:27 pm

Quality Central is getting a lot of attention this week. I found out we had over *two thousand* QC reports in a ‘reported’ state. Yes, the system is expected to be maintained by Sysops and community members, but it’s starting to make CodeGear look like we do not care about our products. We really do :)

So I wrote various thoughts in this developer network article, as well as a behind the scenes look at our bug prioritization criteria.

One of the biggest battles being a quality assurance manager is balancing time for feature development against time for bug fixing. In Delphi 2007, we allocated a large amount of time for bug fixing, and many of these fixes were shared with the recent C++Builder 2007 IDE.

To be fair, with all this effort from the development team on bug fixing - some of the new features in the works mentioned on the Delphi and C++ roadmap were pushed back.

Now we’re looking at Quality Central with a critical eye. We’ve been using it effectively for field tests, but that typically focuses on the current version - not all previous versions. The reason is mainly about use of time - we have Field Test Marshal’s who help prioritize bugs in a feature area, promote QC reports to the internal system and communicate with the development team on the most serious issues. That has been helping us a lot, but it’s not reflected in the backlog of reports in QC from previous releases.

Just this evening we made changes to Quality Central, including moving the Automated Incident Reports to their own section so we can more easily track manually entered reports. AIRs usually do not have steps or description that helps QA, but it’s the trends we data mine in AIRs that let us find and fix several memory leaks and crashes in the past year. They enhance our testing efforts.

So this is a great time to talk to use about Quality Central. I’m scheduling several meetings over the next couple weeks, and directed my QA team to look at 5 QC reports a day. Let’s see how effective taking numerous small bites off leads to solving that huge amount of ‘neglected’ reports.

 

June 13, 2007

Delphi for PHP Update 2, Field Tester wanted

Filed under: Uncategorized — Chris Pattinson @ 11:12 am

We have an update for Delphi for PHP in the works, a more major update then the first one.

It’s ready for testing, and we’re looking for folks interested in testing this update to ensure it’s of good quality.

If you are interested, please fill out this simple form:

 
Thanks for helping us improve Delphi for PHP.
 
 

June 4, 2007

Major D2007 Update coming…

Filed under: Uncategorized — Chris Pattinson @ 10:02 am

A major update to Delphi 2007 is coming soon to an online update near you.

Over 300 fixes made to Delphi as part of the Cogswell release, which includes 102 reports from Quality Central. It’s been about three months, and we made several major improvements in Vista support (the forms.pas fixes we’ve had under test for quite some time) and major improvements in the documentation system (take a look at the index and try searching by using the new filters).

There have also been additional fixes to memory leaks and IDE stability. It’s a mixed bag, with some goodness for everyone.

This update is expected to be made available using the ‘Check for updates’ mechanism included as part of the new installer, as well as available for download online. More details to be announced soon.

The list of QC reports can be found at this developer network post: http://dn.codegear.com/article/36589

June 3, 2007

C++Builder 2007, Compiler changes

Filed under: Uncategorized — Chris Pattinson @ 9:32 am

In C++Builder 2007, we’re making changes which can be considered ‘breaking’ changes to certain previous projects. MSBuild has been introduced, so you may need to be careful on project imports (if you’ve manually edited your project - the importing will not be automatic in all cases), the compiler is stricter and we’re providing an updated and improved STL via Dinkumware. We also made significant progress in Boost compatibility - I’ll let David Dean discuss those results once we actually ship, since that’s his ‘baby’. 

 

The C++Builder 2007 IDE itself is looking really snappy (I’ll blog about some performance tests later), having included many of the upgrades provided in Delphi 2007, we want to help our C++ users move to using C++ Builder 2007, understanding in advance what changes we’ve been making and why.

C++Builder and C++ Compiler Notes

Possible Binary Incompatibility with Previous Version of C++ Compiler

The C++ runtime libraries have been enhanced and altered to be more current, and the standard C++ libraries from Dinkumware have been updated to version 5.01. Because of these changes, the binary objects generated by the compiler in C++Builder 2007 might differ in some cases from previously compiled versions. The incompatibilities are mainly restricted to the C++ standard libraries. There are no changes in the basic C++ application binary interface (ABI).

Delphi Interface Parameter with Default Value of nil Handled Differently
Assigning the Type void * to Delphi Interface Now Fails to Compile

The Delphi compiler (DCC32) previously generated the following code in the .hpp file when an interface parameter had the default value of nil:

  void methodName(_di_IIntf param = (void *)(0×0));

Previously the C++ compiler incorrectly accepted this syntax.

In this release, both the Delphi compiler and the C++ compiler handle this case differently.

The Delphi compiler now emits code like this in the .hpp file for an interface parameter with a nil default value:

  void methodName(_di_IIntf param = _di_IIntf());

The C++ compiler now gives an error for code that assigns the type "void *" to a Delphi interface. For instance, the line

  void methodName(_di_IIntf param = (void *)(0×0));

now fails to compile, emitting the error:

  Cannot convert ‘void *’ to ‘_di_IIntf’

If you have Delphi code containing an interface parameter with a nil default value, recompile it with DCC32. If you do not have the source, edit the .hpp file and modify all occurrences like this

  void methodName(_di_IIntf param = (void *)(0×0));

to this:

  void methodName(_di_IIntf param = _di_IIntf());

Selecting Both of the Symbol Reference Info Options for Delphi Code

If you are compiling Delphi code inside your C++Builder project, you currently can select only one of the Symbol Reference Info options (either Definitions or Reference Info) on the Project>Options>Delphi Compiler>Compiling dialog box. However, you can select both of these options if you use source-level directives ($xx) as follows:
  • Definition Info ($YD) option: the Delphi compiler records information about where each identifier is defined. For most identifiers–variables, constants, classes, and so forth–the compiler records the location of the declaration. For procedures, functions, and methods, the compiler records the location of the implementation. This enables Code editor browsing.
  • Reference Info ($Y+) option: the Delphi compiler records information about where every identifier is used as well as where it is defined. This enables the References page of the Project Browser.
These compiler options are only available when both the Debugging Information ($D) and the Local debug symbols ($L) options are ON.

Specifying WebSnap Directories

If you are using C++Builder with WebSnap, ensure that the executables are written to the same directory as the HTML files. On Project>Options>Paths and Defaults>Final Output, enter a dot (".") so that the executable is written to the project directory.

Location Change for Precompiled Headers

C++ precompiled header files (PCH) are placed in a different file location than BDS2006. If you import a project from BDS2006, it does not import the project’s PCH file location. You can set the PCH file in Project>Options>Precompiled headers>PCH filename.

Generating C++ Files from the Delphi Compiler

If you want to generate .hpp files (and the corresponding .obj) from a .pas file, you should use dcc32.exe with the -JPHNE switch. Or use -JL on the .dpk file containing the .pas file.

Using WinHelp in C++ Applications

To use WinHelp in a C++ application:

  1. Add #include <WinHelpViewer.hpp>.
  2. Refer to an object declared in the WinHelpViewer header file.

    void LinkWinHelp()

    {

      # pragma startup LinkWinHelp 66

      if (WinHelpTester != NULL)

            WinHelpTester->GetHelpPath();

    }

To download the WinHelp viewer, visit the Microsoft website:

http://www.microsoft.com/downloads/details.aspx?familyid=6ebcfad9-d3f5-4365-8070-334cd175d4bb.

C++ Compiler Changes

To more closely obey the rules of the C++ ANSI Standard, the C++ compiler shipping with C++Builder 2007 is stricter than previous versions. Code that did not generate errors in earlier versions of C++Builder might fail to compile with C++Builder 2007. This section lists some of the common areas where the compiler is stricter. Each case is illustrated with an example showing the problem and how to update the code to compile with C++Builder 2007. Note that there are often many ways to bring offending code up to date. The appropriate method depends on the intent of the original code.

  1. Binding of References and Qualifiers

    There are many constructs that now generate error messages with the C++ compiler included with C++Builder 2007. The rules governing this are described in section 8.5.3 of the 2003 C++ ANSII standard. They fall in the following categories:

    1. Binding a non-const lvalue to to non-const reference. Use the compiler switch -Vbr to allow this.
    2. Binding a temporary to a non-const reference. Use the compiler switch -Vbr to allow this.
    3. Binding of const or volatile objects to non-const or non-volatile methods repectively. Use the compiler switch -Vbn to allow this.

    Previous versions of the Borland C++ compilers allowed various forms of binding to non-const reference parameters. In the following example, for instance, one was allowed to simply cast the psize parameter:

      int takesLongRef(long& l);

      int takesUnsignedPtr(unsigned long* psize) {

         return takesLongRef((long)*psize);
      }

    With C++Builder2007, the above code generates these errors:

      Error E2357 test.cpp 3: Reference initialized with ‘long’, needs lvalue of type ‘long’ in function takesUnsignedPtr(unsigned long *)

      Error E2342 test.cpp 3: Type mismatch in parameter ‘l’ (wanted ‘long &’, got ‘long’) in function takesUnsignedPtr(unsigned long *)

    To remedy this, you can cast psize before dereferencing, as in:

      int takesLongRef(long& l);

      int takesUnsignedPtr(unsigned long* psize) {
        return takesLongRef(*reinterpret_cast<long*>(psize));
      }

    Be aware of cases that involve temporaries in unobvious ways. For example, some binary operators imply a temporary:

      enum { zero, one, two } num;

      num |= two; // Not allowed
      num = num | two; // OK

    Another case that involves temporaries in an unobvious way is the return value of a property. The following example illustrates code that compiled with previous versions of the compiler:

      #include <vcl.h>

      class TTest {
        WideString FData ;
      public:
        __property WideString Data = {read = FData };
      };

      void Func(WideString& wref);

      void test() {
        TTest t;
        Func(t.Data);
      }

    With C++Builder 2007, the above generates two errors:

      Error E2357 test.cpp 14: Reference initialized with ‘const WideString’, needs lvalue of type ‘WideString’ in function test()
      Error E2342 test.cpp 14: Type mismatch in parameter ‘wref’ (wanted ‘WideString &’, got ‘WideString’) in function test()

    You can fix this by changing the reference to a const reference, as in:

      void Func(const WideString& wref);

    Here is an example of trying to bind a const object to a non-const method:

      struct X {
        void foo();
      };
      const X x;
      x.foo(); //error

  2. 1.1 Reconcile Error Dialog

    The temporaries and references issue referred to above is encountered in code generated by previous versions of the Reconcile Error Dialog Wizard. To remedy this, look for the VarToAnsiStr method

      AnsiString VarToAnsiStr (Variant &V, TFieldType DataType)

    and change it to take a const Variant&, as in:

      AnsiString VarToAnsiStr (const Variant &V, TFieldType DataType)

  3. String literals are now constants

    String literals are now considered to be of type ‘const char[]‘ by default. This, combined with the stricter qualification binding of const values and types, can generate error messages in code that compiled before.

    You may enable the -Vbs switch to revert string literals to non-const. However, CodeGear recommends that you update the code instead.

    Note that the change in the type of string literals can also change how the compiler resolves calls to overloaded methods. The following example illustrates this:

      void foo(char *);
      void foo(const char *);
      foo("string"); // New Compiler picks foo(const char *)

  4. Template Changes

    The C++ compiler no longer allows an explicit template without the ‘template <>’ prefix. Use the compiler switch -Vbe to allow this. The following example shows this:

      template<class>
      class foo {
        foo();
      };
      foo<int>::foo(); // Error
      template<> foo<int>::foo(); // OK

    Also, the C++ compiler no longer allows explicit template specialization within a class. Use the compiler switch -Vbx to allow this. For example, this generates an error:

      struct S {};
      struct SP
      {
        template <typename T> void foo(const T &) {}
        template <> void foo(const S &) {} // Error
      };

      template <> void SP::foo(const S &) {} //OK
  5. Function Overload Resolution

    One of the areas where the C++Builder 2007 compiler differs the most from the previous version is in overload resolution, which includes the detection of ambiguity. The compiler now better conforms to the rules in section 13.3 of the 2003 C++ ANSI Standard. Several constructs that were previously allowed might now be reported as ambiguous or no match found, requiring that you modify code to clarify its intent.

    The compiler option to revert to the old behavior, not enforcing the new stricter behavior, is -Vbo. However, not all compiler changes can be controlled by this switch, so CodeGear recommends that you update the code instead.

    The following is an example of an ambiguity that was permitted by the previous compiler:

      class X{};
      void foo(X);
      void foo(const X&);
      void ambig() {
        X x;
        foo(x); //error-ambiguous-the previous compiler chose ‘void foo(x)’
      }

  6. 4.1 std::abs() Ambiguity

    The standard abs function might also generate an ambiguity message when invoked with a parameter that does not exactly match the types expected by the various overloaded versions of abs. Here is an example:

      #include <limits>
      bool test(long l) {
        return std::abs(l) > 0;
      }

    The code above generates an error and a warning:

      Error E2015 test.cpp 5: Ambiguity between ’std::abs(int) at C:\devtpscincludemath.h:208′ and ’std::abs(long double) at C:\devtpscincludemath.h:275′ in function test(long)
      Warning W8057 test.cpp 6: Parameter ‘l’ is never used in function test(long)

    To fix this, cast to the type of the overload you wish to invoke. For example,

      #include <limits>
      bool test(long l) {
        return std::abs(static_cast<int>(l)) > 0;
      }

  7. Initialization and Conversion

    The compiler now obeys the rules of 8.5.1 and 13.3.1 of the 2003 C++ ANSI Standard for initialization and conversion:

    1. Direct initialization now requires initialization by a constructor and no longer picks a user conversion sequence.
    2. Copy initialization for objects of the same or derived type now requires a constructor call.
    3. Copy initialization for objects of the different types no longer prefers user conversion over construction. If the compiler finds a suitable user conversion, it now continues to look for (possibly ambiguous) converting constructors. If the chosen conversion function is a converting constructor, the call initializes a temporary of the destination type. The result of the call (which is the temporary for the constructor case) is then used to directinitialize the object. Use the compiler switch -Vbo to revert to the previous behavior.
    4. For an explicit cast, the compiler now performs direct initialization on a temporary.

    This example illustrates the new behavior:

      // In this example, dst is destination type and src is source type
      class A { };
      class V {
      public:
        V() { };
        V( const V & ) { }
        V( const A & ) { }
      };
      class G {
      public:
        G() { }
        operator V() { }
        operator A() { }
      };
      G g; V v;
      // direct initialization
      // ==> constructors are considered.
      V v9(g);
      // Both of these statements previously compiled but now get the error:
      // Error E2015: Ambiguity between ‘V::V(const V &)’ and ‘V::V(const A &)’

      // casts
      // (V)g is treated as V tmp(g) which is direct initialization of ‘tmp’
      // ==> constructors are considered.
      (V)g;
      static_cast<V>(g);
      // Both of these statements previously compiled but now get the error:
      // Error E2015: Ambiguity between ‘V::V(const V &)’ and ‘V::V(const A &)’

      // copy initialization with dst=V src=G
      // ==> user-defined conversion sequences are considered.
      V v4 = g;
      V v5 = G();
      // Both of these statements now compile but previously got the error:
      // Error E2015: Ambiguity between ‘V::V(const A &)’ and ‘V::V(const V &)’

      // copy initialization with dst=V src=V
      // ==> converting constructors of V are considered.
      V v6 = (V)g;
      V v7 = V(g);
      // Both of these statements compiled previously but now get the error:
      // Error E2015: Ambiguity between ‘V::V(const V &)’ and ‘V::V(const A &)’

  8. 5.1 Conversion via user-defined operators

    The new C++Builder 2007 compiler often reports ambiguities for conversions that involve user-defined operators. An example is shown below:

      class AnsiString
      {
      public:
        bool operator ==(const AnsiString& other);
        AnsiString(const wchar_t* src);
      };

      class Variant
      {
      public:
        operator AnsiString() const;/br>     operator wchar_t*() const;
        bool operator ==(const AnsiString& rhs) const
        { return static_cast<AnsiString>(*this) == rhs;}
      };

    C++Builder users might notice that the above is a stripped down version of the VCL AnsiString and Variant classes. Previous versions of the compiler invoked the ‘Variant’ ‘operator AnsiString() const’ for ’static_cast<AnsiString>(*this)’, while C++Builder 2007 uses ‘conversion via constructor’. Since the Variant can be converted to multiple types for which there are AnsiString constructors, the compiler generates an ambiguity error.

    To correct this error, you must eliminate the cast as in:

      bool operator ==(const AnsiString& rhs) const
      { return (*this) == rhs;}

    You can also be explicit about the operator:

      bool operator ==(const AnsiString& rhs) const
      { return this->operator AnsiString() == rhs; }

    5.2 Variant/OleVariant/AnsiString/WideString/TDateTime

    The issue described above with a user-defined conversion operator vs. conversion via constructor might be encountered in several constructs involving the VCL classes Variant, OleVariant, AnsiString, WideString, TDateTime, Currency, and so forth. The following table lists constructs that now generate error messages and the updated syntax.

    Previous Construct Updated Construct Notes
    AnsiString test(OleVariant v) {
      AnsiString ret = (AnsiString) v;
      return ret;
    }
    AnsiString test(OleVariant v) {
      AnsiString ret = /*(AnsiString)*/ v;
      return ret;
    }
    Do not cast RHS when relying on conversion operator in an assignment.
    WideString test(OleVariant v) {
      WideString w(v);
      return w;
    }
    WideString test(OleVariant v) {
      WideString w = v;
      return w;
    }
    Use Copy Initialization instead of the more direct constructor.

    The underlying compiler change for the errors described above is related to the way the compiler now handles initialization and conversion.

May 22, 2007

C++ Builder 2007 and Delphi 2007

Filed under: Uncategorized — Chris Pattinson @ 3:56 pm

C++ Builder 2007 is coming soon, and we’ve done a few new things on the development side that help us release to market much quicker, and focus our testing efforts.

One of those things is ‘One install image to rule them all’. The DVD media that will ship with C++ Builder 2007 contains install information for all languages - and includes an update to Delphi 2007.

Now that’s a great thing, since a Delphi 2007 user who installs the new C++ Builder 2007 will automatically update Delphi 2007 files. This includes 1369 fixes to date for C++ (299 of those from Quality Central), and 285 on the Delphi side (92 from QC) so far. We have more in the works too planned fixed before release, though we’re locking down to keep things as stable as possible now.

A very important point that we’ll need to communicate to ‘the world’ : for users who own both C++ Builder 2007 and Delphi 2007, they MUST use the C++ Builder 2007 DVD for any future repairs, installs or uninstalls. This includes Delphi 2007 re-installs.

So when you do buy C++ Builder 2007, and you already have Delphi 2007, and want to reinstall or modify the features for Delphi 2007 - just put in the C++ Builder 2007 DVD (or run setup.exe from the cached files copied over during install, or use the shortcuts on the start menu - just DO NOT use the Delphi 2007 DVD).

Speaking of Install, we added a few nice options to the installer to allow you the option to keep the cached files on hard drive or remove them. Now, we do actually recommend keeping the 1.5GB of files since future upgrades and repairs will be possible without original DVD media.

We plan to post a Developer Network article explaining how the installer works with multiple products installed, how to install those products, limitations and general information.

April 10, 2007

The hidden testers…

Filed under: Uncategorized — Chris Pattinson @ 1:13 pm

At CodeGear, we often use Field Tests (essentially Beta tests) to do user acceptance testing of our products. We are extremely fortunate to have a constructive, dedicated and supportive group of customers with literally thousands of ideas and requests to improve our products. We always have a list of things we could do, the big trick then becomes picking the highest priority ones.

How do you become a field tester? In many cases, making a request via email or in the public forums, or logging a set of constructive and well detailed QualityCentral (QC) reports will result in an invitation. For example, I’m always happy to hear from long time users familiar with our products, and have serious issues making it hard for them to be successful! How do you contact me? chris.pattinson@codegear.com, and I don’t mind customers CC’ing to my private account of chris.pattinson@gmail.com in case the CodeGear spam filters are too aggressive (sometimes they are).

In the field test, we’ve assigned some individuals as Field Test Marshals. These are often some of our long term customers who are vocal, but constructive, and have a good aptitude to summarizing and prioritizing problems. If there are 100 issues/defects/bugs/feature requests- which are the most important to our customers? Having customers help answer those questions ensures the development team focuses on what’s important to the product. We’ve even gone so far as hiring several field test marshals as part time and full time staff in CodeGear. For those looking for a career in software development, it’s one possible place to start. One problem the field test marshals help solve is keeping track of QC reports and promoting the severe and reproducible ones to the internal database. This lets the test engineering team focus on automated test development, and ensure that when issues are fixed - they stay fixed.

We do periodic builds during a field test, and with those user surveys asking questions such as stability and performance, a matrix of features and their readiness to ship, and a section for user specific feedback where we encourage references to QualityCentral reports. This helps provide the basic information to determine readiness to ship, and trends of quality in the product. We can filter surveys on a build by build basis and it is simple to compare results.

Overall, the field test is a key part of our development process and I’d like to take this opportunity to thank everyone involved in making both the field tests and our products successful!

 

March 18, 2007

Delphi 2007 - Under the hood of Quality Assurance

Filed under: Uncategorized — Chris Pattinson @ 3:28 pm

Now that Delphi 2007, codenamed ‘Spacely’ has been released, I’d like to share what happened in the Quality Assurance team during the Spacely project cycle.

It all started on a dark and stormy night in February, in a small town called Scotts Valley…

Ok more seriously, the CodeGear crew wanted to do something special and different for our next release. During the transition from Borland to CodeGear, we had done a fairly detailed (long!) survey for our existing Delphi customers to better understand their needs, wants and pain points. From the feedback of this survey it was clear we had a feature full IDE, and the main issues were related to performance and stability. A feature is only solid and ‘good’ if the IDE can stay up and let you use it reliably! The result of this was the theme for our next project, Highlander at the time, was stability and performance. Do as small a set of critical features as possible, and spend significant time and make noticeable improvements to stability and performance in the IDE. To me, noticeable meant ’statistically significant’.

My teams challenge was to scientifically validate the statement “our next Delphi release is more stable and performs better then previous Delphi releases“. We went so far as to hire a QA engineer who had the specific task of performance and stability testing. Looking back, that was a smart move since we now have a set of suites that run against previous and current builds, and monitor such metrics as IDE start and shutdown time, compile time, project open and close time, build and run time, desktop switching time and many other metrics in a very scientific fashion.

Backing up a step - we also made a resolution to fix our test system. By that, I mean the QA team had been switching platforms and frameworks as a result of developing Kylix, then Delphi for .NET, and that had played havoc with our automation tooling. We use a flexible and powerful automation framework affectionately called ‘Zombie’ to drive our IDE automation. (It’s not all we use - we also have various command line based automation tools and frameworks, but the IDE has always been the most challenging to automate). This year we created a special team called the Automation Mini-Team which included QA engineers, R&D engineers, managers and integration engineer to dramatically improve our tooling. The results really paid off - every build we release to QA is automatically tested by a suite of IDE and command line tests, which then report to a test report server summarizing current run vs last run, and provides a graphical chart of test run time and test results. It’s slick, but not only that it works. Almost too well - this next year, we’ll be looking to improve to the next level : automatic email notification when results deviate statistically. For example, the QA and R&D engineer could have email notification if a test that used to run 95% success suddenly performed at 90%, or took twice as long to run, or any other metric we wanted to monitor such as memory use. Right now, we have too many results to quickly data mine so we need to automate that review of test results.

One particular R&D engineer, Steve Trefethen, spent a couple months helping to dramatically improve the Zombie framework. He went so far as to creating tools and a process to automatically generate models required by the automation on a per build basis. This meant the manual intervention required to maintain our tests went down a VERY large amount, so more effort could be put on writing and improving tests then on the library of test models! So, a big thanks goes out to Steve for that help. (And for those that don’t know - Steve used to be a QA engineer).

We hit the quality and testing problem from a number of angles- besides improvements to the test automation system, we also had field test surveys to identify pain points on the fly and monitor the IDE ‘feel’. We also nominated some outstanding individuals as ‘Field Test Marshals’. These folks  co-ordinated Quality Central reports and feedback and summarized into ‘Top Ten’ lists that the R&D team referred to. To be fair, not all Top Ten issues could be addressed in the Spacely timeframe, since some required changes that would invalidate the ‘no breaking changes’ mantra. But we have those lists now and can focus on them for the breaking release later in the year, and we did fix a number of the issues for Spacely.

I’ll also give a huge thumbs up to VMWare - VMWare is used in our test automation system extensively, and helps to isolate changes especially in terms of performance and stability. You can always revert to a known state using VMWare, and do it in a few minutes. In addition, we could test rapidly against a number of different operating systems, configurations and languages. We extended our test system by a number of VMWare ESX servers - and they became so desirable and useful to testing, they hit maximum capacity after only a few weeks. I expect we’ll be after additional servers in the future - especially as we add additional automated tests.

So lots of great things happened in CodeGear Quality Assurance for Delphi 2007. And more still planned!

February 21, 2007

Delphi 2007 - Hands on Testing and

Filed under: Uncategorized — Chris Pattinson @ 1:11 pm

Now that we’ve announced our Q1 Delphi 2007 release, I’d like to share a story of a couple users we had come in with their real world application.

 

A little bit of background- as we close in on a new release, we often bring in folks we call ‘Beta Bangers’. These are developers with a long history of using our products, with fairly large projects, and help us to do customer acceptance testing. Typically it involves them driving the latest build of the product, with their own source code, and a number of engineers and managers in the room. That’s how it went this time.

 

The project through brought in was just shy of 400 000 lines of code. It was a GUI application to manage manufacturing equipment, and they had a simulator to help test their application. The test system, a laptop, had Windows XP with 1.5GB of memory, and only a few GB of free hard drive space. It also had BDS 2006 installed on it.

 

We had to uninstall a previous beta build and clean up the registry of entries in HK_CurrentUser to begin testing. For Delphi 2007, we are working to do some package renaming to ensure it resides well with BDS 2006.

 

The install went smoothly, and they chatted a bit about their business. It took about 20 minutes to install.

 

Then we opened up their major project, it was based off of code from Delphi 5 and upgraded to BDS2006. We did not have the latest build of Rave reports available, so spent some time debugging and removing the reporting module – a good start to testing, since error insight was working perfectly. They were quite impressed, since this was one of the items on their ‘issues list’.

 

To back up a second - we ask our beta bangers to bring a list of issues with them, so we can reproduce on the new build. They had a list of error messages, odd behavior and performance that we went through.

 

Overall feedback was really good – error insight which used to give many false positives on a major part of their source displayed correctly with no false errors, compile and build numerous times was snappy (they commented it ‘felt’ snappy, and R&D indicated as a result of optimizations it should be 25% to 30% faster), and other comments were made that we’d identified and fixed race conditions involved in code insight.

 

We imported a custom control package from BDS 2006 without issues, and then reopened it back in BDS2006. We tested the theme features and explained how the IDE is themed, and built applications may or may not be themed. It’s something the team is still looking into – basically the form designer right now displays the application as if were themed.

 

Old applications will not have runtime themes turned on (this would be intrusive), but with the IDE themed, this means run time can look different then design time. There has been a simple option added to the projection options, in the application settings, that will enabled runtime themes to be turned on or off.

 

We tested performance, which the beta bangers were really impressed at with startup speed, code insight improvements (they indicated used to take 3-5 seconds for things that now take l second) and memory management which is much more solid then BDS2006.

 

We took almost an hour to test out the debugger and debugging. No crashes, fast evaluations, and demonstrated the improvements to the debugger for call stack enhancements in the structure pane view. No evident memory leaks so we made good progress there.

  

A couple enhancements are enabling breakpoint in the call stack view, and putting a colored ball to indicate that debug information is available for that module.  And as usually you can double click on the call stack items to go to the relevant code.

 

We took note of a couple feature requests, including the ability to change a file to read/write if the IDE encountered it as read only – instead of having to leave the IDE.

 

Later we worked with the project groups and added/removed projects. We identified a potentially serious crash bug with the adding and removing of projects, then rebuilding. Exactly what we hoped to catch as a result of these sessions!

 

Overall, I considered this a highly productive meeting with some of our real customers. We got to show off some real improvements in Delphi 2007, and identify some areas to add some polish. We also identified a ‘good’ crash bug which we have reproducible steps for, and should nail it before release. And, we actually had fun while doing the work, and learned more about one of our long term customers.

 

 

 

November 15, 2006

CodeGear, and focus on Quality.

Filed under: Uncategorized — Chris Pattinson @ 1:37 pm

Quality. Quality. Quality!

I’d be happy to do a rant A-La-Steve Ballmer on quality, since it’s one of those difficult to prove and define - yet everyone KNOWS if a product is high quality or not. And word travels fast!

The team always wants to ship a high quality product. In the past, we’ve shipped too early after significant code changes, and with testing underway to determine the state of the final product. Hindsight is always 20/20, and Delphi 7 was ‘fairly solid’, where Delphi 8 was not. The community determines the quality of a product, though the internal QA team really should have a clear grasp on what we’re shipping. It’s especially tough when the QA team says the product is not ready, and then it ‘has’ to ship for business reasons, something I won’t go into too much detail - but anytime the development group (combination of QA and R&D) have not given joint thumbs up on a product release- it’s been tough times.

How does that happen, and how to stop it? In the past 6 years, I’ve watched an automated test system fall into various levels of disfunction. There were ‘reasons’ behind this - work on the Linux platform meant existing Windows based testing would not operate on Linux, and new frameworks such as .NET which meant our testing tools had to be greatly extended to connect to and drive .NET based controls. This is exactly what occured after Delphi 7 into Delphi 8, and meant that by Delphi 8 and BDS 2005, we had far less automated testing coverage then in the past. And the results were dramatic - the product quality of the shipping product, was MUCH less then expected.

The good news is, the team is on the path to recovery. And making great progress. We’re close (some would say well past since we include .NET and C++ - but I’ll be conservative) to Delphi 7 levels of automated testing, and even working on having our tests run on multiple OS, and in multiple cultures (English, Japanese, French, German, etc… ). We also have considerable more testing on the check-in and build side, where we run CruiseControl that builds and runs a number of basic acceptance tests as part of the build process. Then after a build passes those tests, it’s sent to integration to make a ‘customer’ build, which additional tests are run by the QA group - first a smoke test, then additional functional tests. And all of this is completely automated.

I’d like to give kudos to VMWare for having great technology to helping make this happen - we use a mix of VMWare workstation, ESX and GSX servers to drive this testing system. A side benefit is after tests are run, a clean virtual machine is available for QA in the morning, so we can start testing immediately on the latest build - AND already have a set of visible results from tests to compare to.

What does this mean for our customers?

The short version is - product quality will not stray far from ‘good’ as in the past. We know VERY rapidly if a new failure is introduced, and we can fix it almost immediately since we have a record of that exact check-in. Late found defects make for unstable products and code-base. Also, defects found early means code is fresh in developments mind and they can rapidly change/tweak code as well as catch it before any new dependancies are introduced.

Testing early, and often, is the right thing to do. And having repeatable, visible and auditable testing plus a revamped test harness can only result in a win/win for both the development team, and our customers. It’s been a true team effort to get things on track for Highlander, and we’re still mid-project. I’m really quite excited to see how the end result will come out!

Go CodeGear! Go Team!

« Previous PageNext Page »