The hidden testers…
At CodeGear, we often use Field Tests (essentially Beta tests) to do user acceptance testing of our products. We are extremely fortunate to have a constructive, dedicated and supportive group of customers with literally thousands of ideas and requests to improve our products. We always have a list of things we could do, the big trick then becomes picking the highest priority ones.
How do you become a field tester? In many cases, making a request via email or in the public forums, or logging a set of constructive and well detailed QualityCentral (QC) reports will result in an invitation. For example, I’m always happy to hear from long time users familiar with our products, and have serious issues making it hard for them to be successful! How do you contact me? chris.pattinson@codegear.com, and I don’t mind customers CC’ing to my private account of chris.pattinson@gmail.com in case the CodeGear spam filters are too aggressive (sometimes they are).
In the field test, we’ve assigned some individuals as Field Test Marshals. These are often some of our long term customers who are vocal, but constructive, and have a good aptitude to summarizing and prioritizing problems. If there are 100 issues/defects/bugs/feature requests- which are the most important to our customers? Having customers help answer those questions ensures the development team focuses on what’s important to the product. We’ve even gone so far as hiring several field test marshals as part time and full time staff in CodeGear. For those looking for a career in software development, it’s one possible place to start. One problem the field test marshals help solve is keeping track of QC reports and promoting the severe and reproducible ones to the internal database. This lets the test engineering team focus on automated test development, and ensure that when issues are fixed - they stay fixed.
We do periodic builds during a field test, and with those user surveys asking questions such as stability and performance, a matrix of features and their readiness to ship, and a section for user specific feedback where we encourage references to QualityCentral reports. This helps provide the basic information to determine readiness to ship, and trends of quality in the product. We can filter surveys on a build by build basis and it is simple to compare results.
Overall, the field test is a key part of our development process and I’d like to take this opportunity to thank everyone involved in making both the field tests and our products successful!