Quality Central Discussion Session
Spent an hour with Nick Hodges, John Kaster, Calvin Tang, Jeff Wilsbacher and Mike Devery discussing Quality Central use in CodeGear.
We discussed a variety of topics, John also gave a walkthrough of the reporting QC provides, and we took a look at several QC clients.
John also made a good point - QA should first be looking at the ‘Needs Attention’ reports in QC. I only saw 10 listed for Delphi, so really Quality Central was in better shape then we first realized. We also raised several feature requests and ideas to help Quality Central self-clean, such as if a Reported issues hasn’t been commented, voted or raised in a certain period of time (such as two years) the issue could be set to a type of ‘Not Active’. And we looked at the voting system, seperating feature requests from defects, and John pointed me to Jeremy North’s client to use for searching/processing through QC reports: http://www.jed-software.com/qc_download.htm
Then we went through some of the reports that are available. QC reports show Sysop stats, user stats, projects stats, and a breakdown of the ’state’ of features areas in Quality Central.
June 26th, 2007 at 2:20 am
> I only saw 10 listed for Delphi, so really Quality Central was in better shape then we first realized.
I think that’s wishful thinking. A report will only get flagged as "needs attention" when a sysop has evaluated it and noted that it "something" should be done with it (opened, closed, whatever). If a report is in the "reported" state but does not "need attention", that most usually means it’s simply not been looked at in any detail.
- Roddy
(a somewhat "lapsed" L1 sysop)
June 27th, 2007 at 1:12 am
do i understand correctly? you want to do nothing with a report for two years and then hide it?
how about " if a Reported issues hasn’t been commented, voted or raised in a certain period of time (such as two years) " then you open-source Delphi and let us fix it ourselves.
June 27th, 2007 at 7:06 am
Chris - I suggest you spend time with David Dean, and get his feedback on what was involved in cleaning up the C++ Builder area. Ignoring reports that are in the "reported" state is NOT an option.
When I first reported a bug via QC, the process was dispiriting. I entered the bug, added workarounds, steps, etc. And *nothing* happened. Nobody opened it, nobody seemed to give a damn. It stayed "reported". For months. It made me wonder why I bothered, and I’m unsure about what made me persist with entering reports despite this.
Each of those 3,900+ "reported" issues is saying "we don’t care" to the person who took the time to report the problem: You have to find a better way of handling this.
- Roddy
June 27th, 2007 at 8:50 am
David and I carpool every morning. We talk about Quality Central a lot. He knows clearly the problem of dealing with both an internal bug tracking system and an internal one - and we’re ALL working to solve it.
Keep in mind the original Quality Central theme was the community itself would process/promote reports. So the community has not been keeping pace with all the reports. Recently I’ve been directing the QA team to get more involved in Quality Central. It would be unrealistic to say we could clean up 4000 overnight - we’re doing test development and run on several projects/updates, and maintaining the internal bug tracking system at the same time.
That being said - it *is* a problem I think we can solve. It’s likely in the future we’ll need to combine the two bug tracking systems to ensure faster line of communication between the community and R&D.
What I don’t want to do is slap the people in the face who spent dozens if not hundreds of hours to report and reproduce serious issues.
If that the case, and we have ~4000 reports to deal with, we need to find a solution to review them in as timely a manner as possible. Perhaps make a community contest out of it - those in the community who help us clean up the most, get some free licenses, product, recognition or such.
June 27th, 2007 at 1:33 pm
> If that the case, and we have ~4000 reports to deal with, we need to find a solution to review them in as timely a manner as possible.
Thanks for the encouraging comments - Here’s my positive suggestions:
a: *really* blow your trumpet about the C++ cleanup, and how many fixes have gone into BCB2007 as a result. That’s the proof of the pudding with QC as far as I’m concerned.
b: The real problem is the Delphi project is short of *active* L1 or greater sysops. Only they can "clean up" QC, so the ‘contest’ is unlikely to very successful. I think you need to actively recruit more community >L1 Delphi sysops, particularly in those esoteric product areas where existing sysops may have little knowledge (ECO, IOTA, translation manager, ec…).
Maybe you could even *pay* them - they’re doing Codegear’s work, IMO…
c: "Team B" could be more active in QC? Some are, some are not.
June 27th, 2007 at 3:18 pm
Good suggestions. Keep in mind the cleanup of QC ‘Reported’ is one thing- then the ‘Open’ is another (getting them fixed!). David Dean is/was working on the summary of C++ fixes, I believe we’ll see a developer network article soon.
b) Agree completely, especially on the point that the less known areas get less coverage. In the field tests, I started a ‘Field Test Marshal’ program which worked very well for Delphi. Less coverage on C++ as less folks in the Highlander field test are C++ centric (those that did help REALLY rocked, I am seriously amazed…)
Once I get a field test marshal in place for every newsgroup, that will help immensely. The compensation question is a great one too. I’d like to publicly recognize all the field test marshals, and/or have them part of Team B if not already. It’s a similar, if not more product specific, role. Don’t want to stomp on any of Team B’s toes either, but it seems logical to build up Team B.
c) Team B does a heck of a lot inside and outside of QC. It’s up to us/CodeGear to reward QC participation in general… It’s a similar statement to ‘R&D/QA needs to spend more time in QC’ - technically true, however if QA/R&D is currently working on a fix to a top rated QC report, that should be top priority to get in a build.
Overall, we need more bodies helping in QC. If QA is going to take over that responsibility from the community itself, I can think of several ways to make it happen- just using SurveyMonkey to set up a form where anyone can apply to be L1 SysOp, and I can review the applications, seems one of the simplest.
Tomorrow, we’re meeting again to discuss more about QC.
In any case, thanks for the comments.
June 28th, 2007 at 12:45 am
If you want clean and good reports, I suggest you make a new thin report client app. The QC app is more made for browsing than discover a problem and then report it. And that application should be the default bug reporter in the IDE (Menu item like "Enter bug/feature request"). The current one could still be there, but as a general QC browser.
How this QC should appear to the user:
Login (only first time as now)
Active project should be defined automatically by detecting what personality you are within.
Next is a search window that is optimized to search for keywords and browse through reports containing those keywords. This one should be as optimal as possible to see if your report already exist.
If you find one, you should be able to add comments.
If you don’t you should be able to create a new report.
This will then open a optimized new report window. Project should already be filled, so should platform and version. Only "Area", "Type", "Severity", "Description", "Steps", "Comments", "Workaround", "Attachments" should be available. And "Attachments" should also be possible to add before posting the report. Everything in a clean and nice optimized form.
Endnote:
Todays solution contain way to many things to enter. It’s stressing to look for reports. Query, New Query, Select project, Keywords, find rund current query button anyone.. This causes people to drop looking for reports before they send a new one.
Forcing people to do a search before they enter their report is the proper way.
And making sure they only have to enter the necessary information will give them a better feeling about reporting (and they might do it more often).
June 28th, 2007 at 1:41 pm
Part of what we’re discussing with John today is what’s different between RAID (our internal tool) and QC. For RAID, we do much of what you outlined - build number is pulled in dynamically (allows you to refer to a .exe), project and platform is set based on last entry, and then you just need to put in the custom information.
A search should be mandatory, it is for the QA team. Our own tool does not prompt it, it is a team guideline. That’s a good suggestion for public use of QC, especially if searching is made very easy (ask for keywords first, for example).
We do use duplicates in RAID a bit differently then QC where a duplicate is closed once discovered with a reference to original, not tied in to another report.
As we make QC better, we need to ensure we have the bandwidth to review and process the reports as they come in. Code samples in form of unit tests that the IDE could generate and attach to a report are one method. Or some IDE recording of last actions which include timing would dramatically help that issue.