Quality Central November 2007 Revisit
Quality Central is one of the best methods to formally communicate with our customers.
One interesting statistic - 10472 reports have been pushed into RAID (internal bug tracking system) from QC since inception, and ‘Closed’. Closed can mean several things, so taking a slice of ‘Fixed’ and ‘Retest’ ones I see 5335. That seems pretty good.
On the other hand, we still have several thousand open and reported issues. And many of these have 0 votes, and not all product areas have QA or Sysops reviewing them on a periodic basis (since we’re all human, and folks tend to focus on the areas they are best at.)
Internally, we’re still continuing the plan to move our internal bug system into QC, so that the internal engineers can work with the one system and make the use of time much more efficient - especially when researching for similar or duplicated issues, or cleaning up ‘old’ areas that haven’t had much attention, or have been gathering reports on deprecated technology.
Of the Open issues in Quality Central, we’re looking to clean up as much as possible during the next year. This will take time, on average it takes 5 minutes of a bug council’s time to review one report. 3077 reports are ‘Open’ from Quality Central under the Delphi project (this includes internal field test reports). 2347 are ‘unique’ reports with the others being duplicates, and 511 of those are Feature Requests.
The numbers may seem big, but in the context that over 10 000 have been addressed and 2358 of those since 1/1/2006 - it seems doable.
So then it comes to priorities- we want to tackle the best ‘bang for the buck’ issues. I’ve proposed in borland.public.bdn.qualitycentral some changes to maximum voting and weighing, to encourage more Sysop participation. We still have a large backlog of Reported, though we’re are finally making slow progress knocking that down.
Some interesting ideas and points have been raised. The goal is to make it clearer to the development team which issues are being run into by several people. I see a focus too much on the voting of one individual, which is fine if weighing is given slightly to an expert, but I personally believe it is critical to determine the exposure of a problem, to help prioritize it. Now, that also means QC itself has to be friendly and usable by the community - plus have value and a feeling that time spent voting and putting in detailed reports is well worth the effort.
So for those visiting today, leave a comment on what you like/dislike/want/need/hate/love about Quality Central and what CodeGear behaviors you’d like to see demonstrated.
I’ll start with some examples:
*Feedback on reports from CodeGear in a timely manner
*Action on serious reports in form of patches or hotfixes in a timely manner
*Action on top voted feature requests
*Dislike - seperation of internal and external bug tracking systems
November 13th, 2007 at 11:02 am
Chris, I think the examples you gave are perfect starting points.
November 13th, 2007 at 11:44 am
What I’d like to see is some kind of feedback when a bug has been "accepted for fixing". This means that the team has decided to fix the bug (or has already fixed the bug). Then any votes on that report could be used for something else. This is the status that comes after "open", but before CodeGear wants to (or can) admit that a bug has been fixed indeed.
Sometimes it appears that the comments do not make it into the internal bug tracking systems. This is one of the things that should be improved. Why should I waste my time on comments if the engineers don’t even see them?
Hotfixes for serious reports (like the context-menu bug #QC52099 http://qc.codegear.com/wc/qcmain.aspx?d=52099 ) would be very appreciated. Maybe you could even release a hotfix as public beta?
November 13th, 2007 at 12:47 pm
What would be the best option to add a "be careful" flag to a report if the proposed solution will break existing code?
For example QC53738 has been opened. But if it gets implemented it will break existing code, because it is impossible to derive from a record list enumerator.
http://qc.codegear.com/wc/qcmain.aspx?d=53738
November 13th, 2007 at 1:46 pm
Chris,
The very thing I hate about QC is itself!!! It’s old technology, and there exist more advanced systems which are widely used.
For example, the leader in this respect, IMHO is JIRA (you can see it in action in tracker.firebirdsql.org)
Aside from it’s superb user management, it gives you RSS feeds on your filters, let’s you define roadmaps on issues, gives you automatic release notes & ….
That aside, I’d like to see more activity on CodeGear’s side on issues, for example stats on monthly progress, ….
VCL has not changed for a long long time, and as one of the people who has heavily invested my life and business on Delphi, I’m getting scared. It was less than 2 months ago where some of your less experienced engineers bloged about the VCL 2 man team. 2 men on that monster.
VCL were on the verge of business intelligence developement in 2000, where it were left off.
Also, Borland/CodeGear do not choose it’s included third party components lately.
The last one being Rave, when it is not superior to the FastReports and ReportBuilder.
I’d love QC to evolve as the whole-in-one interface of customer interaction with CodeGear, where engineers interact with reporters.
I’m sorry I’m not a native english speaker, so I can’t make my points in short descriptive and sane(!) statements
November 13th, 2007 at 4:27 pm
One of the things I dislike is this sysop rule: "Take an oldest first approach when possible. The report posted 3 months ago needs your attention a lot more than the one posted 3 hours ago. This will increase overall QC user satisfaction."
This rule would work if the oldest report was 3-4 weeks old, but in the current situation where you have reports that are way way older the rule makes QC seem unresponsive to those that are adding reports today because noone will look at them until 6 months from now.
Change the rule so you work on the "stack" from both ends. This will make the system more responsive and give those who post and/or vote an incentive to keep the dialog going.
November 13th, 2007 at 8:21 pm
If you want to widen QC’s reach, someone needs to pay some attention to basic usability in the QC client. Don’t hijack basic editing keystrokes like Ctrl+Home and Ctrl+End in the default keybindings. Make searching easier. (It always takes me a couple of tries to figure out those bizarre toolbar buttons for searching, and there needs to be a keystroke like Enter or Ctrl+Enter for "okay, I’m done entering search criteria, go search now". It would also be great if the searching interface was vastly simplified — get a GMail account and look at their search interface, and take some cues from it.) Give some explanation of what a blue report number means. Add some spit and polish.
I’ll definitely second Sebastian’s suggestion of saying *something* about an item’s status, especially for huge and far-reaching bugs like the TEdit context menu. As it is, once a report has been entered into RAID, that doesn’t mean anyone is actually working on it, or ever will.
It would also be nice if the lists of fixes that get posted on CDN tied back to QC somehow. Last time I looked at one of those lists, half of the reports that were listed as "fixed" on CDN were still listed as "Open" in QC. Many others, it wasn’t clear what the fix was. And a huge number of reports referenced in the list didn’t even exist!
November 14th, 2007 at 6:54 am
When I find a report of a problem that I have already encountered too, I would like there to be an easy way to report that fact. This would help CodeGear prioritize bug reports. I guess that this is how "votes" were intended to be used. However, it seems people like to use votes more for feature requests than bug reports.
Users should not be able to report that they have encountered a particular bug more than once but they should be able to report that they have encountered many particular bugs. I suppose, some people might try to claim falsely that they have encountered every bug. However, if they have to do that one bug at a time, I doubt that they will get very far.
By the way, how do I report a bug in this web page when viewed through BDS 2006? Every time I type an "H" I get the error message "No help found for context." (I got around that by typing my name and this comment in a text editor and then pasting in the web page.)
November 16th, 2007 at 12:45 am
Hi Chris, I see that you try to put some effort in QC. Very much appreciated. So, here are some small suggestions (besides the ones which I’ve posted in .qualitycentral newsgroup). (I will try to broke the themes in more posts, as you wish…
)
1. Response time problem. One of the main goals of any IT company (yours, ours) is to minimize the response time. How quick can you fulfill the customer’s requests. You must be focused on this. In fact, this is the main reason of existence of QC. No one from your customers (including me) will spend one of their most expensive resources (time) with QC if they doesn’t have pretty clear in their minds that their requests will be handled in a timely manner & in an appropriate way. Imho, you (CG) must clarify this first (see my other post in ng).
November 16th, 2007 at 12:52 am
2. About formalism. One of the hottest & most disputed in interminable threads in .non-tech ng was the Delphi’s cross-platform issue (IIRC, one of the latest threads on the theme ("Cross-Kylix discontinued") had more than 500 (five hundreds) messages. Can you give me a QC entry for this? Is it in Top 10? Top 50? Top 100? Why? Imho, one of the reasons is that the community lost their confidence in QC as a vehicle to carry their will/wishes to you (but about this I wrote a lot around here, also in the ng), but another reason is that the QC is _formal_ so you have only _one_ wing to fly to your perfect product. Customers always will prefer an informal way of communication (even if they spend more time on this) while a company will prefer the other, for obvious reasons. So, you must provide both of them discussing the ideas in a non-informal way and after this, crystallizing them in a formal one. Personally I use the .non-tech ng ("polluting" it when the others give me the opportunity) and, now, the Allen’s blog. Imho, this is the classical solution as other companies have it. In fact (almost) all OSS work in this way. A dedicated ng will fit perfectly for this, imho, but also a blog engine will do (like Allen does now), even if the latter has some drawbacks. Also the QC has the ‘Comments’ tab but it seems that this tab isn’t properly advertised by you and is hard to use (no quoting for ex.), so very little folks use it & know how to monitor the updates. Also, there exists the perception that you (ie. CG employees) doesn’t take part on that discussion, and, hence, ‘doesn’t really matter what’s happening there’. Generally speaking, the way in which is organized QC now doesn’t help at all in collaborative refinement of requests.
November 16th, 2007 at 1:19 am
3. The QC twins. Can you give a reason why do you have two bug-trackers? Except legacy, perhaps. I’m asking seriously. Because I suspect if a report is ’submitted internally’ any update (which means an enhancement, isn’t?) to that report isn’t taken in account by the internal team. Of course, I don’t know if this is a technical problem also or just a workflow one. This is, obviously, besides the classical problems with the parallel systems which you know.
November 16th, 2007 at 1:23 am
4. We don’t know what you do. You don’t know what we want. Users are very good in feeling their needs. But they are very dangerous in their solutions. Unfortunately, usually they write down their solutions (eg. "I need a thread-safe VCL") rather their needs ("I need to use all the cores of the CPU"). So you must have a dialogue with the users to see which was the initial impulse which generated the report. The sysops try to do this by posting comments but AFAIS from the eMail log which I receive, these are rarely responded. Why? You must find out. Possible reasons (_except_ all the above points):
- Bad design of QC website - very hard to find how to have (setup) an eMail notification of QC activity. (This is more general, imho. Perhaps sometime I’ll post you a ‘novel’ about how to effectively communicate a message to the others… :-))
- Report is old. The poster lost his hope and doesn’t track it anymore. Now it’s a Java/VS/Linux developer.
- "Oh, anyway they will fix in two-thee years, _if_ they will fix it. So why bother?" - it’s a variant of the above
- "OMG, now I must fight with the sysops. To explain, to try to convince… And all of this for what? No one assures me that someone will try to do my request" - another variant of above. Here something to note: at least in some period (winter of 2006 IIRC) the ‘price of innovation’ in QC was very high. In order to get a report promoted by sysops it must be at least stellar.
If the folks will ‘get’ clearly that their effort has an effect on the final product will change the things dramatically, imho. A possible status ‘Worked on’ would help a lot here. Btw, isn’t quite clearly what ’submitted internally’ really means… (I’m speaking now from ‘it will be handled/not handled properly in a timely manner’ POV)
November 16th, 2007 at 1:48 am
5. The QC is very granular. Look, imho, the data binding in VCL should be updated. This is also the opinion of many community members. So, if I post a QC entry with ‘Please update the VCL data binding’ do you think that I did something good? You’ll promote it? Imho, this must be discussed (a lot) somewhere in an ng and after this posted. See the above points about discussions.
November 16th, 2007 at 1:58 am
6. We are all programmers, remember?
You are in this happy position. You (and CG team) don’t have to deal with accountants, secretary men, bosses, HR & warehouse crowd and all the economic stuff with which some of us must deal and find a bridge to find their ever-changing requirements and try to understand from their i€ckonomik$ (yuck!) language what they meant. Your symbiotic life with your customers should be much smoother. Leverage this. Give them work to do. Give them QC entries to discuss as Allen does. Mind you, there exists a very well verified human law which says: Lack of work is mother of all evil. Personally, I’m fed up with the interminable threads discussing "How horrible CG are" ™, Win32 vs .NET, LindozeOSX etc. etc. again and again without doing nothing. More that 90% of us are programmers with (much) more than 5 years experience in Delphi (according with Chris’s (Bensen) blog). Can we become more effective? Mind you, it’s a management problem here… But courage. It’s doable. The guard dies but never surrenders.
Sorry for posting (again) a novel.
November 16th, 2007 at 8:58 am
Thanks Sebastian, the whole revenue recognition and SOX made the communication of intent so much more complex and cumbersome then I ever imagined! We cannot even discuss or communicate what bugs are on our ‘to fix’ list for upcoming patches - or even if such patches are in the works.
Since voting is so cumbersome, you will all see a change very soon - dramatically increased number of votes, and Sysops will have an increase to the number of votes they can put to an issue.
We’re trying to encourage more Sysops - QC was designed to be a community supported system, and we need plenty of Sysops to make that happen.
As to risk of fix - that’s taken into consideration by R&D during development of either major releases or of patches. The Delphi team puts a huge amount of effort into maintaining backwards compatibility. A flag such as ‘risky’ could be confusing unless used consistently.
Comments to and from QC - we’ll be taking care of that as part of merging QC and RAID, so that engineers work off the same system. I expect that will resolve that particular problem. It’s not productive to have to look at two systems to get all the current information.
November 16th, 2007 at 9:06 am
Hi Joe,
Ok in the future I’ll post fix lists referring directly to QC reports. These should sync up on the day of release when the version number of the patch is pushed public. I was learning about that system when QA decided to nose in more with QC, so now am a full admin.
We’re looking at the clients in detail too, the team is happy with the client we use internally, and plan to improve the QC one to bring it up to similar level of functionality. The searching in the internal tool is MUCH more powerful, and I agree intuitive colors and information are a must.
More on the client as things progress. I’m pushing for about a year worth of improvements and work in several steps.
November 16th, 2007 at 9:15 am
Hi Richard,
I agree about voting being problematic, we’ll be changing maximum number of votes. For the most part, developers look at QC as a repository for customer bug reports, and then product management looks at it for feature requests. We want to seperate the two purposes as much as possible, though some issues are ‘grey’ - major rework for performance improvements could be seen as either a feature or a fix
We are also considering the option of ‘I can reproduce this’ so that users can check a box, then a value is calculated, clearly showing how many users run into the issue. Personally, I think that is very useful especially to QA.
You can use the QC Windows client to report an issue, a link can be found here, where you must agree to the disclaimer :
http://qc.codegear.com/BCDownloadCGI.exe/disclaimer
November 16th, 2007 at 9:43 am
Hi m. Th,
Two key points come out of your discussions - information sharing, which is surprisingly difficult in the US now due to revenue recognition (my girlfriend who is a tax accountant is helping me understand the laws here), and then QC usage itself. QC started as a public self-support project ’skunkworks’ by developer relations, which we’re looking to enhance a great deal.
And not to discredit the hundreds of hours of work that resulted in over 10 000 issues reported in QC from being addressed! That’s a really good success story. Even with it’s weaknesses, QC is very usable and provides the team great information.
However, as it stands now - reports are coming in faster then CG internal staff can deal with them, and we need more active Sysops. Sysops will look into the areas they are interested, and the areas they are not will get neglected.
Voting help prioritize work, and limited votes focuses on the important issues - however for the serious but not very high profile issues, the small number of votes did not provide needed granularity.
I’ve taken samples of hundreds of QC reports in various areas, and a large number have 0 votes, a few have 1 to 5 votes, and a small number have over 5. Hence the directive to increase maximum number of votes to see if we can get more general feedback and create something closer to a bell curve.
As to solving the information problem - the basic ‘is codegear looking at it?’ - if it’s Open, it’s on our radar. Then it’s prioritized, which is the hard part right now with such limited voting feedback.
So recent changes will help to address those two major issues. We still can’t ever say ‘this list of bugs are what we’re planning to fix in an update’ - we do those discussions with our internal field test, which are under NDA.
Now, we’re fairly casual about who we let into our field test - anyone who wants to install the latest patch or build, which is usually a beta quality by the time we release to field test, can send me an email and ask to join. I’m usually fairly communicative and open with the internal field test. It’s much safer then worrying about SOX or revenue recognition by some inferred promise (as much as folks would like to hear it).