Desktop QA - Bugzilla keywords: Proposal

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Desktop QA - Bugzilla keywords: Proposal

Robert Kaiser-2
tl;dr: I propose that desktop QA will move from using some whiteboard
tags to keywords for usual bug markings, most importantly "verifyme"
instead of [qa+] and "noverify" (NEW) instead of [qa-].



The activity on our discussion about "Desktop QA: Bugzilla
tags/keywords" has died down in the last weeks, and it doesn't look like
we did have a clear agreement there.
 From talks in meetings, it sounds like going with something that has
clearly defined states like keywords or flags is preferred by most people.

I thought a bit about that and have the following proposal based on that:

1) Let's use keywords for the core long-term repeating things.
2) Free-form comments or markers of temporary importance (for a single
project, like Australis, Sync, etc.) should go into the QA whiteboard.

The following keywords would be used:
a) "qawanted" for anything where investigation from QA is needed
b) "regressionwindow-wanted" where a regression window is needed
c) "steps-wanted" where we need steps to reproduce
d) "verifyme" for bugs that (will) need QA verification
e) "noverify" (NEW) for bugs that do not require QA verification
f) "qablocked" for QA to specify bugs that block our testing
g) "qaurgent" for telling QA that the request made with other keywords
is highest priority - we should always try to keep this at zero

a-d and g would be removed when the request has been fulfilled, when
removing "verifyme", the appropriate verified status fields would be set
or the bug would be reopened.


And that's it, bascially.

The only really new thing here would be that "noverify" would replace
the [qa-] whiteboard marker and we set "verifyme" right away instead of
[qa+] as we know to only actually do the verification (have it show up
on our radar) when a bug is resolved.


Q&A:

Q: How would this deal with the amount of bugmail that Firefox
developers get?
A: As we'll try to get "verifyme"/"noverify" set by the Firefox team
right when putting bugs on the team backlog, there will not be a ton of
additional bugmail for that - and developers should get bugmail when we
verify the bug and their work is officially done or even more when a bug
is reopened.

Q: How does this compare to the existing attributes:
A: Let's go over the existing list:

 > * qa+, qa-
Moved to "verifyme" and "noverify".

 > * qa?, qa!
Not really needed. Anything not clearly "noverify" should have
"verifyme" set, where verification is done, we see that via the bug
status fields.

 > * regressionwindow-wanted
 > * qawanted
 > * steps-wanted
Unchanged.

 > * verifyme
Bascially unchanged, but appended so that setting it on an open bug
means "this will need verification when resolved".

 > * qablocked
 > * qaurgent
Unchanged.

Q: How can we know what we need to triage for "verifyme"/"noverify"?
A: By the bugs not having any of those keywords and no verified status.

Q: How can we make reports on how much work we did?
A: While Bugzilla itself isn't good at queries like "bugs that had this
keyword set but it was removed", we are investigating options that
hopefully are (all Bugzilla data also goes into an ES database that
should be able to do those queries and dashboards for them).


Feel free to ask more questions or put forward other proposals. If no
feedback comes within a week or two, we will decide on this within the
Firefox QA team and move forward with the result of that decision. (I
will post the decision to dev-platform and firex-dev as well, then.)


KaiRo
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Matt Wobensmith
Hi KaiRo -

I've been using [qa-] to mark bugs that we (QA) have no intention of verifying, due to time and/or technical constraints. I'm not sure if that's an agreed-upon practice - or even correct - but is there a keyword in your proposal that fits this workflow? Note that QA deciding a bug will not be verified is a bit different than a developer making the same decision. I think these two conditions merit two different keywords, to show that a process is in place.


Thanks,


Matt Wobensmith
QA Engineer
Mozilla Corporation

----- Original Message -----
From: "Robert Kaiser" <[hidden email]>
To: [hidden email]
Sent: Friday, May 30, 2014 6:07:59 AM
Subject: Desktop QA - Bugzilla keywords: Proposal

tl;dr: I propose that desktop QA will move from using some whiteboard
tags to keywords for usual bug markings, most importantly "verifyme"
instead of [qa+] and "noverify" (NEW) instead of [qa-].



The activity on our discussion about "Desktop QA: Bugzilla
tags/keywords" has died down in the last weeks, and it doesn't look like
we did have a clear agreement there.
 From talks in meetings, it sounds like going with something that has
clearly defined states like keywords or flags is preferred by most people.

I thought a bit about that and have the following proposal based on that:

1) Let's use keywords for the core long-term repeating things.
2) Free-form comments or markers of temporary importance (for a single
project, like Australis, Sync, etc.) should go into the QA whiteboard.

The following keywords would be used:
a) "qawanted" for anything where investigation from QA is needed
b) "regressionwindow-wanted" where a regression window is needed
c) "steps-wanted" where we need steps to reproduce
d) "verifyme" for bugs that (will) need QA verification
e) "noverify" (NEW) for bugs that do not require QA verification
f) "qablocked" for QA to specify bugs that block our testing
g) "qaurgent" for telling QA that the request made with other keywords
is highest priority - we should always try to keep this at zero

a-d and g would be removed when the request has been fulfilled, when
removing "verifyme", the appropriate verified status fields would be set
or the bug would be reopened.


And that's it, bascially.

The only really new thing here would be that "noverify" would replace
the [qa-] whiteboard marker and we set "verifyme" right away instead of
[qa+] as we know to only actually do the verification (have it show up
on our radar) when a bug is resolved.


Q&A:

Q: How would this deal with the amount of bugmail that Firefox
developers get?
A: As we'll try to get "verifyme"/"noverify" set by the Firefox team
right when putting bugs on the team backlog, there will not be a ton of
additional bugmail for that - and developers should get bugmail when we
verify the bug and their work is officially done or even more when a bug
is reopened.

Q: How does this compare to the existing attributes:
A: Let's go over the existing list:

 > * qa+, qa-
Moved to "verifyme" and "noverify".

 > * qa?, qa!
Not really needed. Anything not clearly "noverify" should have
"verifyme" set, where verification is done, we see that via the bug
status fields.

 > * regressionwindow-wanted
 > * qawanted
 > * steps-wanted
Unchanged.

 > * verifyme
Bascially unchanged, but appended so that setting it on an open bug
means "this will need verification when resolved".

 > * qablocked
 > * qaurgent
Unchanged.

Q: How can we know what we need to triage for "verifyme"/"noverify"?
A: By the bugs not having any of those keywords and no verified status.

Q: How can we make reports on how much work we did?
A: While Bugzilla itself isn't good at queries like "bugs that had this
keyword set but it was removed", we are investigating options that
hopefully are (all Bugzilla data also goes into an ES database that
should be able to do those queries and dashboards for them).


Feel free to ask more questions or put forward other proposals. If no
feedback comes within a week or two, we will decide on this within the
Firefox QA team and move forward with the result of that decision. (I
will post the decision to dev-platform and firex-dev as well, then.)


KaiRo
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

mhoye
On 2014-05-30, 3:30 PM, Matt Wobensmith wrote:
> Hi KaiRo -
>
> I've been using [qa-] to mark bugs that we (QA) have no intention of verifying, due to time and/or technical constraints. I'm not sure if that's an agreed-upon practice - or even correct - but is there a keyword in your proposal that fits this workflow? Note that QA deciding a bug will not be verified is a bit different than a developer making the same decision. I think these two conditions merit two different keywords, to show that a process is in place.
Are these time or technical constraints something that our larger
community could help us with?

Over in Firefox-Desktop land right now, we've got this thing called a
"Diamond" bug. They're bugs that we flag as a priority, that make it all
the way to our biweekly triage meetings, but don't leave those meetings
assigned to an engineer. They're important bugs, just not quite as
important as everything ahead of them.

We advertise them in various forums, though, and we commit to returning
contributors' calls quickly if they decide they'd like to take one up.

I'd like you to consider things that may be important, but that you
don't have the resources or bandwidth to address right away in that
light - this is an opportunity to engage your community of something of
demonstrably high value -  rather than setting them aside to be
retriaged later.

- mhoye
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Matt Wobensmith
Hi Mike -

My comment about time and technical constraints generally refer to the verification of fixed security bugs, which pose big challenges to community involvement.

For starters, they're private. They also frequently involve either custom builds and/or use of specialized tools.

Another limiting factor is that security bugs can be overly cautious, fixing a problem that may be very difficult to reproduce in real life. Or near impossible, but fixed for safety's sake.

Just wanted to convey that such decisions aren't made lightly. Security - IMHO - is probably the most important class of bugs to merit verification, as so much is at stake. I'm admittedly biased as this is the majority of what I do all day.

When a decision has been made to not verify a fixed security bug, we always leave a comment as to why.

For what it's worth, we do involve community whenever possible.  

Thanks,


Matt Wobensmith
QA Engineer
Mozilla Corporation

----- Original Message -----
From: "Mike Hoye" <[hidden email]>
To: [hidden email]
Sent: Friday, May 30, 2014 12:40:59 PM
Subject: Re: Desktop QA - Bugzilla keywords: Proposal

On 2014-05-30, 3:30 PM, Matt Wobensmith wrote:
> Hi KaiRo -
>
> I've been using [qa-] to mark bugs that we (QA) have no intention of verifying, due to time and/or technical constraints. I'm not sure if that's an agreed-upon practice - or even correct - but is there a keyword in your proposal that fits this workflow? Note that QA deciding a bug will not be verified is a bit different than a developer making the same decision. I think these two conditions merit two different keywords, to show that a process is in place.
Are these time or technical constraints something that our larger
community could help us with?

Over in Firefox-Desktop land right now, we've got this thing called a
"Diamond" bug. They're bugs that we flag as a priority, that make it all
the way to our biweekly triage meetings, but don't leave those meetings
assigned to an engineer. They're important bugs, just not quite as
important as everything ahead of them.

We advertise them in various forums, though, and we commit to returning
contributors' calls quickly if they decide they'd like to take one up.

I'd like you to consider things that may be important, but that you
don't have the resources or bandwidth to address right away in that
light - this is an opportunity to engage your community of something of
demonstrably high value -  rather than setting them aside to be
retriaged later.

- mhoye
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Liz Henry
In reply to this post by mhoye
On 5/30/14 12:40 PM, Mike Hoye wrote:


> Are these time or technical constraints something that our larger
> community could help us with?

Hi Mike,

We do have a [good first verify] whiteboard tag for bugs that developers
and/or QA don't think are a priority, but would be nice to verify.
That's a good place to jump in on a bug verification day
(https://wiki.mozilla.org/Bugdays/Bug-verification)

I think there are other possible ways to invite community involvement,
possibly by adding a few more tags to specify what operating systems,
versions, or architectures need to have verification or other triage work.


- liz





--
Liz Henry
[hidden email]
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Matthew N.
In reply to this post by Robert Kaiser-2
On 5/30/14, 6:07 AM, Robert Kaiser wrote:
> The activity on our discussion about "Desktop QA: Bugzilla
> tags/keywords" has died down in the last weeks, and it doesn't look like
> we did have a clear agreement there.

Thanks for following-up on this, I'm looking forward to streamlined QA
bugmail.

> 1) Let's use keywords for the core long-term repeating things.
> 2) Free-form comments or markers of temporary importance (for a single
> project, like Australis, Sync, etc.) should go into the QA whiteboard.

For others who are interested, some popular examples of other
information for the QA whiteboard seems to be:
* [good first verify]
* [bugday-YYYYMMDD]
* [testday-YYYYMMDD]

Source:
https://bugzilla.mozilla.org/buglist.cgi?f1=cf_qa_whiteboard&o1=isnotempty&query_format=advanced&query_based_on=&columnlist=changeddate%2Cproduct%2Ccomponent%2Cbug_severity%2Cpriority%2Cop_sys%2Cassigned_to%2Cbug_status%2Cresolution%2Cshort_desc%2Ccf_qa_whiteboard 


> Q: How would this deal with the amount of bugmail that Firefox
> developers get?
> A: As we'll try to get "verifyme"/"noverify" set by the Firefox team
> right when putting bugs on the team backlog, there will not be a ton of
> additional bugmail for that - and developers should get bugmail when we
> verify the bug and their work is officially done or even more when a bug
> is reopened.

Note that using keywords over whiteboards is an improvement for bugmail
reading since changes to keywords show only the keywords added or
removed in the email. Contrast this with whiteboard changes where the
contents of the entire whiteboard before and after the change are sent
and it's left up to the reader to do a diff of the tokens in their mind.
This is especially a problem on narrow viewports such as mobile devices
or with the reading pane on the side in Zimbra.

> Q: How does this compare to the existing attributes:
> …
>  > * qa?, qa!
> Not really needed. Anything not clearly "noverify" should have
> "verifyme" set, where verification is done, we see that via the bug
> status fields.

+1 for getting rid of "qa?" as it was a source of unnecessary noise IMO
as I agree it's not needed given the other fields.

TBH, I can't find documentation on what "qa!" means so I can't comment
on that.

Thanks,
Matthew N.
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Robert Kaiser
In reply to this post by Robert Kaiser-2
Matt,

> I've been using [qa-] to mark bugs that we (QA) have no intention of verifying, due to time and/or technical constraints. I'm not sure if that's an agreed-upon practice - or even correct - but is there a keyword in your proposal that fits this workflow? Note that QA deciding a bug will not be verified is a bit different than a developer making the same decision. I think these two conditions merit two different keywords, to show that a process is in place.

What you describe is how we all have been using [qa-] so far and I'm
just proposing in this thread that we replace that with the new
"noverify" keyword (if/when we agree on this proposal, I'll file a bug
for this to be created).

There is a separate proposal floating around to make developers set
those attributes (verifyme/noverify in my proposal) at the time when
they triage which bugs they actually work on, so that we need to spend
fewer QA resources on triaging. This is of course a tradeoff but we have
very few people in QA compared to engineering and need to make as
efficient use of their time as possible. That said, I see your argument
that we probably want more scrutiny for security stuff. Let's take that
to a different thread, though. I wanted to get the bugzilla keywords
settled before we focus on the process discussion that we also need to have.

KaiRo
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Robert Kaiser
In reply to this post by Matthew N.
Matthew N. schrieb:
> On 5/30/14, 6:07 AM, Robert Kaiser wrote:
>> 2) Free-form comments or markers of temporary importance (for a single
>> project, like Australis, Sync, etc.) should go into the QA whiteboard.
>
> For others who are interested, some popular examples of other
> information for the QA whiteboard seems to be:
> * [good first verify]
> * [bugday-YYYYMMDD]
> * [testday-YYYYMMDD]

Yes, thanks, those are good examples.


> Note that using keywords over whiteboards is an improvement for bugmail
> reading since changes to keywords show only the keywords added or
> removed in the email.

Thanks, that's a good pointer (and additional argument).

> TBH, I can't find documentation on what "qa!" means so I can't comment
> on that.

It was used to some degree (but not consistently from what I've seen) to
mean "QA done". But then, as I mentioned, either marking as verified or
reopening should state that anyhow.

KaiRo
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

RE: Desktop QA - Bugzilla keywords: Proposal

Florin Mezei
In reply to this post by Robert Kaiser-2
The proposal sounds good to me.

-----Original Message-----
From: dev-quality
[mailto:dev-quality-bounces+florin.mezei=[hidden email]]
On Behalf Of Robert Kaiser
Sent: Friday, May 30, 2014 4:08 PM
To: [hidden email]
Subject: Desktop QA - Bugzilla keywords: Proposal

tl;dr: I propose that desktop QA will move from using some whiteboard tags
to keywords for usual bug markings, most importantly "verifyme"
instead of [qa+] and "noverify" (NEW) instead of [qa-].



The activity on our discussion about "Desktop QA: Bugzilla
tags/keywords" has died down in the last weeks, and it doesn't look like
we did have a clear agreement there.
 From talks in meetings, it sounds like going with something that has
clearly defined states like keywords or flags is preferred by most people.

I thought a bit about that and have the following proposal based on that:

1) Let's use keywords for the core long-term repeating things.
2) Free-form comments or markers of temporary importance (for a single
project, like Australis, Sync, etc.) should go into the QA whiteboard.

The following keywords would be used:
a) "qawanted" for anything where investigation from QA is needed
b) "regressionwindow-wanted" where a regression window is needed
c) "steps-wanted" where we need steps to reproduce
d) "verifyme" for bugs that (will) need QA verification
e) "noverify" (NEW) for bugs that do not require QA verification
f) "qablocked" for QA to specify bugs that block our testing
g) "qaurgent" for telling QA that the request made with other keywords
is highest priority - we should always try to keep this at zero

a-d and g would be removed when the request has been fulfilled, when
removing "verifyme", the appropriate verified status fields would be set
or the bug would be reopened.


And that's it, bascially.

The only really new thing here would be that "noverify" would replace
the [qa-] whiteboard marker and we set "verifyme" right away instead of
[qa+] as we know to only actually do the verification (have it show up
on our radar) when a bug is resolved.


Q&A:

Q: How would this deal with the amount of bugmail that Firefox
developers get?
A: As we'll try to get "verifyme"/"noverify" set by the Firefox team
right when putting bugs on the team backlog, there will not be a ton of
additional bugmail for that - and developers should get bugmail when we
verify the bug and their work is officially done or even more when a bug
is reopened.

Q: How does this compare to the existing attributes:
A: Let's go over the existing list:

 > * qa+, qa-
Moved to "verifyme" and "noverify".

 > * qa?, qa!
Not really needed. Anything not clearly "noverify" should have
"verifyme" set, where verification is done, we see that via the bug
status fields.

 > * regressionwindow-wanted
 > * qawanted
 > * steps-wanted
Unchanged.

 > * verifyme
Bascially unchanged, but appended so that setting it on an open bug
means "this will need verification when resolved".

 > * qablocked
 > * qaurgent
Unchanged.

Q: How can we know what we need to triage for "verifyme"/"noverify"?
A: By the bugs not having any of those keywords and no verified status.

Q: How can we make reports on how much work we did?
A: While Bugzilla itself isn't good at queries like "bugs that had this
keyword set but it was removed", we are investigating options that
hopefully are (all Bugzilla data also goes into an ES database that
should be able to do those queries and dashboards for them).


Feel free to ask more questions or put forward other proposals. If no
feedback comes within a week or two, we will decide on this within the
Firefox QA team and move forward with the result of that decision. (I
will post the decision to dev-platform and firex-dev as well, then.)


KaiRo
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality


_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

ashughes
In reply to this post by Robert Kaiser-2
The way I see it, we're trying to solve two problems:
A) Make QA as it exists in Bugzilla a lot less ambiguous
B) Reduce the noise in Bugzilla

I'm not sure adding more descriptive keywords and/or moving things into another text field (ie. QA Whiteboard) is going to solve either of those problems.

The more I think about this the more I think having a flag to indicate "QA status" would be far more useful. It's certainly easier to query on changes and allows us more granularity than flooding a text field. We could even break that down into two QA flags, one to reflect status and one to act as a requestor.

If we've decided to stick to a combination of keywords and whiteboard tags then I have no real preference. As long as it's something developers and QA alike can buy in to.  

FWIW, we're also getting pressure from the Firefox team to come up with something they can start using ASAP.

On Monday, June 2, 2014 8:16:55 AM UTC-7, Florin Mezei wrote:

> The proposal sounds good to me.
>
>
>
> -----Original Message-----
>
> From: dev-quality
>
> [mailto:dev-quality-bounces+florin.mezei=[hidden email]]
>
> On Behalf Of Robert Kaiser
>
> Sent: Friday, May 30, 2014 4:08 PM
>
> To: [hidden email]
>
> Subject: Desktop QA - Bugzilla keywords: Proposal
>
>
>
> tl;dr: I propose that desktop QA will move from using some whiteboard tags
>
> to keywords for usual bug markings, most importantly "verifyme"
>
> instead of [qa+] and "noverify" (NEW) instead of [qa-].
>
>
>
>
>
>
>
> The activity on our discussion about "Desktop QA: Bugzilla
>
> tags/keywords" has died down in the last weeks, and it doesn't look like
>
> we did have a clear agreement there.
>
>  From talks in meetings, it sounds like going with something that has
>
> clearly defined states like keywords or flags is preferred by most people.
>
>
>
> I thought a bit about that and have the following proposal based on that:
>
>
>
> 1) Let's use keywords for the core long-term repeating things.
>
> 2) Free-form comments or markers of temporary importance (for a single
>
> project, like Australis, Sync, etc.) should go into the QA whiteboard.
>
>
>
> The following keywords would be used:
>
> a) "qawanted" for anything where investigation from QA is needed
>
> b) "regressionwindow-wanted" where a regression window is needed
>
> c) "steps-wanted" where we need steps to reproduce
>
> d) "verifyme" for bugs that (will) need QA verification
>
> e) "noverify" (NEW) for bugs that do not require QA verification
>
> f) "qablocked" for QA to specify bugs that block our testing
>
> g) "qaurgent" for telling QA that the request made with other keywords
>
> is highest priority - we should always try to keep this at zero
>
>
>
> a-d and g would be removed when the request has been fulfilled, when
>
> removing "verifyme", the appropriate verified status fields would be set
>
> or the bug would be reopened.
>
>
>
>
>
> And that's it, bascially.
>
>
>
> The only really new thing here would be that "noverify" would replace
>
> the [qa-] whiteboard marker and we set "verifyme" right away instead of
>
> [qa+] as we know to only actually do the verification (have it show up
>
> on our radar) when a bug is resolved.
>
>
>
>
>
> Q&A:
>
>
>
> Q: How would this deal with the amount of bugmail that Firefox
>
> developers get?
>
> A: As we'll try to get "verifyme"/"noverify" set by the Firefox team
>
> right when putting bugs on the team backlog, there will not be a ton of
>
> additional bugmail for that - and developers should get bugmail when we
>
> verify the bug and their work is officially done or even more when a bug
>
> is reopened.
>
>
>
> Q: How does this compare to the existing attributes:
>
> A: Let's go over the existing list:
>
>
>
>  > * qa+, qa-
>
> Moved to "verifyme" and "noverify".
>
>
>
>  > * qa?, qa!
>
> Not really needed. Anything not clearly "noverify" should have
>
> "verifyme" set, where verification is done, we see that via the bug
>
> status fields.
>
>
>
>  > * regressionwindow-wanted
>
>  > * qawanted
>
>  > * steps-wanted
>
> Unchanged.
>
>
>
>  > * verifyme
>
> Bascially unchanged, but appended so that setting it on an open bug
>
> means "this will need verification when resolved".
>
>
>
>  > * qablocked
>
>  > * qaurgent
>
> Unchanged.
>
>
>
> Q: How can we know what we need to triage for "verifyme"/"noverify"?
>
> A: By the bugs not having any of those keywords and no verified status.
>
>
>
> Q: How can we make reports on how much work we did?
>
> A: While Bugzilla itself isn't good at queries like "bugs that had this
>
> keyword set but it was removed", we are investigating options that
>
> hopefully are (all Bugzilla data also goes into an ES database that
>
> should be able to do those queries and dashboards for them).
>
>
>
>
>
> Feel free to ask more questions or put forward other proposals. If no
>
> feedback comes within a week or two, we will decide on this within the
>
> Firefox QA team and move forward with the result of that decision. (I
>
> will post the decision to dev-platform and firex-dev as well, then.)
>
>
>
>
>
> KaiRo
>
> _______________________________________________
>
> dev-quality mailing list
>
> [hidden email]
>
> https://lists.mozilla.org/listinfo/dev-quality

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Robert Kaiser
[hidden email] schrieb:
> The way I see it, we're trying to solve two problems:
> A) Make QA as it exists in Bugzilla a lot less ambiguous
> B) Reduce the noise in Bugzilla
>
> I'm not sure adding more descriptive keywords and/or moving things into another text field (ie. QA Whiteboard) is going to solve either of those problems.

Another textfield won't solve A for sure, a fixed set of more
descriptive keywords should help it though.
For B, another textfield actually helps if it can be filtered out in
bugmail, and the Bugzilla team is working on that. Also, as Matthew N.
stated in this thread, keywords are already better than whiteboards in
terms of bugmail because only the actually added/removed keywords are
listed and not the whole whiteboard field before and after the change.

> The more I think about this the more I think having a flag to indicate "QA status" would be far more useful. It's certainly easier to query on changes and allows us more granularity than flooding a text field. We could even break that down into two QA flags, one to reflect status and one to act as a requestor.

I'd be just as happy with that but it's not easy for me to actually find
a decent set of what should be in there, if we want to fold all our uses
into it. Note that only one value of the field can be set at a given time.

> If we've decided to stick to a combination of keywords and whiteboard tags then I have no real preference. As long as it's something developers and QA alike can buy in to.

My proposal clearly goes for keywords for all the main uses. Whiteboards
will never be 100% out of any picture just because temporary markers and
notes are what they are entirely for. In our case, marking a testday
that had found this or such things will always be a use of them. I
personally think though that the separate QA whiteboard has been created
prematurely and this shouldn't have been done without larger coordination.

> FWIW, we're also getting pressure from the Firefox team to come up with something they can start using ASAP.

We first should care about what we want or need for our purposes and not
let other teams create pressure. That said, I think there cannot be one
solution that will make everyone completely happy.

KaiRo

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Marc Schifer

I have two proposals on how we can address the keyword/whiteboard flags and bug spam issues that have been discussed lately. The goal of these proposals is give QA a better way of tracking status of bugs, managing our work flows and being able to collect statistics about the health of the releases as well as give us an easy means of filtering out bug mail until the selective bug notification system is added to Bugzilla.


First proposal:
  Move all standard QA related keyword/whiteboard tags into flags.  This would allow us to track various QA activities by project/release and allow for easy bug mail filtering. Flags currently set an X Header, X-Bugzilla-Flags, in the mail sent that will be simple to have a client side filter on until the Bugzilla selective email filtering is completed.

In this proposal we would create the following flags:

  QA-status-[projectXX] [qawanted,qaurgent,qablocked]  =Triage verification need, +=Needs Verification, -=No Verification to be done
  QA-verify-[projectXX] [?|+|-]                        - ?=Triage verification need, +=Needs Verification, -=No Verification to be done  QA-steps-wanted[?|+|-]                               ?=Request for STR, +=STR Provided, -=Unable to provide STR (reason in comments)
  QA-regression-window-wanted [?|+|-]                  ?=Request for Regression Window  +=Window Provided, -=Unable to providw (reason in comments)
  QA-testcase-wanted [?|+|-]                           ?=Request for testcase  +=Testcase Provided (set proper intestcase flag), -=Unable to provide  (reason in comments)

In all cases a blank status would mean no action is requested

The QA-verify-[projectXX] flag would be release specific requesting a verification on which ever projects and releases were required + would indicate request for verification, - would indicate that no verification was going to be done and ? would be a triage requested flag.  We could leave off the ? and leave the initital state blank as well to indicate that a request for verification has been made and then the + would indicate that QA has the bug on their to be verified list. This would allow the various project teams to track which bugs may require

QA-status-[projectXX] flag would indicate which project/release a particular request or status applies too. For instance a bug filed originally against B2G for graphics issues may also require QA attention on Desktop or Android. In which case we could set flags such:
  QA-status-Firefox30 = qawanted
  QA-status-fennec30  = qawanted
  QA-status-b2fv-1.4  = qaurgent

The remaining flags QA-steps-wanted, QA-regression-window-wanted and QA-testcase-wanted would be non-project specific and simply flag the need for QA's input for the particular item.

All unset flags would be hidden by default as we do with the tracking flags to prevent flag spam in the UI. We should also have some sort of auto policing for older flags that would hide those that are more than say 4 releases old. This would then only show flags for those releases currently in flight to prevent long lived bugs from having long list of flags.

This process would give us a consistent workflow across all of QA and by using the existing features that let us track how long a flag spends in each state we can easily build dashboards to track our performance and the current status of each release.  

If we adopt this proposal, we will keep the keywords around for a period of time to allow for an orderly transition and give teams time to adjust their processes.



Second Proposal:
This would be more of a hybrid of the current processes and the above.   Keywords would be used by external/non-QA groups to request QA action and we flags would be used to track internal QA processes.  This would require less re-training and still reduce some of the bug spam by keeping the QA process changes easily filterable.

In this proposal we would have the following:
Keywords:
  qawanted
  qaurgent
  steps-wanted
  regression-window-wanted
  testcase-wanted

Flags:
  QA-verify-[projectXX] [?|+|-]

 
What we would lose in this versions would be the ability to track QA requirements across projects and releases.

In both cases we would drop the QA* white board tags entirely and the QA White board itself would be used for one off, project specific needs that are out side of the normal work flows.  We would also retain the use of the existing flags ,in-testsuite, in-moztrap and tracking-[projectXX]=verified, though we might need to require their usage more often the they are currently used.

This proposal would require the least amount of change in our processes, but still give an easily filterable way of handling  the bug verification emails . Keyword changes would still have bug mail issues that will have to wait to be addressed by the Buzilla selective email filtering feature.

In either case I would also propose a small process change where we do not mark a bug verified until it has all the QA-verify[projectXX] and tracking-[projectXX]=verify flags set  

Marc S.



----- Original Message -----
From: "Robert Kaiser" <[hidden email]>
To: [hidden email]
Sent: Thursday, June 5, 2014 7:16:26 AM
Subject: Re: Desktop QA - Bugzilla keywords: Proposal

[hidden email] schrieb:
> The way I see it, we're trying to solve two problems:
> A) Make QA as it exists in Bugzilla a lot less ambiguous
> B) Reduce the noise in Bugzilla
>
> I'm not sure adding more descriptive keywords and/or moving things into another text field (ie. QA Whiteboard) is going to solve either of those problems.

Another textfield won't solve A for sure, a fixed set of more
descriptive keywords should help it though.
For B, another textfield actually helps if it can be filtered out in
bugmail, and the Bugzilla team is working on that. Also, as Matthew N.
stated in this thread, keywords are already better than whiteboards in
terms of bugmail because only the actually added/removed keywords are
listed and not the whole whiteboard field before and after the change.

> The more I think about this the more I think having a flag to indicate "QA status" would be far more useful. It's certainly easier to query on changes and allows us more granularity than flooding a text field. We could even break that down into two QA flags, one to reflect status and one to act as a requestor.

I'd be just as happy with that but it's not easy for me to actually find
a decent set of what should be in there, if we want to fold all our uses
into it. Note that only one value of the field can be set at a given time.

> If we've decided to stick to a combination of keywords and whiteboard tags then I have no real preference. As long as it's something developers and QA alike can buy in to.

My proposal clearly goes for keywords for all the main uses. Whiteboards
will never be 100% out of any picture just because temporary markers and
notes are what they are entirely for. In our case, marking a testday
that had found this or such things will always be a use of them. I
personally think though that the separate QA whiteboard has been created
prematurely and this shouldn't have been done without larger coordination.

> FWIW, we're also getting pressure from the Firefox team to come up with something they can start using ASAP.

We first should care about what we want or need for our purposes and not
let other teams create pressure. That said, I think there cannot be one
solution that will make everyone completely happy.

KaiRo

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Mihaela Velimiroviciu
Hi,

I added a suggestion inline.

Thanks,,
Mihaela

On 16.06.2014 20:49, Marc Schifer wrote:

> I have two proposals on how we can address the keyword/whiteboard flags and bug spam issues that have been discussed lately. The goal of these proposals is give QA a better way of tracking status of bugs, managing our work flows and being able to collect statistics about the health of the releases as well as give us an easy means of filtering out bug mail until the selective bug notification system is added to Bugzilla.
>
>
> First proposal:
>    Move all standard QA related keyword/whiteboard tags into flags.  This would allow us to track various QA activities by project/release and allow for easy bug mail filtering. Flags currently set an X Header, X-Bugzilla-Flags, in the mail sent that will be simple to have a client side filter on until the Bugzilla selective email filtering is completed.
>
> In this proposal we would create the following flags:
>
>    QA-status-[projectXX] [qawanted,qaurgent,qablocked]  =Triage verification need, +=Needs Verification, -=No Verification to be done
>    QA-verify-[projectXX] [?|+|-]                        - ?=Triage verification need, +=Needs Verification, -=No Verification to be done  QA-steps-wanted[?|+|-]                               ?=Request for STR, +=STR Provided, -=Unable to provide STR (reason in comments)
>    QA-regression-window-wanted [?|+|-]                  ?=Request for Regression Window  +=Window Provided, -=Unable to providw (reason in comments)
>    QA-testcase-wanted [?|+|-]                           ?=Request for testcase  +=Testcase Provided (set proper intestcase flag), -=Unable to provide  (reason in comments)
>
> In all cases a blank status would mean no action is requested
>
> The QA-verify-[projectXX] flag would be release specific requesting a verification on which ever projects and releases were required + would indicate request for verification, - would indicate that no verification was going to be done and ? would be a triage requested flag.  We could leave off the ? and leave the initital state blank as well to indicate that a request for verification has been made and then the + would indicate that QA has the bug on their to be verified list. This would allow the various project teams to track which bugs may require
>
> QA-status-[projectXX] flag would indicate which project/release a particular request or status applies too. For instance a bug filed originally against B2G for graphics issues may also require QA attention on Desktop or Android. In which case we could set flags such:
>    QA-status-Firefox30 = qawanted
>    QA-status-fennec30  = qawanted
>    QA-status-b2fv-1.4  = qaurgent
>
> The remaining flags QA-steps-wanted, QA-regression-window-wanted and QA-testcase-wanted would be non-project specific and simply flag the need for QA's input for the particular item.
>
> All unset flags would be hidden by default as we do with the tracking flags to prevent flag spam in the UI. We should also have some sort of auto policing for older flags that would hide those that are more than say 4 releases old. This would then only show flags for those releases currently in flight to prevent long lived bugs from having long list of flags.
>
> This process would give us a consistent workflow across all of QA and by using the existing features that let us track how long a flag spends in each state we can easily build dashboards to track our performance and the current status of each release.
>
> If we adopt this proposal, we will keep the keywords around for a period of time to allow for an orderly transition and give teams time to adjust their processes.
>
>
>
> Second Proposal:
> This would be more of a hybrid of the current processes and the above.   Keywords would be used by external/non-QA groups to request QA action and we flags would be used to track internal QA processes.  This would require less re-training and still reduce some of the bug spam by keeping the QA process changes easily filterable.
>
> In this proposal we would have the following:
> Keywords:
>    qawanted
>    qaurgent
>    steps-wanted
>    regression-window-wanted
>    testcase-wanted
>
> Flags:
>    QA-verify-[projectXX] [?|+|-]
>
>  
> What we would lose in this versions would be the ability to track QA requirements across projects and releases.
>
> In both cases we would drop the QA* white board tags entirely and the QA White board itself would be used for one off, project specific needs that are out side of the normal work flows.  We would also retain the use of the existing flags ,in-testsuite, in-moztrap and tracking-[projectXX]=verified, though we might need to require their usage more often the they are currently used.
>
> This proposal would require the least amount of change in our processes, but still give an easily filterable way of handling  the bug verification emails . Keyword changes would still have bug mail issues that will have to wait to be addressed by the Buzilla selective email filtering feature.
>
> In either case I would also propose a small process change where we do not mark a bug verified until it has all the QA-verify[projectXX] and tracking-[projectXX]=verify flags set

We could also continue to mark the bug "Verified" when it is verified on
the target milestone version and add the "Closed" status in Bugzilla for
when it is verified on all the other tracking versions and the flags are
set.

>
> Marc S.
>
>
>
> ----- Original Message -----
> From: "Robert Kaiser" <[hidden email]>
> To: [hidden email]
> Sent: Thursday, June 5, 2014 7:16:26 AM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
> [hidden email] schrieb:
>> The way I see it, we're trying to solve two problems:
>> A) Make QA as it exists in Bugzilla a lot less ambiguous
>> B) Reduce the noise in Bugzilla
>>
>> I'm not sure adding more descriptive keywords and/or moving things into another text field (ie. QA Whiteboard) is going to solve either of those problems.
> Another textfield won't solve A for sure, a fixed set of more
> descriptive keywords should help it though.
> For B, another textfield actually helps if it can be filtered out in
> bugmail, and the Bugzilla team is working on that. Also, as Matthew N.
> stated in this thread, keywords are already better than whiteboards in
> terms of bugmail because only the actually added/removed keywords are
> listed and not the whole whiteboard field before and after the change.
>
>> The more I think about this the more I think having a flag to indicate "QA status" would be far more useful. It's certainly easier to query on changes and allows us more granularity than flooding a text field. We could even break that down into two QA flags, one to reflect status and one to act as a requestor.
> I'd be just as happy with that but it's not easy for me to actually find
> a decent set of what should be in there, if we want to fold all our uses
> into it. Note that only one value of the field can be set at a given time.
>
>> If we've decided to stick to a combination of keywords and whiteboard tags then I have no real preference. As long as it's something developers and QA alike can buy in to.
> My proposal clearly goes for keywords for all the main uses. Whiteboards
> will never be 100% out of any picture just because temporary markers and
> notes are what they are entirely for. In our case, marking a testday
> that had found this or such things will always be a use of them. I
> personally think though that the separate QA whiteboard has been created
> prematurely and this shouldn't have been done without larger coordination.
>
>> FWIW, we're also getting pressure from the Firefox team to come up with something they can start using ASAP.
> We first should care about what we want or need for our purposes and not
> let other teams create pressure. That said, I think there cannot be one
> solution that will make everyone completely happy.
>
> KaiRo
>
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Gavin Sharp-2
In reply to this post by Marc Schifer
Thanks for this, Marc.

I prefer proposal #2, since it seems simpler in scope, and less of a departure from our existing processes.

I'm not sure I understand this part, though:

> In either case I would also propose a small process change where we do not
> mark a bug verified until it has all the QA-verify[projectXX] and
> tracking-[projectXX]=verify flags set

Can you elaborate?

Gavin

----- Original Message -----

> From: "Marc Schifer" <[hidden email]>
> To: [hidden email]
> Cc: "Gavin Sharp" <[hidden email]>
> Sent: Monday, June 16, 2014 1:49:08 PM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
>
> I have two proposals on how we can address the keyword/whiteboard flags and
> bug spam issues that have been discussed lately. The goal of these proposals
> is give QA a better way of tracking status of bugs, managing our work flows
> and being able to collect statistics about the health of the releases as
> well as give us an easy means of filtering out bug mail until the selective
> bug notification system is added to Bugzilla.
>
>
> First proposal:
>   Move all standard QA related keyword/whiteboard tags into flags.  This
>   would allow us to track various QA activities by project/release and allow
>   for easy bug mail filtering. Flags currently set an X Header,
>   X-Bugzilla-Flags, in the mail sent that will be simple to have a client
>   side filter on until the Bugzilla selective email filtering is completed.
>
> In this proposal we would create the following flags:
>
>   QA-status-[projectXX] [qawanted,qaurgent,qablocked]  =Triage verification
>   need, +=Needs Verification, -=No Verification to be done
>   QA-verify-[projectXX] [?|+|-]                        - ?=Triage
>   verification need, +=Needs Verification, -=No Verification to be done
>   QA-steps-wanted[?|+|-]                               ?=Request for STR,
>   +=STR Provided, -=Unable to provide STR (reason in comments)
>   QA-regression-window-wanted [?|+|-]                  ?=Request for
>   Regression Window  +=Window Provided, -=Unable to providw (reason in
>   comments)
>   QA-testcase-wanted [?|+|-]                           ?=Request for testcase
>   +=Testcase Provided (set proper intestcase flag), -=Unable to provide
>   (reason in comments)
>
> In all cases a blank status would mean no action is requested
>
> The QA-verify-[projectXX] flag would be release specific requesting a
> verification on which ever projects and releases were required + would
> indicate request for verification, - would indicate that no verification was
> going to be done and ? would be a triage requested flag.  We could leave off
> the ? and leave the initital state blank as well to indicate that a request
> for verification has been made and then the + would indicate that QA has the
> bug on their to be verified list. This would allow the various project teams
> to track which bugs may require
>
> QA-status-[projectXX] flag would indicate which project/release a particular
> request or status applies too. For instance a bug filed originally against
> B2G for graphics issues may also require QA attention on Desktop or Android.
> In which case we could set flags such:
>   QA-status-Firefox30 = qawanted
>   QA-status-fennec30  = qawanted
>   QA-status-b2fv-1.4  = qaurgent
>
> The remaining flags QA-steps-wanted, QA-regression-window-wanted and
> QA-testcase-wanted would be non-project specific and simply flag the need
> for QA's input for the particular item.
>
> All unset flags would be hidden by default as we do with the tracking flags
> to prevent flag spam in the UI. We should also have some sort of auto
> policing for older flags that would hide those that are more than say 4
> releases old. This would then only show flags for those releases currently
> in flight to prevent long lived bugs from having long list of flags.
>
> This process would give us a consistent workflow across all of QA and by
> using the existing features that let us track how long a flag spends in each
> state we can easily build dashboards to track our performance and the
> current status of each release.
>
> If we adopt this proposal, we will keep the keywords around for a period of
> time to allow for an orderly transition and give teams time to adjust their
> processes.
>
>
>
> Second Proposal:
> This would be more of a hybrid of the current processes and the above.
> Keywords would be used by external/non-QA groups to request QA action and
> we flags would be used to track internal QA processes.  This would require
> less re-training and still reduce some of the bug spam by keeping the QA
> process changes easily filterable.
>
> In this proposal we would have the following:
> Keywords:
>   qawanted
>   qaurgent
>   steps-wanted
>   regression-window-wanted
>   testcase-wanted
>
> Flags:
>   QA-verify-[projectXX] [?|+|-]
>
>  
> What we would lose in this versions would be the ability to track QA
> requirements across projects and releases.
>
> In both cases we would drop the QA* white board tags entirely and the QA
> White board itself would be used for one off, project specific needs that
> are out side of the normal work flows.  We would also retain the use of the
> existing flags ,in-testsuite, in-moztrap and tracking-[projectXX]=verified,
> though we might need to require their usage more often the they are
> currently used.
>
> This proposal would require the least amount of change in our processes, but
> still give an easily filterable way of handling  the bug verification emails
> . Keyword changes would still have bug mail issues that will have to wait to
> be addressed by the Buzilla selective email filtering feature.
>
> In either case I would also propose a small process change where we do not
> mark a bug verified until it has all the QA-verify[projectXX] and
> tracking-[projectXX]=verify flags set
>
> Marc S.
>
>
>
> ----- Original Message -----
> From: "Robert Kaiser" <[hidden email]>
> To: [hidden email]
> Sent: Thursday, June 5, 2014 7:16:26 AM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
> [hidden email] schrieb:
> > The way I see it, we're trying to solve two problems:
> > A) Make QA as it exists in Bugzilla a lot less ambiguous
> > B) Reduce the noise in Bugzilla
> >
> > I'm not sure adding more descriptive keywords and/or moving things into
> > another text field (ie. QA Whiteboard) is going to solve either of those
> > problems.
>
> Another textfield won't solve A for sure, a fixed set of more
> descriptive keywords should help it though.
> For B, another textfield actually helps if it can be filtered out in
> bugmail, and the Bugzilla team is working on that. Also, as Matthew N.
> stated in this thread, keywords are already better than whiteboards in
> terms of bugmail because only the actually added/removed keywords are
> listed and not the whole whiteboard field before and after the change.
>
> > The more I think about this the more I think having a flag to indicate "QA
> > status" would be far more useful. It's certainly easier to query on
> > changes and allows us more granularity than flooding a text field. We
> > could even break that down into two QA flags, one to reflect status and
> > one to act as a requestor.
>
> I'd be just as happy with that but it's not easy for me to actually find
> a decent set of what should be in there, if we want to fold all our uses
> into it. Note that only one value of the field can be set at a given time.
>
> > If we've decided to stick to a combination of keywords and whiteboard tags
> > then I have no real preference. As long as it's something developers and
> > QA alike can buy in to.
>
> My proposal clearly goes for keywords for all the main uses. Whiteboards
> will never be 100% out of any picture just because temporary markers and
> notes are what they are entirely for. In our case, marking a testday
> that had found this or such things will always be a use of them. I
> personally think though that the separate QA whiteboard has been created
> prematurely and this shouldn't have been done without larger coordination.
>
> > FWIW, we're also getting pressure from the Firefox team to come up with
> > something they can start using ASAP.
>
> We first should care about what we want or need for our purposes and not
> let other teams create pressure. That said, I think there cannot be one
> solution that will make everyone completely happy.
>
> KaiRo
>
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Juan Becerra
In reply to this post by Marc Schifer
I think I prefer the simplicity of the second proposal, and I think it would help if we had one example to illustrate it.

juanb

----- Original Message -----
From: "Marc Schifer" <[hidden email]>
To: [hidden email]
Cc: "Gavin Sharp" <[hidden email]>
Sent: Monday, June 16, 2014 10:49:08 AM
Subject: Re: Desktop QA - Bugzilla keywords: Proposal


I have two proposals on how we can address the keyword/whiteboard flags and bug spam issues that have been discussed lately. The goal of these proposals is give QA a better way of tracking status of bugs, managing our work flows and being able to collect statistics about the health of the releases as well as give us an easy means of filtering out bug mail until the selective bug notification system is added to Bugzilla.


First proposal:
  Move all standard QA related keyword/whiteboard tags into flags.  This would allow us to track various QA activities by project/release and allow for easy bug mail filtering. Flags currently set an X Header, X-Bugzilla-Flags, in the mail sent that will be simple to have a client side filter on until the Bugzilla selective email filtering is completed.

In this proposal we would create the following flags:

  QA-status-[projectXX] [qawanted,qaurgent,qablocked]  =Triage verification need, +=Needs Verification, -=No Verification to be done
  QA-verify-[projectXX] [?|+|-]                        - ?=Triage verification need, +=Needs Verification, -=No Verification to be done  QA-steps-wanted[?|+|-]                               ?=Request for STR, +=STR Provided, -=Unable to provide STR (reason in comments)
  QA-regression-window-wanted [?|+|-]                  ?=Request for Regression Window  +=Window Provided, -=Unable to providw (reason in comments)
  QA-testcase-wanted [?|+|-]                           ?=Request for testcase  +=Testcase Provided (set proper intestcase flag), -=Unable to provide  (reason in comments)

In all cases a blank status would mean no action is requested

The QA-verify-[projectXX] flag would be release specific requesting a verification on which ever projects and releases were required + would indicate request for verification, - would indicate that no verification was going to be done and ? would be a triage requested flag.  We could leave off the ? and leave the initital state blank as well to indicate that a request for verification has been made and then the + would indicate that QA has the bug on their to be verified list. This would allow the various project teams to track which bugs may require

QA-status-[projectXX] flag would indicate which project/release a particular request or status applies too. For instance a bug filed originally against B2G for graphics issues may also require QA attention on Desktop or Android. In which case we could set flags such:
  QA-status-Firefox30 = qawanted
  QA-status-fennec30  = qawanted
  QA-status-b2fv-1.4  = qaurgent

The remaining flags QA-steps-wanted, QA-regression-window-wanted and QA-testcase-wanted would be non-project specific and simply flag the need for QA's input for the particular item.

All unset flags would be hidden by default as we do with the tracking flags to prevent flag spam in the UI. We should also have some sort of auto policing for older flags that would hide those that are more than say 4 releases old. This would then only show flags for those releases currently in flight to prevent long lived bugs from having long list of flags.

This process would give us a consistent workflow across all of QA and by using the existing features that let us track how long a flag spends in each state we can easily build dashboards to track our performance and the current status of each release.  

If we adopt this proposal, we will keep the keywords around for a period of time to allow for an orderly transition and give teams time to adjust their processes.



Second Proposal:
This would be more of a hybrid of the current processes and the above.   Keywords would be used by external/non-QA groups to request QA action and we flags would be used to track internal QA processes.  This would require less re-training and still reduce some of the bug spam by keeping the QA process changes easily filterable.

In this proposal we would have the following:
Keywords:
  qawanted
  qaurgent
  steps-wanted
  regression-window-wanted
  testcase-wanted

Flags:
  QA-verify-[projectXX] [?|+|-]

 
What we would lose in this versions would be the ability to track QA requirements across projects and releases.

In both cases we would drop the QA* white board tags entirely and the QA White board itself would be used for one off, project specific needs that are out side of the normal work flows.  We would also retain the use of the existing flags ,in-testsuite, in-moztrap and tracking-[projectXX]=verified, though we might need to require their usage more often the they are currently used.

This proposal would require the least amount of change in our processes, but still give an easily filterable way of handling  the bug verification emails . Keyword changes would still have bug mail issues that will have to wait to be addressed by the Buzilla selective email filtering feature.

In either case I would also propose a small process change where we do not mark a bug verified until it has all the QA-verify[projectXX] and tracking-[projectXX]=verify flags set  

Marc S.



----- Original Message -----
From: "Robert Kaiser" <[hidden email]>
To: [hidden email]
Sent: Thursday, June 5, 2014 7:16:26 AM
Subject: Re: Desktop QA - Bugzilla keywords: Proposal

[hidden email] schrieb:
> The way I see it, we're trying to solve two problems:
> A) Make QA as it exists in Bugzilla a lot less ambiguous
> B) Reduce the noise in Bugzilla
>
> I'm not sure adding more descriptive keywords and/or moving things into another text field (ie. QA Whiteboard) is going to solve either of those problems.

Another textfield won't solve A for sure, a fixed set of more
descriptive keywords should help it though.
For B, another textfield actually helps if it can be filtered out in
bugmail, and the Bugzilla team is working on that. Also, as Matthew N.
stated in this thread, keywords are already better than whiteboards in
terms of bugmail because only the actually added/removed keywords are
listed and not the whole whiteboard field before and after the change.

> The more I think about this the more I think having a flag to indicate "QA status" would be far more useful. It's certainly easier to query on changes and allows us more granularity than flooding a text field. We could even break that down into two QA flags, one to reflect status and one to act as a requestor.

I'd be just as happy with that but it's not easy for me to actually find
a decent set of what should be in there, if we want to fold all our uses
into it. Note that only one value of the field can be set at a given time.

> If we've decided to stick to a combination of keywords and whiteboard tags then I have no real preference. As long as it's something developers and QA alike can buy in to.

My proposal clearly goes for keywords for all the main uses. Whiteboards
will never be 100% out of any picture just because temporary markers and
notes are what they are entirely for. In our case, marking a testday
that had found this or such things will always be a use of them. I
personally think though that the separate QA whiteboard has been created
prematurely and this shouldn't have been done without larger coordination.

> FWIW, we're also getting pressure from the Firefox team to come up with something they can start using ASAP.

We first should care about what we want or need for our purposes and not
let other teams create pressure. That said, I think there cannot be one
solution that will make everyone completely happy.

KaiRo

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Anthony Hughes-3
In reply to this post by Marc Schifer
I think I prefer Proposal 2 as a good iterative step toward the ideal.

> QA-verify-[projectXX] [?|+|-]
What is *minus* intended to communicate? Does it mean can't/won't/failed verification or is it intended for a catch-all for negative states?

Anthony Hughes
Senior Test Engineer
Mozilla Corporation


----- Original Message -----

> From: "Marc Schifer" <[hidden email]>
> To: [hidden email]
> Cc: "Gavin Sharp" <[hidden email]>
> Sent: Monday, June 16, 2014 10:49:08 AM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
>
> I have two proposals on how we can address the keyword/whiteboard flags and
> bug spam issues that have been discussed lately. The goal of these proposals
> is give QA a better way of tracking status of bugs, managing our work flows
> and being able to collect statistics about the health of the releases as
> well as give us an easy means of filtering out bug mail until the selective
> bug notification system is added to Bugzilla.
>
>
> First proposal:
>   Move all standard QA related keyword/whiteboard tags into flags.  This
>   would allow us to track various QA activities by project/release and allow
>   for easy bug mail filtering. Flags currently set an X Header,
>   X-Bugzilla-Flags, in the mail sent that will be simple to have a client
>   side filter on until the Bugzilla selective email filtering is completed.
>
> In this proposal we would create the following flags:
>
>   QA-status-[projectXX] [qawanted,qaurgent,qablocked]  =Triage verification
>   need, +=Needs Verification, -=No Verification to be done
>   QA-verify-[projectXX] [?|+|-]                        - ?=Triage
>   verification need, +=Needs Verification, -=No Verification to be done
>   QA-steps-wanted[?|+|-]                               ?=Request for STR,
>   +=STR Provided, -=Unable to provide STR (reason in comments)
>   QA-regression-window-wanted [?|+|-]                  ?=Request for
>   Regression Window  +=Window Provided, -=Unable to providw (reason in
>   comments)
>   QA-testcase-wanted [?|+|-]                           ?=Request for testcase
>   +=Testcase Provided (set proper intestcase flag), -=Unable to provide
>   (reason in comments)
>
> In all cases a blank status would mean no action is requested
>
> The QA-verify-[projectXX] flag would be release specific requesting a
> verification on which ever projects and releases were required + would
> indicate request for verification, - would indicate that no verification was
> going to be done and ? would be a triage requested flag.  We could leave off
> the ? and leave the initital state blank as well to indicate that a request
> for verification has been made and then the + would indicate that QA has the
> bug on their to be verified list. This would allow the various project teams
> to track which bugs may require
>
> QA-status-[projectXX] flag would indicate which project/release a particular
> request or status applies too. For instance a bug filed originally against
> B2G for graphics issues may also require QA attention on Desktop or Android.
> In which case we could set flags such:
>   QA-status-Firefox30 = qawanted
>   QA-status-fennec30  = qawanted
>   QA-status-b2fv-1.4  = qaurgent
>
> The remaining flags QA-steps-wanted, QA-regression-window-wanted and
> QA-testcase-wanted would be non-project specific and simply flag the need
> for QA's input for the particular item.
>
> All unset flags would be hidden by default as we do with the tracking flags
> to prevent flag spam in the UI. We should also have some sort of auto
> policing for older flags that would hide those that are more than say 4
> releases old. This would then only show flags for those releases currently
> in flight to prevent long lived bugs from having long list of flags.
>
> This process would give us a consistent workflow across all of QA and by
> using the existing features that let us track how long a flag spends in each
> state we can easily build dashboards to track our performance and the
> current status of each release.
>
> If we adopt this proposal, we will keep the keywords around for a period of
> time to allow for an orderly transition and give teams time to adjust their
> processes.
>
>
>
> Second Proposal:
> This would be more of a hybrid of the current processes and the above.
> Keywords would be used by external/non-QA groups to request QA action and
> we flags would be used to track internal QA processes.  This would require
> less re-training and still reduce some of the bug spam by keeping the QA
> process changes easily filterable.
>
> In this proposal we would have the following:
> Keywords:
>   qawanted
>   qaurgent
>   steps-wanted
>   regression-window-wanted
>   testcase-wanted
>
> Flags:
>   QA-verify-[projectXX] [?|+|-]
>
>  
> What we would lose in this versions would be the ability to track QA
> requirements across projects and releases.
>
> In both cases we would drop the QA* white board tags entirely and the QA
> White board itself would be used for one off, project specific needs that
> are out side of the normal work flows.  We would also retain the use of the
> existing flags ,in-testsuite, in-moztrap and tracking-[projectXX]=verified,
> though we might need to require their usage more often the they are
> currently used.
>
> This proposal would require the least amount of change in our processes, but
> still give an easily filterable way of handling  the bug verification emails
> . Keyword changes would still have bug mail issues that will have to wait to
> be addressed by the Buzilla selective email filtering feature.
>
> In either case I would also propose a small process change where we do not
> mark a bug verified until it has all the QA-verify[projectXX] and
> tracking-[projectXX]=verify flags set
>
> Marc S.
>
>
>
> ----- Original Message -----
> From: "Robert Kaiser" <[hidden email]>
> To: [hidden email]
> Sent: Thursday, June 5, 2014 7:16:26 AM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
> [hidden email] schrieb:
> > The way I see it, we're trying to solve two problems:
> > A) Make QA as it exists in Bugzilla a lot less ambiguous
> > B) Reduce the noise in Bugzilla
> >
> > I'm not sure adding more descriptive keywords and/or moving things into
> > another text field (ie. QA Whiteboard) is going to solve either of those
> > problems.
>
> Another textfield won't solve A for sure, a fixed set of more
> descriptive keywords should help it though.
> For B, another textfield actually helps if it can be filtered out in
> bugmail, and the Bugzilla team is working on that. Also, as Matthew N.
> stated in this thread, keywords are already better than whiteboards in
> terms of bugmail because only the actually added/removed keywords are
> listed and not the whole whiteboard field before and after the change.
>
> > The more I think about this the more I think having a flag to indicate "QA
> > status" would be far more useful. It's certainly easier to query on
> > changes and allows us more granularity than flooding a text field. We
> > could even break that down into two QA flags, one to reflect status and
> > one to act as a requestor.
>
> I'd be just as happy with that but it's not easy for me to actually find
> a decent set of what should be in there, if we want to fold all our uses
> into it. Note that only one value of the field can be set at a given time.
>
> > If we've decided to stick to a combination of keywords and whiteboard tags
> > then I have no real preference. As long as it's something developers and
> > QA alike can buy in to.
>
> My proposal clearly goes for keywords for all the main uses. Whiteboards
> will never be 100% out of any picture just because temporary markers and
> notes are what they are entirely for. In our case, marking a testday
> that had found this or such things will always be a use of them. I
> personally think though that the separate QA whiteboard has been created
> prematurely and this shouldn't have been done without larger coordination.
>
> > FWIW, we're also getting pressure from the Firefox team to come up with
> > something they can start using ASAP.
>
> We first should care about what we want or need for our purposes and not
> let other teams create pressure. That said, I think there cannot be one
> solution that will make everyone completely happy.
>
> KaiRo
>
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Robert Kaiser
In reply to this post by Marc Schifer
Anthony Hughes schrieb:
> I think I prefer Proposal 2 as a good iterative step toward the ideal.
>
>> QA-verify-[projectXX] [?|+|-]
> What is *minus* intended to communicate? Does it mean can't/won't/failed verification or is it intended for a catch-all for negative states?

Marc said in his #1 proposal:
?=Triage verification need,
+=Needs Verification,
-=No Verification to be done

So, *minux* would mean what we are now using [qa-] for, i.e. we do not
need verification. Verification failed is signed with reopening the bug.
Verification done is marked with the bug status field/flags.

KaiRo

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Robert Kaiser
In reply to this post by Robert Kaiser
Marc Schifer schrieb:
> In either case I would also propose a small process change where we do not mark a bug verified until it has all the QA-verify[projectXX] and tracking-[projectXX]=verify flags set

I think it's best to keep the bug status field in line with the status
for whatever is set in "target milestone", as that matches how it's
being marked as "resolved" as well.


Overall, I find the #1 proposal somewhat appealing but it would require
a lot of additional flag work on the Bugzilla side, and either way I
think we should check with the Bugzilla team if the amount of flags we
need for those proposals is OK for them to do.

That said, I think it's easiest and best to start off with #2 and keep
#1 in mind for a possible future as the changes for #1 are a strict
superset to the changes for #2, right?


KaiRo
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Anthony Hughes-3
In reply to this post by Robert Kaiser
That makes sense, thanks for clarifying.

Anthony Hughes
Senior Test Engineer
Mozilla Corporation


----- Original Message -----

> From: "Robert Kaiser" <[hidden email]>
> To: [hidden email]
> Sent: Wednesday, June 18, 2014 2:06:37 PM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
> Anthony Hughes schrieb:
> > I think I prefer Proposal 2 as a good iterative step toward the ideal.
> >
> >> QA-verify-[projectXX] [?|+|-]
> > What is *minus* intended to communicate? Does it mean can't/won't/failed
> > verification or is it intended for a catch-all for negative states?
>
> Marc said in his #1 proposal:
> ?=Triage verification need,
> +=Needs Verification,
> -=No Verification to be done
>
> So, *minux* would mean what we are now using [qa-] for, i.e. we do not
> need verification. Verification failed is signed with reopening the bug.
> Verification done is marked with the bug status field/flags.
>
> KaiRo
>
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Marc Schifer
In reply to this post by Gavin Sharp-2

The idea behind this is when a bug affects multiple releases/projects we currently tend to mark the the bug as verified as soon as one of the releases/projects has tested it and verified it. This can result in a bug falling off the verification list for those other affected items unless the tracking flag has been set and someone in QA is watching those.   My proposal is to not set the bug status to Verified until all of the QA-Verify flags have been addressed and the appropriate tracking flags are also moved to verified.

Example a graphics bug has been filed originally on Desktop release 33 and it is later determined to also affect Fennec and B2G. We would then set the QA-Verify-Firefox33 +
QA-Verify-Fennec33 +
QA-Verify-B2G-v1.24 +


In common practice, when the bug is fixed, the reporter/QA owner would then verify the bug in Desktop and mark the bug status as verified. In this proposal, the over all bug would not be marked as verified until all of the QA-Verify* flags have been addressed and a corresponding status-* flag has been set to verified. This would help ensure the bug gets the proper attention on all the affected branches.

 
Marc S.
----- Original Message -----
From: "Gavin Sharp" <[hidden email]>
To: "Marc Schifer" <[hidden email]>
Cc: [hidden email]
Sent: Wednesday, June 18, 2014 12:02:42 PM
Subject: Re: Desktop QA - Bugzilla keywords: Proposal

Thanks for this, Marc.

I prefer proposal #2, since it seems simpler in scope, and less of a departure from our existing processes.

I'm not sure I understand this part, though:

> In either case I would also propose a small process change where we do not
> mark a bug verified until it has all the QA-verify[projectXX] and
> tracking-[projectXX]=verify flags set

Can you elaborate?

Gavin

----- Original Message -----

> From: "Marc Schifer" <[hidden email]>
> To: [hidden email]
> Cc: "Gavin Sharp" <[hidden email]>
> Sent: Monday, June 16, 2014 1:49:08 PM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
>
> I have two proposals on how we can address the keyword/whiteboard flags and
> bug spam issues that have been discussed lately. The goal of these proposals
> is give QA a better way of tracking status of bugs, managing our work flows
> and being able to collect statistics about the health of the releases as
> well as give us an easy means of filtering out bug mail until the selective
> bug notification system is added to Bugzilla.
>
>
> First proposal:
>   Move all standard QA related keyword/whiteboard tags into flags.  This
>   would allow us to track various QA activities by project/release and allow
>   for easy bug mail filtering. Flags currently set an X Header,
>   X-Bugzilla-Flags, in the mail sent that will be simple to have a client
>   side filter on until the Bugzilla selective email filtering is completed.
>
> In this proposal we would create the following flags:
>
>   QA-status-[projectXX] [qawanted,qaurgent,qablocked]  =Triage verification
>   need, +=Needs Verification, -=No Verification to be done
>   QA-verify-[projectXX] [?|+|-]                        - ?=Triage
>   verification need, +=Needs Verification, -=No Verification to be done
>   QA-steps-wanted[?|+|-]                               ?=Request for STR,
>   +=STR Provided, -=Unable to provide STR (reason in comments)
>   QA-regression-window-wanted [?|+|-]                  ?=Request for
>   Regression Window  +=Window Provided, -=Unable to providw (reason in
>   comments)
>   QA-testcase-wanted [?|+|-]                           ?=Request for testcase
>   +=Testcase Provided (set proper intestcase flag), -=Unable to provide
>   (reason in comments)
>
> In all cases a blank status would mean no action is requested
>
> The QA-verify-[projectXX] flag would be release specific requesting a
> verification on which ever projects and releases were required + would
> indicate request for verification, - would indicate that no verification was
> going to be done and ? would be a triage requested flag.  We could leave off
> the ? and leave the initital state blank as well to indicate that a request
> for verification has been made and then the + would indicate that QA has the
> bug on their to be verified list. This would allow the various project teams
> to track which bugs may require
>
> QA-status-[projectXX] flag would indicate which project/release a particular
> request or status applies too. For instance a bug filed originally against
> B2G for graphics issues may also require QA attention on Desktop or Android.
> In which case we could set flags such:
>   QA-status-Firefox30 = qawanted
>   QA-status-fennec30  = qawanted
>   QA-status-b2fv-1.4  = qaurgent
>
> The remaining flags QA-steps-wanted, QA-regression-window-wanted and
> QA-testcase-wanted would be non-project specific and simply flag the need
> for QA's input for the particular item.
>
> All unset flags would be hidden by default as we do with the tracking flags
> to prevent flag spam in the UI. We should also have some sort of auto
> policing for older flags that would hide those that are more than say 4
> releases old. This would then only show flags for those releases currently
> in flight to prevent long lived bugs from having long list of flags.
>
> This process would give us a consistent workflow across all of QA and by
> using the existing features that let us track how long a flag spends in each
> state we can easily build dashboards to track our performance and the
> current status of each release.
>
> If we adopt this proposal, we will keep the keywords around for a period of
> time to allow for an orderly transition and give teams time to adjust their
> processes.
>
>
>
> Second Proposal:
> This would be more of a hybrid of the current processes and the above.
> Keywords would be used by external/non-QA groups to request QA action and
> we flags would be used to track internal QA processes.  This would require
> less re-training and still reduce some of the bug spam by keeping the QA
> process changes easily filterable.
>
> In this proposal we would have the following:
> Keywords:
>   qawanted
>   qaurgent
>   steps-wanted
>   regression-window-wanted
>   testcase-wanted
>
> Flags:
>   QA-verify-[projectXX] [?|+|-]
>
>  
> What we would lose in this versions would be the ability to track QA
> requirements across projects and releases.
>
> In both cases we would drop the QA* white board tags entirely and the QA
> White board itself would be used for one off, project specific needs that
> are out side of the normal work flows.  We would also retain the use of the
> existing flags ,in-testsuite, in-moztrap and tracking-[projectXX]=verified,
> though we might need to require their usage more often the they are
> currently used.
>
> This proposal would require the least amount of change in our processes, but
> still give an easily filterable way of handling  the bug verification emails
> . Keyword changes would still have bug mail issues that will have to wait to
> be addressed by the Buzilla selective email filtering feature.
>
> In either case I would also propose a small process change where we do not
> mark a bug verified until it has all the QA-verify[projectXX] and
> tracking-[projectXX]=verify flags set
>
> Marc S.
>
>
>
> ----- Original Message -----
> From: "Robert Kaiser" <[hidden email]>
> To: [hidden email]
> Sent: Thursday, June 5, 2014 7:16:26 AM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
> [hidden email] schrieb:
> > The way I see it, we're trying to solve two problems:
> > A) Make QA as it exists in Bugzilla a lot less ambiguous
> > B) Reduce the noise in Bugzilla
> >
> > I'm not sure adding more descriptive keywords and/or moving things into
> > another text field (ie. QA Whiteboard) is going to solve either of those
> > problems.
>
> Another textfield won't solve A for sure, a fixed set of more
> descriptive keywords should help it though.
> For B, another textfield actually helps if it can be filtered out in
> bugmail, and the Bugzilla team is working on that. Also, as Matthew N.
> stated in this thread, keywords are already better than whiteboards in
> terms of bugmail because only the actually added/removed keywords are
> listed and not the whole whiteboard field before and after the change.
>
> > The more I think about this the more I think having a flag to indicate "QA
> > status" would be far more useful. It's certainly easier to query on
> > changes and allows us more granularity than flooding a text field. We
> > could even break that down into two QA flags, one to reflect status and
> > one to act as a requestor.
>
> I'd be just as happy with that but it's not easy for me to actually find
> a decent set of what should be in there, if we want to fold all our uses
> into it. Note that only one value of the field can be set at a given time.
>
> > If we've decided to stick to a combination of keywords and whiteboard tags
> > then I have no real preference. As long as it's something developers and
> > QA alike can buy in to.
>
> My proposal clearly goes for keywords for all the main uses. Whiteboards
> will never be 100% out of any picture just because temporary markers and
> notes are what they are entirely for. In our case, marking a testday
> that had found this or such things will always be a use of them. I
> personally think though that the separate QA whiteboard has been created
> prematurely and this shouldn't have been done without larger coordination.
>
> > FWIW, we're also getting pressure from the Firefox team to come up with
> > something they can start using ASAP.
>
> We first should care about what we want or need for our purposes and not
> let other teams create pressure. That said, I think there cannot be one
> solution that will make everyone completely happy.
>
> KaiRo
>
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
12