Desktop QA - Bugzilla keywords: Proposal

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Gavin Sharp-2
The VERIFIED bug state to me has always meant "verified on trunk in at least one affected product" (for a Gecko bug that often means desktop Firefox, but could also mean b2g or fennec). It sounds like you're proposing changing its meaning to "verified on trunk and all relevant branches it was uplifted to, in all affected products". What does that definition change get us? It sounds like the intent is to make it easier to distinguish bugs that aren't "fully verified", at the cost of losing the "was verified on trunk" annotation (which can't easily be determined solely by looking at the status flags).

Since I think that currently the vast majority of our bugs never reach the "fully verified" state (and it would probably not be cost effective to shoot for 100% "full verification" across all bugs), I wonder if there's much value in making it easier to find them - particularly since it's currently possible to construct queries to identify them in other - granted, less reliable - ways (e.g. searching for verified bugs without all the status-X: verified flags).

Gavin

----- Original Message -----

> From: "Marc Schifer" <[hidden email]>
> To: "Gavin Sharp" <[hidden email]>
> Cc: [hidden email]
> Sent: Wednesday, June 18, 2014 5:13:31 PM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
>
> The idea behind this is when a bug affects multiple releases/projects we
> currently tend to mark the the bug as verified as soon as one of the
> releases/projects has tested it and verified it. This can result in a bug
> falling off the verification list for those other affected items unless the
> tracking flag has been set and someone in QA is watching those.   My
> proposal is to not set the bug status to Verified until all of the QA-Verify
> flags have been addressed and the appropriate tracking flags are also moved
> to verified.
>
> Example a graphics bug has been filed originally on Desktop release 33 and it
> is later determined to also affect Fennec and B2G. We would then set the
> QA-Verify-Firefox33 +
> QA-Verify-Fennec33 +
> QA-Verify-B2G-v1.24 +
>
>
> In common practice, when the bug is fixed, the reporter/QA owner would then
> verify the bug in Desktop and mark the bug status as verified. In this
> proposal, the over all bug would not be marked as verified until all of the
> QA-Verify* flags have been addressed and a corresponding status-* flag has
> been set to verified. This would help ensure the bug gets the proper
> attention on all the affected branches.
>
>  
> Marc S.
> ----- Original Message -----
> From: "Gavin Sharp" <[hidden email]>
> To: "Marc Schifer" <[hidden email]>
> Cc: [hidden email]
> Sent: Wednesday, June 18, 2014 12:02:42 PM
> Subject: Re: Desktop QA - Bugzilla keywords: Proposal
>
> Thanks for this, Marc.
>
> I prefer proposal #2, since it seems simpler in scope, and less of a
> departure from our existing processes.
>
> I'm not sure I understand this part, though:
>
> > In either case I would also propose a small process change where we do not
> > mark a bug verified until it has all the QA-verify[projectXX] and
> > tracking-[projectXX]=verify flags set
>
> Can you elaborate?
>
> Gavin
>
> ----- Original Message -----
> > From: "Marc Schifer" <[hidden email]>
> > To: [hidden email]
> > Cc: "Gavin Sharp" <[hidden email]>
> > Sent: Monday, June 16, 2014 1:49:08 PM
> > Subject: Re: Desktop QA - Bugzilla keywords: Proposal
> >
> >
> > I have two proposals on how we can address the keyword/whiteboard flags and
> > bug spam issues that have been discussed lately. The goal of these
> > proposals
> > is give QA a better way of tracking status of bugs, managing our work flows
> > and being able to collect statistics about the health of the releases as
> > well as give us an easy means of filtering out bug mail until the selective
> > bug notification system is added to Bugzilla.
> >
> >
> > First proposal:
> >   Move all standard QA related keyword/whiteboard tags into flags.  This
> >   would allow us to track various QA activities by project/release and
> >   allow
> >   for easy bug mail filtering. Flags currently set an X Header,
> >   X-Bugzilla-Flags, in the mail sent that will be simple to have a client
> >   side filter on until the Bugzilla selective email filtering is completed.
> >
> > In this proposal we would create the following flags:
> >
> >   QA-status-[projectXX] [qawanted,qaurgent,qablocked]  =Triage verification
> >   need, +=Needs Verification, -=No Verification to be done
> >   QA-verify-[projectXX] [?|+|-]                        - ?=Triage
> >   verification need, +=Needs Verification, -=No Verification to be done
> >   QA-steps-wanted[?|+|-]                               ?=Request for STR,
> >   +=STR Provided, -=Unable to provide STR (reason in comments)
> >   QA-regression-window-wanted [?|+|-]                  ?=Request for
> >   Regression Window  +=Window Provided, -=Unable to providw (reason in
> >   comments)
> >   QA-testcase-wanted [?|+|-]                           ?=Request for
> >   testcase
> >   +=Testcase Provided (set proper intestcase flag), -=Unable to provide
> >   (reason in comments)
> >
> > In all cases a blank status would mean no action is requested
> >
> > The QA-verify-[projectXX] flag would be release specific requesting a
> > verification on which ever projects and releases were required + would
> > indicate request for verification, - would indicate that no verification
> > was
> > going to be done and ? would be a triage requested flag.  We could leave
> > off
> > the ? and leave the initital state blank as well to indicate that a request
> > for verification has been made and then the + would indicate that QA has
> > the
> > bug on their to be verified list. This would allow the various project
> > teams
> > to track which bugs may require
> >
> > QA-status-[projectXX] flag would indicate which project/release a
> > particular
> > request or status applies too. For instance a bug filed originally against
> > B2G for graphics issues may also require QA attention on Desktop or
> > Android.
> > In which case we could set flags such:
> >   QA-status-Firefox30 = qawanted
> >   QA-status-fennec30  = qawanted
> >   QA-status-b2fv-1.4  = qaurgent
> >
> > The remaining flags QA-steps-wanted, QA-regression-window-wanted and
> > QA-testcase-wanted would be non-project specific and simply flag the need
> > for QA's input for the particular item.
> >
> > All unset flags would be hidden by default as we do with the tracking flags
> > to prevent flag spam in the UI. We should also have some sort of auto
> > policing for older flags that would hide those that are more than say 4
> > releases old. This would then only show flags for those releases currently
> > in flight to prevent long lived bugs from having long list of flags.
> >
> > This process would give us a consistent workflow across all of QA and by
> > using the existing features that let us track how long a flag spends in
> > each
> > state we can easily build dashboards to track our performance and the
> > current status of each release.
> >
> > If we adopt this proposal, we will keep the keywords around for a period of
> > time to allow for an orderly transition and give teams time to adjust their
> > processes.
> >
> >
> >
> > Second Proposal:
> > This would be more of a hybrid of the current processes and the above.
> > Keywords would be used by external/non-QA groups to request QA action and
> > we flags would be used to track internal QA processes.  This would require
> > less re-training and still reduce some of the bug spam by keeping the QA
> > process changes easily filterable.
> >
> > In this proposal we would have the following:
> > Keywords:
> >   qawanted
> >   qaurgent
> >   steps-wanted
> >   regression-window-wanted
> >   testcase-wanted
> >
> > Flags:
> >   QA-verify-[projectXX] [?|+|-]
> >
> >  
> > What we would lose in this versions would be the ability to track QA
> > requirements across projects and releases.
> >
> > In both cases we would drop the QA* white board tags entirely and the QA
> > White board itself would be used for one off, project specific needs that
> > are out side of the normal work flows.  We would also retain the use of the
> > existing flags ,in-testsuite, in-moztrap and tracking-[projectXX]=verified,
> > though we might need to require their usage more often the they are
> > currently used.
> >
> > This proposal would require the least amount of change in our processes,
> > but
> > still give an easily filterable way of handling  the bug verification
> > emails
> > . Keyword changes would still have bug mail issues that will have to wait
> > to
> > be addressed by the Buzilla selective email filtering feature.
> >
> > In either case I would also propose a small process change where we do not
> > mark a bug verified until it has all the QA-verify[projectXX] and
> > tracking-[projectXX]=verify flags set
> >
> > Marc S.
> >
> >
> >
> > ----- Original Message -----
> > From: "Robert Kaiser" <[hidden email]>
> > To: [hidden email]
> > Sent: Thursday, June 5, 2014 7:16:26 AM
> > Subject: Re: Desktop QA - Bugzilla keywords: Proposal
> >
> > [hidden email] schrieb:
> > > The way I see it, we're trying to solve two problems:
> > > A) Make QA as it exists in Bugzilla a lot less ambiguous
> > > B) Reduce the noise in Bugzilla
> > >
> > > I'm not sure adding more descriptive keywords and/or moving things into
> > > another text field (ie. QA Whiteboard) is going to solve either of those
> > > problems.
> >
> > Another textfield won't solve A for sure, a fixed set of more
> > descriptive keywords should help it though.
> > For B, another textfield actually helps if it can be filtered out in
> > bugmail, and the Bugzilla team is working on that. Also, as Matthew N.
> > stated in this thread, keywords are already better than whiteboards in
> > terms of bugmail because only the actually added/removed keywords are
> > listed and not the whole whiteboard field before and after the change.
> >
> > > The more I think about this the more I think having a flag to indicate
> > > "QA
> > > status" would be far more useful. It's certainly easier to query on
> > > changes and allows us more granularity than flooding a text field. We
> > > could even break that down into two QA flags, one to reflect status and
> > > one to act as a requestor.
> >
> > I'd be just as happy with that but it's not easy for me to actually find
> > a decent set of what should be in there, if we want to fold all our uses
> > into it. Note that only one value of the field can be set at a given time.
> >
> > > If we've decided to stick to a combination of keywords and whiteboard
> > > tags
> > > then I have no real preference. As long as it's something developers and
> > > QA alike can buy in to.
> >
> > My proposal clearly goes for keywords for all the main uses. Whiteboards
> > will never be 100% out of any picture just because temporary markers and
> > notes are what they are entirely for. In our case, marking a testday
> > that had found this or such things will always be a use of them. I
> > personally think though that the separate QA whiteboard has been created
> > prematurely and this shouldn't have been done without larger coordination.
> >
> > > FWIW, we're also getting pressure from the Firefox team to come up with
> > > something they can start using ASAP.
> >
> > We first should care about what we want or need for our purposes and not
> > let other teams create pressure. That said, I think there cannot be one
> > solution that will make everyone completely happy.
> >
> > KaiRo
> >
> > _______________________________________________
> > dev-quality mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-quality
> >
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Henrik Skupin-3
Gavin Sharp wrote on 06/18/2014 11:56 PM:

> relevant branches it was uplifted to, in all affected products". What
> does that definition change get us? It sounds like the intent is to
> make it easier to distinguish bugs that aren't "fully verified", at
> the cost of losing the "was verified on trunk" annotation (which
> can't easily be determined solely by looking at the status flags).

We have two options here in how we can determine where it has been fixed
and verified:

1. By using the bug status and comparing it with the target milestone.
Meanwhile we are good in setting the target milestone, so the
relationship should always exist. With this information we know which
trunk version got the fix/enhancement.

2. By using the status flags like status-firefox32. The benefit of those
flags is that you can directly see the version when it has been landed,
and the status.

Option 2 I like more, given that you will be able to see this kind of
information also for older versions (branches). And that's also for the
disabled and wontfix status.

So what does Bugzilla say:

> FIXED
>     A fix for this bug is checked into the tree and tested.

We may take this as landed on trunk, and tested by one of our various
test suites on buildbot.

> RESOLVED
> A resolution has been performed, and it is awaiting verification by
> QA. From here bugs are either reopened and given some open status, or
> are verified by QA and marked VERIFIED.
>
> VERIFIED
> QA has looked at the bug and the resolution and agrees that the
appropriate resolution has been taken. Any zombie bugs who choose to
walk the earth again must do so by becoming REOPENED.

It's actually not that clearly described, and can lead into different
interpretations in how it can be used. So I like Marc's idea by using
the status to indicate that the bug has been completely verified, and
not only on trunk.

> Since I think that currently the vast majority of our bugs never
> reach the "fully verified" state (and it would probably not be cost
> effective to shoot for 100% "full verification" across all bugs), I
> wonder if there's much value in making it easier to find them -
> particularly since it's currently possible to construct queries to
> identify them in other - granted, less reliable - ways (e.g.
> searching for verified bugs without all the status-X: verified
> flags).

This may depend on how everyone of us would interpret the bug status.
Even when lots of bugs may not reach that final verification state as of
now, it would be good to see how successful QA was for verifications in
the long term. Right now that part is somewhat hard to measure.

--
Henrik Skupin
Senior Test Engineer
Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Henrik Skupin-3
In reply to this post by Robert Kaiser
Robert Kaiser wrote on 06/18/2014 11:10 PM:

> That said, I think it's easiest and best to start off with #2 and keep
> #1 in mind for a possible future as the changes for #1 are a strict
> superset to the changes for #2, right?

I agree with Robert's idea here. Also in terms that we should talk to
the Bugzilla admins in how much work that would mean for them. Also the
names of the flags will become very long, so one proposal would be to
add a QA section like "QA flags", which would kill the "QA-" prefix.

QA flags:
        status-projectXX
        [...]

I'm not sure about regression-window-wanted and testcase-wanted. This
looks like a duplication to me, and we should keep it as keyword only.
But the benefit of this proposal would be (as Marc told me) that we have
time information when it has been updated.

--
Henrik Skupin
Senior Test Engineer
Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

RE: Desktop QA - Bugzilla keywords: Proposal

Florin Mezei
I like #1 best, but I also don't mind starting with #2 and see how that goes
(as Robert suggested).

With regards to Marc's suggestion to "not mark a bug verified until it has
all the QA-verify[projectXX] and tracking-[projectXX]=verify flags set", I
actually support Mihaela's suggestion to "continue to mark the bug
"Verified" when it is verified on the target milestone version and add the
"Closed" status in Bugzilla for when it is verified on all the other
tracking versions and the flags are set". I'm not sure why the "Closed"
status is not used, since it would fit perfectly: Verified=Partially
Verified (there are still some versions to verify), Closed=Fully Verified on
all versions (no more work needed on the bug).

Regards,
Florin.

-----Original Message-----
From: dev-quality
[mailto:dev-quality-bounces+florin.mezei=[hidden email]]
On Behalf Of Henrik Skupin
Sent: Thursday, June 19, 2014 9:35 AM
To: Robert Kaiser; [hidden email]
Subject: Re: Desktop QA - Bugzilla keywords: Proposal

Robert Kaiser wrote on 06/18/2014 11:10 PM:

> That said, I think it's easiest and best to start off with #2 and keep
> #1 in mind for a possible future as the changes for #1 are a strict
> superset to the changes for #2, right?

I agree with Robert's idea here. Also in terms that we should talk to the
Bugzilla admins in how much work that would mean for them. Also the names of
the flags will become very long, so one proposal would be to add a QA
section like "QA flags", which would kill the "QA-" prefix.

QA flags:
        status-projectXX
        [...]

I'm not sure about regression-window-wanted and testcase-wanted. This looks
like a duplication to me, and we should keep it as keyword only.
But the benefit of this proposal would be (as Marc told me) that we have
time information when it has been updated.

--
Henrik Skupin
Senior Test Engineer
Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality


_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Liz Henry
In reply to this post by Juan Becerra
On 6/19/14 8:59 AM, Liz Henry wrote:

>
> The first line from the first proposal didn't make sense to me, without
> taking out the last part of the line from "=Triage" onwards.
>
>> QA-status-[projectXX] [qawanted,qaurgent,qablocked]  =Triage
>> verification need, +=Needs Verification, -=No Verification to be
>> done
>
> So, there would be
>

Oops.  That was meant to be a draft.  :)

I can see the value of the first proposal. It seems a little confusing
to implement, and would need some work in BMO.  Its main advantage seems
to be to let us easily track how long particular flags are assigned, ie
how much time it takes to move a bug from one state to another.     We
may be able to do this through ElasticSearch reports but it is harder to
track when a lot of states are in the whiteboard field.


I like the second proposal's clear division between "stuff that external
teams are requesting of QA" and flags that QA puts onto bugs for its own
purposes (including communicating with release management)

I can see a lot of potential for using the QA whiteboard for queries and
lists we use for testdays, specific features or platforms, and so on.
Like Marc said, good for one-off projects, where developers don't
necessarily need to be involved & can avoid the extra bugmail.

Best,

Liz








--
Liz Henry
[hidden email]
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Marc Schifer

Thank you for all of your feed back. This has been a good discussion. It looks like everyone is in favor of the second option with a few votes for maybe exploring option one at a later date if this works and we still have other issues to address.

So the proposal going forward is:
Use the keywords: qawanted, qaurgent, steps-wanted, regression-window-wanted and testcase-wanted to signal QA attention is requested. Drop the verifyme keyword and replace it with a QA-verify-[projectXX]=[?|+|-0 flag to identify the verification status or needs and no change to how and when we set the bug status to Verified.

A question was raised about how exactly we may want to implement the flag. I had originally conceived it as one flag for each project/release which differs slightly how the current tracking/status flags are implemented, one for each train and one for each base project. I am still in favor of the individual flag for each project/train to give us the best visibility about what is required but it may lead to a large number of flags being required for each bug. If we have some intelligent automagic to hide those no longer relevant i think it will be manageable, but I would rather address this upfront before implementation than after the fact.

The difference would be between:
QA-verify-firefox31=-
QA-verify-firefox32=+
QA-verify-firefox33=+
QA-verify-fennec31=-
QA-verify-fennec32=+
QA-verify-fennec33=+
QA-verify-b2g-v1.3=-
QA-verify-b2g-v1.4=+
QA-verify-b2g-v2.0=+

And:
QA-verify-firefox31=-
QA-verify-firefox32=+
QA-verify-firefox33=+
QA-verify-fennec=+
QA-verify-b2g=+


I would like to close this out and hand it over to the Bugzilla team in the next couple of days.
Marc S.



----- Original Message -----
From: "Liz Henry" <[hidden email]>
To: [hidden email]
Sent: Thursday, June 19, 2014 2:14:51 PM
Subject: Re: Desktop QA - Bugzilla keywords: Proposal

On 6/19/14 8:59 AM, Liz Henry wrote:

>
> The first line from the first proposal didn't make sense to me, without
> taking out the last part of the line from "=Triage" onwards.
>
>> QA-status-[projectXX] [qawanted,qaurgent,qablocked]  =Triage
>> verification need, +=Needs Verification, -=No Verification to be
>> done
>
> So, there would be
>

Oops.  That was meant to be a draft.  :)

I can see the value of the first proposal. It seems a little confusing
to implement, and would need some work in BMO.  Its main advantage seems
to be to let us easily track how long particular flags are assigned, ie
how much time it takes to move a bug from one state to another.     We
may be able to do this through ElasticSearch reports but it is harder to
track when a lot of states are in the whiteboard field.


I like the second proposal's clear division between "stuff that external
teams are requesting of QA" and flags that QA puts onto bugs for its own
purposes (including communicating with release management)

I can see a lot of potential for using the QA whiteboard for queries and
lists we use for testdays, specific features or platforms, and so on.
Like Marc said, good for one-off projects, where developers don't
necessarily need to be involved & can avoid the extra bugmail.

Best,

Liz








--
Liz Henry
[hidden email]
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Desktop QA - Bugzilla keywords: Proposal

Robert Kaiser
In reply to this post by Liz Henry
Marc Schifer schrieb:
> So the proposal going forward is:
> [...] Drop the verifyme keyword and replace it with a QA-verify-[projectXX]=[?|+|-0 flag to identify the verification status or needs and no change to how and when we set the bug status to Verified.
>
> A question was raised about how exactly we may want to implement the flag. I had originally conceived it as one flag for each project/release which differs slightly how the current tracking/status flags are implemented, one for each train and one for each base project. I am still in favor of the individual flag for each project/train to give us the best visibility about what is required but it may lead to a large number of flags being required for each bug. If we have some intelligent automagic to hide those no longer relevant i think it will be manageable, but I would rather address this upfront before implementation than after the fact.

Bugzilla only hides non-set flags, it always shows all those that are
set to anything but empty.

That said, I have been thinking about this some more and I really start
to wonder why we even need per-train flags there. All we indicate there
is if this bug needs to have verification happening or not, and I think
that's actually generic to the bug at hand and not specific to a train.
What is specific to a train is if the verification has been actually
done (which marks the per-train status flag as "verified" and/or the
main bug status to "verified") or if verification failed (which should
flip the per-train status flag back from "fixed" to "affected" next to
reopening the bug).

So, why again do we even need the QA-verify flag per train?

(Note that per product makes some sense to me as there's easily things
we need no verification for on one product but can/must do it on another.)

KaiRo

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
12