Triage Plan for Firefox Components

classic Classic list List threaded Threaded
40 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Triage Plan for Firefox Components

Emma Humphries
tl;dr

In Quarter Two I'm implementing the work we’ve been doing to improve
triage, make actionable decisions on new bugs, and prevent us from shipping
regressions in Firefox.

Today I’m asking for feedback on the plan which is posted at:

https://docs.google.com/document/d/1FFrtS0u6gNBE1mxsGJA9JLseJ_U6tW-1NJvHMq551ko

Allowing bugs to sit around without a decision on what we will do about
them sends the wrong message to Mozillans about how we treat bugs, how we
value their involvement, and reduces quality.

The Firefox quality team (myself, Mike Hoye, Ryan VanderMeulen, Mark Cote,
and Benjamin Smedberg) want to make better assertions about the quality of
our releases by giving you tools to make clear decisions about which bugs
must be fixed for each release (urgent) and actively tracking those bugs.
What We Learned From The Pilot Program

During the past 6 weeks, we have prototyped and tested a triage process
with the DOM, Hello, and Developer Tools teams.

Andrew Overholt, who participated in the pilot for the DOM team, said, “A
consistent bug triage process can help us spread the load of watching
incoming bugs and help avoid issues falling through the cracks."

During the pilot, the DOM team uncovered critical bugs quickly so that
people could be assigned to them.

The pilot groups also found that the triage process needs to be fast and
have tooling to make going through bugs fast. It’s easy to fall behind on
triage for a component, but if you stay up to date it will take no more
than 15 minutes a day.

You can find the bugs we triaged during the pilot by looking for whiteboard
tags containing ‘btpp-’.

It is also important to have consistent, shared definitions for regression
across components so triagers do not waste effort on mis-labeled bugs.
Comments?

I am posting this plan now for comment over the next week. I intend to
finalize the triage plan for implementation by Tuesday, April 5th. Feedback
and questions are welcome on the document, privately via email or IRC
(where I’m emceeaich) or on the [hidden email] mailing list.
Timeline

January: finish finding component responsible parties

February: pilot review of NEW bugs with four groups of components, draft
new process

Now: comment period for new process, finalize process

Q2: implement new process across all components involved in shipping Firefox
Q3: all newly triaged bugs following the new process

-- Emma Humphries, Bugmaster
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Eric Rescorla
I'm trying to figure out the scope of this proposal. Are you expecting it
to apply merely to Firefox or to Gecko as well?

-Ekr


On Tue, Mar 29, 2016 at 1:07 PM, Emma Humphries <[hidden email]> wrote:

> tl;dr
>
> In Quarter Two I'm implementing the work we’ve been doing to improve
> triage, make actionable decisions on new bugs, and prevent us from shipping
> regressions in Firefox.
>
> Today I’m asking for feedback on the plan which is posted at:
>
>
> https://docs.google.com/document/d/1FFrtS0u6gNBE1mxsGJA9JLseJ_U6tW-1NJvHMq551ko
>
> Allowing bugs to sit around without a decision on what we will do about
> them sends the wrong message to Mozillans about how we treat bugs, how we
> value their involvement, and reduces quality.
>
> The Firefox quality team (myself, Mike Hoye, Ryan VanderMeulen, Mark Cote,
> and Benjamin Smedberg) want to make better assertions about the quality of
> our releases by giving you tools to make clear decisions about which bugs
> must be fixed for each release (urgent) and actively tracking those bugs.
> What We Learned From The Pilot Program
>
> During the past 6 weeks, we have prototyped and tested a triage process
> with the DOM, Hello, and Developer Tools teams.
>
> Andrew Overholt, who participated in the pilot for the DOM team, said, “A
> consistent bug triage process can help us spread the load of watching
> incoming bugs and help avoid issues falling through the cracks."
>
> During the pilot, the DOM team uncovered critical bugs quickly so that
> people could be assigned to them.
>
> The pilot groups also found that the triage process needs to be fast and
> have tooling to make going through bugs fast. It’s easy to fall behind on
> triage for a component, but if you stay up to date it will take no more
> than 15 minutes a day.
>
> You can find the bugs we triaged during the pilot by looking for
> whiteboard tags containing ‘btpp-’.
>
> It is also important to have consistent, shared definitions for regression
> across components so triagers do not waste effort on mis-labeled bugs.
> Comments?
>
> I am posting this plan now for comment over the next week. I intend to
> finalize the triage plan for implementation by Tuesday, April 5th. Feedback
> and questions are welcome on the document, privately via email or IRC
> (where I’m emceeaich) or on the [hidden email] mailing list.
> Timeline
>
> January: finish finding component responsible parties
>
> February: pilot review of NEW bugs with four groups of components, draft
> new process
>
> Now: comment period for new process, finalize process
>
> Q2: implement new process across all components involved in shipping
> Firefox
> Q3: all newly triaged bugs following the new process
>
> -- Emma Humphries, Bugmaster
>
> _______________________________________________
> firefox-dev mailing list
> [hidden email]
> https://mail.mozilla.org/listinfo/firefox-dev
>
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Emma Humphries
Components that feed into FFx, so that's Core, Toolbox, Firefox, Fennec and
the pieces of platform we're using to ship.

While there's not a 'Gecko' component in bugzilla, it does cover the
components there which are Gecko.

-- Emma

On Tue, Mar 29, 2016 at 3:16 PM, Eric Rescorla <[hidden email]> wrote:

> I'm trying to figure out the scope of this proposal. Are you expecting it
> to apply merely to Firefox or to Gecko as well?
>
> -Ekr
>
>
> On Tue, Mar 29, 2016 at 1:07 PM, Emma Humphries <[hidden email]> wrote:
>
>> tl;dr
>>
>> In Quarter Two I'm implementing the work we’ve been doing to improve
>> triage, make actionable decisions on new bugs, and prevent us from shipping
>> regressions in Firefox.
>>
>> Today I’m asking for feedback on the plan which is posted at:
>>
>>
>> https://docs.google.com/document/d/1FFrtS0u6gNBE1mxsGJA9JLseJ_U6tW-1NJvHMq551ko
>>
>> Allowing bugs to sit around without a decision on what we will do about
>> them sends the wrong message to Mozillans about how we treat bugs, how we
>> value their involvement, and reduces quality.
>>
>> The Firefox quality team (myself, Mike Hoye, Ryan VanderMeulen, Mark
>> Cote, and Benjamin Smedberg) want to make better assertions about the
>> quality of our releases by giving you tools to make clear decisions about
>> which bugs must be fixed for each release (urgent) and actively tracking
>> those bugs.
>> What We Learned From The Pilot Program
>>
>> During the past 6 weeks, we have prototyped and tested a triage process
>> with the DOM, Hello, and Developer Tools teams.
>>
>> Andrew Overholt, who participated in the pilot for the DOM team, said, “A
>> consistent bug triage process can help us spread the load of watching
>> incoming bugs and help avoid issues falling through the cracks."
>>
>> During the pilot, the DOM team uncovered critical bugs quickly so that
>> people could be assigned to them.
>>
>> The pilot groups also found that the triage process needs to be fast and
>> have tooling to make going through bugs fast. It’s easy to fall behind on
>> triage for a component, but if you stay up to date it will take no more
>> than 15 minutes a day.
>>
>> You can find the bugs we triaged during the pilot by looking for
>> whiteboard tags containing ‘btpp-’.
>>
>> It is also important to have consistent, shared definitions for
>> regression across components so triagers do not waste effort on mis-labeled
>> bugs.
>> Comments?
>>
>> I am posting this plan now for comment over the next week. I intend to
>> finalize the triage plan for implementation by Tuesday, April 5th. Feedback
>> and questions are welcome on the document, privately via email or IRC
>> (where I’m emceeaich) or on the [hidden email] mailing list.
>> Timeline
>>
>> January: finish finding component responsible parties
>>
>> February: pilot review of NEW bugs with four groups of components, draft
>> new process
>>
>> Now: comment period for new process, finalize process
>>
>> Q2: implement new process across all components involved in shipping
>> Firefox
>> Q3: all newly triaged bugs following the new process
>>
>> -- Emma Humphries, Bugmaster
>>
>> _______________________________________________
>> firefox-dev mailing list
>> [hidden email]
>> https://mail.mozilla.org/listinfo/firefox-dev
>>
>>
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Eric Rescorla
Hmmm... Who do you believe is the decider here and what do you believe is
the decision procedure for these components? Generally, the way things work
isn't that Firefox promulgates policy for how other components handle their
bugs.

On a more substantive, less procedural note, this seems to be designed for
a particular workflow in which there is an assumption that bugs are for
immediate processing. However, in may cases we use bugs as placeholders or
assemble big dependency trees of all the bugs that are needed to do a large
complex feature. From this perspective, a three-tier system of "urgent",
"non-urgent", and "wishlist" is either a regression from more fine-grained
systems such as dependency trees/priorities or is redundant with them. This
is especially true for long-running efforts. In other words, this may be a
useful change for some components while not being useful for others. For
the specific case of NSS (where I currently do a lot of my work) this
doesn't seem like it would be a helpful change.

-Ekr



On Tue, Mar 29, 2016 at 4:33 PM, Emma Humphries <[hidden email]> wrote:

> Components that feed into FFx, so that's Core, Toolbox, Firefox, Fennec
> and the pieces of platform we're using to ship.
>
> While there's not a 'Gecko' component in bugzilla, it does cover the
> components there which are Gecko.
>
> -- Emma
>
> On Tue, Mar 29, 2016 at 3:16 PM, Eric Rescorla <[hidden email]> wrote:
>
>> I'm trying to figure out the scope of this proposal. Are you expecting it
>> to apply merely to Firefox or to Gecko as well?
>>
>> -Ekr
>>
>>
>> On Tue, Mar 29, 2016 at 1:07 PM, Emma Humphries <[hidden email]> wrote:
>>
>>> tl;dr
>>>
>>> In Quarter Two I'm implementing the work we’ve been doing to improve
>>> triage, make actionable decisions on new bugs, and prevent us from shipping
>>> regressions in Firefox.
>>>
>>> Today I’m asking for feedback on the plan which is posted at:
>>>
>>>
>>> https://docs.google.com/document/d/1FFrtS0u6gNBE1mxsGJA9JLseJ_U6tW-1NJvHMq551ko
>>>
>>> Allowing bugs to sit around without a decision on what we will do about
>>> them sends the wrong message to Mozillans about how we treat bugs, how we
>>> value their involvement, and reduces quality.
>>>
>>> The Firefox quality team (myself, Mike Hoye, Ryan VanderMeulen, Mark
>>> Cote, and Benjamin Smedberg) want to make better assertions about the
>>> quality of our releases by giving you tools to make clear decisions about
>>> which bugs must be fixed for each release (urgent) and actively tracking
>>> those bugs.
>>> What We Learned From The Pilot Program
>>>
>>> During the past 6 weeks, we have prototyped and tested a triage process
>>> with the DOM, Hello, and Developer Tools teams.
>>>
>>> Andrew Overholt, who participated in the pilot for the DOM team, said,
>>> “A consistent bug triage process can help us spread the load of watching
>>> incoming bugs and help avoid issues falling through the cracks."
>>>
>>> During the pilot, the DOM team uncovered critical bugs quickly so that
>>> people could be assigned to them.
>>>
>>> The pilot groups also found that the triage process needs to be fast and
>>> have tooling to make going through bugs fast. It’s easy to fall behind on
>>> triage for a component, but if you stay up to date it will take no more
>>> than 15 minutes a day.
>>>
>>> You can find the bugs we triaged during the pilot by looking for
>>> whiteboard tags containing ‘btpp-’.
>>>
>>> It is also important to have consistent, shared definitions for
>>> regression across components so triagers do not waste effort on mis-labeled
>>> bugs.
>>> Comments?
>>>
>>> I am posting this plan now for comment over the next week. I intend to
>>> finalize the triage plan for implementation by Tuesday, April 5th. Feedback
>>> and questions are welcome on the document, privately via email or IRC
>>> (where I’m emceeaich) or on the [hidden email] mailing list.
>>> Timeline
>>>
>>> January: finish finding component responsible parties
>>>
>>> February: pilot review of NEW bugs with four groups of components, draft
>>> new process
>>>
>>> Now: comment period for new process, finalize process
>>>
>>> Q2: implement new process across all components involved in shipping
>>> Firefox
>>> Q3: all newly triaged bugs following the new process
>>>
>>> -- Emma Humphries, Bugmaster
>>>
>>> _______________________________________________
>>> firefox-dev mailing list
>>> [hidden email]
>>> https://mail.mozilla.org/listinfo/firefox-dev
>>>
>>>
>>
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

mhoye
On 2016-03-29 8:09 PM, Eric Rescorla wrote:
>
> On a more substantive, less procedural note, this seems to be designed
> for a particular workflow in which there is an assumption that bugs
> are for immediate processing. However, in may cases we use bugs as
> placeholders or assemble big dependency trees of all the bugs that are
> needed to do a large complex feature.
We're definitely talking about untriaged incoming bugs here, not
placeholder or planning bugs, though it's true that the doc doesn't call
that out explicitly. I'll add a comment to that effect.

- mhoye
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Emma Humphries
In reply to this post by Eric Rescorla
On Tue, Mar 29, 2016 at 5:09 PM, Eric Rescorla <[hidden email]> wrote:

> On a more substantive, less procedural note, this seems to be designed for
> a particular workflow in which there is an assumption that bugs are for
> immediate processing. However, in may cases we use bugs as placeholders or
> assemble big dependency trees of all the bugs that are needed to do a large
> complex feature. From this perspective, a three-tier system of "urgent",
> "non-urgent", and "wishlist" is either a regression from more fine-grained
> systems such as dependency trees/priorities or is redundant with them. This
> is especially true for long-running efforts. In other words, this may be a
> useful change for some components while not being useful for others. For
> the specific case of NSS (where I currently do a lot of my work) this
> doesn't seem like it would be a helpful change.


​This is where it would be nice to have a way of saying, feature vs. bug as
part of the triage process, even it that's not a clean separation to make,
because that could get long running feature work out of the triage process,
but I don't have an immediate answer for this. It may be that we change the
scope of components this applies to.

I'll follow up with Doug Turner to see if changing the scope of this for
platform related bugs is warranted.


 --
​ Emma​
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Axel Hecht
In reply to this post by Emma Humphries
Hi Emma,

for those of us that are addicted to data: You have about a 1000 bugs of
data, and I'd love to hear some of the good parts, and maybe also some
of the bad parts.

Also, you tested on three teams, and you report a success story from
one. Could you frame that a bit? Is that within the expectations, or
above or below?

Axel

On 29/03/16 22:07, Emma Humphries wrote:

> tl;dr
>
> In Quarter Two I'm implementing the work we’ve been doing to improve
> triage, make actionable decisions on new bugs, and prevent us from shipping
> regressions in Firefox.
>
> Today I’m asking for feedback on the plan which is posted at:
>
> https://docs.google.com/document/d/1FFrtS0u6gNBE1mxsGJA9JLseJ_U6tW-1NJvHMq551ko
>
> Allowing bugs to sit around without a decision on what we will do about
> them sends the wrong message to Mozillans about how we treat bugs, how we
> value their involvement, and reduces quality.
>
> The Firefox quality team (myself, Mike Hoye, Ryan VanderMeulen, Mark Cote,
> and Benjamin Smedberg) want to make better assertions about the quality of
> our releases by giving you tools to make clear decisions about which bugs
> must be fixed for each release (urgent) and actively tracking those bugs.
> What We Learned From The Pilot Program
>
> During the past 6 weeks, we have prototyped and tested a triage process
> with the DOM, Hello, and Developer Tools teams.
>
> Andrew Overholt, who participated in the pilot for the DOM team, said, “A
> consistent bug triage process can help us spread the load of watching
> incoming bugs and help avoid issues falling through the cracks."
>
> During the pilot, the DOM team uncovered critical bugs quickly so that
> people could be assigned to them.
>
> The pilot groups also found that the triage process needs to be fast and
> have tooling to make going through bugs fast. It’s easy to fall behind on
> triage for a component, but if you stay up to date it will take no more
> than 15 minutes a day.
>
> You can find the bugs we triaged during the pilot by looking for whiteboard
> tags containing ‘btpp-’.
>
> It is also important to have consistent, shared definitions for regression
> across components so triagers do not waste effort on mis-labeled bugs.
> Comments?
>
> I am posting this plan now for comment over the next week. I intend to
> finalize the triage plan for implementation by Tuesday, April 5th. Feedback
> and questions are welcome on the document, privately via email or IRC
> (where I’m emceeaich) or on the [hidden email] mailing list.
> Timeline
>
> January: finish finding component responsible parties
>
> February: pilot review of NEW bugs with four groups of components, draft
> new process
>
> Now: comment period for new process, finalize process
>
> Q2: implement new process across all components involved in shipping Firefox
> Q3: all newly triaged bugs following the new process
>
> -- Emma Humphries, Bugmaster
>

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Milan Sreckovic
In reply to this post by Emma Humphries
We do have a feature keyword today.  While it may be most used for the documentation purposes, the feedback graphics team got when we started using it to tag feature requests was positive.  As in, it’s OK to use for that.

- Milan



> On Mar 29, 2016, at 22:32 , Emma Humphries <[hidden email]> wrote:
>
> On Tue, Mar 29, 2016 at 5:09 PM, Eric Rescorla <[hidden email]> wrote:
>
>> On a more substantive, less procedural note, this seems to be designed for
>> a particular workflow in which there is an assumption that bugs are for
>> immediate processing. However, in may cases we use bugs as placeholders or
>> assemble big dependency trees of all the bugs that are needed to do a large
>> complex feature. From this perspective, a three-tier system of "urgent",
>> "non-urgent", and "wishlist" is either a regression from more fine-grained
>> systems such as dependency trees/priorities or is redundant with them. This
>> is especially true for long-running efforts. In other words, this may be a
>> useful change for some components while not being useful for others. For
>> the specific case of NSS (where I currently do a lot of my work) this
>> doesn't seem like it would be a helpful change.
>
>
> ​This is where it would be nice to have a way of saying, feature vs. bug as
> part of the triage process, even it that's not a clean separation to make,
> because that could get long running feature work out of the triage process,
> but I don't have an immediate answer for this. It may be that we change the
> scope of components this applies to.
>
> I'll follow up with Doug Turner to see if changing the scope of this for
> platform related bugs is warranted.
>
> ​
> --
> ​ Emma​
> _______________________________________________
> dev-platform mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-platform

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Lawrence Mandel-2
Softvision also makes use of the feature keyword as one way to identify new
feature work to test in upcoming releases.

Lawrence

On Thu, Mar 31, 2016 at 10:49 AM, Milan Sreckovic <[hidden email]>
wrote:

> We do have a feature keyword today.  While it may be most used for the
> documentation purposes, the feedback graphics team got when we started
> using it to tag feature requests was positive.  As in, it’s OK to use for
> that.
> —
> - Milan
>
>
>
> > On Mar 29, 2016, at 22:32 , Emma Humphries <[hidden email]> wrote:
> >
> > On Tue, Mar 29, 2016 at 5:09 PM, Eric Rescorla <[hidden email]> wrote:
> >
> >> On a more substantive, less procedural note, this seems to be designed
> for
> >> a particular workflow in which there is an assumption that bugs are for
> >> immediate processing. However, in may cases we use bugs as placeholders
> or
> >> assemble big dependency trees of all the bugs that are needed to do a
> large
> >> complex feature. From this perspective, a three-tier system of "urgent",
> >> "non-urgent", and "wishlist" is either a regression from more
> fine-grained
> >> systems such as dependency trees/priorities or is redundant with them.
> This
> >> is especially true for long-running efforts. In other words, this may
> be a
> >> useful change for some components while not being useful for others. For
> >> the specific case of NSS (where I currently do a lot of my work) this
> >> doesn't seem like it would be a helpful change.
> >
> >
> > ​This is where it would be nice to have a way of saying, feature vs. bug
> as
> > part of the triage process, even it that's not a clean separation to
> make,
> > because that could get long running feature work out of the triage
> process,
> > but I don't have an immediate answer for this. It may be that we change
> the
> > scope of components this applies to.
> >
> > I'll follow up with Doug Turner to see if changing the scope of this for
> > platform related bugs is warranted.
> >
> > ​
> > --
> > ​ Emma​
> > _______________________________________________
> > dev-platform mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-platform
>
> _______________________________________________
> dev-platform mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-platform
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Milan Sreckovic
In reply to this post by Emma Humphries
This may be somewhat long winded.  I will make it in the context of Gecko graphics, because that’s where I have the most data and experience.  It may or may not apply to other components.

Reviewing all the incoming bugs, in a timely matter, is very important to us.  Over the past few years, the graphics team fixed about half the bugs that came in (roughly, we fix a thousand out of the two thousand that come in.)  The main goal of any kind of a bug triage process is thus to choose the right half of the bugs to fix, and spend as little time as possible on the rest.

With that in mind we started a much less documented and much more minimal process in 2015.  I don’t know that we have the data to back the “avoid issues falling between the cracks”, but it certainly seems that way.  One data point we do have - the average amount of time it took to fix the bugs triaged in 2015 was almost half of that for 2014 and the previous four years.

This to illustrate that I believe in the bug triage, looking at bugs as they come in, and making some quick decisions how to proceed.

I’m also a big believer in lightweight processes and minimal documentation, so there are a few comments I have on what the document below describes, and in general.

The more states we have, the more not-so-useful conversation we’ll have about assigning those states.  I’m glad to see that we have a small number, currently named fix-now, non-urgent and wishlist (the naming is in flux and being argued in the document.)  I’m mentally mapping these to “blocking the release”, “can’t ship too many releases with this” and “I hope we eventually fix this”, but perhaps there is a different interpretation.

I would expect to see the majority of the graphics bugs end up in the last group.  Why?  Since you never plan for the full capacity, as that actually reduces your throughput, and since we only fix half the bugs that come in, it stands to reason that we should not have even close to half of the bugs in the first two categories.  In other words, we want to be fixing some of the “wishlist” bugs.  And we need to be very careful about not making the fix-now turn into “we really should fix this”, and only allow the “team works on nothing else while there are fix-now bugs open”.  Otherwise, well, it loses its value.

Step aside - my thoughts on the “How Triage Will Work in Bugzilla” section.  I would stick with the definition of the states and have dashboards that show the bug counts in different categories for different teams.  The current description is a bit too detailed and inflexible.  It suggests that we can figure out the best way to do this before we’ve even started doing it (the pilot program non-withstanding), and it mixes the mechanics with the goals.

I’m going to start and keep arguing that we do not want to have an explicit name for that largest bucket of “wishlist” bugs, and should instead have it marked by the absence of a tag.  This is not about the heavily involved community, those that will stick around regardless of what we do to them, and anybody that reads this e-mail.  This is about people that are reporting their first bugs, that are occasionally involved, who are vital to our success.  If I was one of those, and I started seeing that most, if not all, of my bugs are marked as wishlist or deferred or in-copious-spare-time, I’m going to get discouraged and stop doing it.  After all, I don’t seem to be valuably contributing, because I’m just telling you things you don’t care about.  Or, I could start arguing in the bug, that this should be higher priority, and fill up the comments with non-technical information.

Getting close to a full page, I’ll stop now.  I’m available for live conversations on the topic :)


- Milan



> On Mar 29, 2016, at 16:07 , Emma Humphries <[hidden email]> wrote:
>
> tl;dr
>
> In Quarter Two I'm implementing the work we’ve been doing to improve
> triage, make actionable decisions on new bugs, and prevent us from shipping
> regressions in Firefox.
>
> Today I’m asking for feedback on the plan which is posted at:
>
> https://docs.google.com/document/d/1FFrtS0u6gNBE1mxsGJA9JLseJ_U6tW-1NJvHMq551ko
>
> Allowing bugs to sit around without a decision on what we will do about
> them sends the wrong message to Mozillans about how we treat bugs, how we
> value their involvement, and reduces quality.
>
> The Firefox quality team (myself, Mike Hoye, Ryan VanderMeulen, Mark Cote,
> and Benjamin Smedberg) want to make better assertions about the quality of
> our releases by giving you tools to make clear decisions about which bugs
> must be fixed for each release (urgent) and actively tracking those bugs.
> What We Learned From The Pilot Program
>
> During the past 6 weeks, we have prototyped and tested a triage process
> with the DOM, Hello, and Developer Tools teams.
>
> Andrew Overholt, who participated in the pilot for the DOM team, said, “A
> consistent bug triage process can help us spread the load of watching
> incoming bugs and help avoid issues falling through the cracks."
>
> During the pilot, the DOM team uncovered critical bugs quickly so that
> people could be assigned to them.
>
> The pilot groups also found that the triage process needs to be fast and
> have tooling to make going through bugs fast. It’s easy to fall behind on
> triage for a component, but if you stay up to date it will take no more
> than 15 minutes a day.
>
> You can find the bugs we triaged during the pilot by looking for whiteboard
> tags containing ‘btpp-’.
>
> It is also important to have consistent, shared definitions for regression
> across components so triagers do not waste effort on mis-labeled bugs.
> Comments?
>
> I am posting this plan now for comment over the next week. I intend to
> finalize the triage plan for implementation by Tuesday, April 5th. Feedback
> and questions are welcome on the document, privately via email or IRC
> (where I’m emceeaich) or on the [hidden email] mailing list.
> Timeline
>
> January: finish finding component responsible parties
>
> February: pilot review of NEW bugs with four groups of components, draft
> new process
>
> Now: comment period for new process, finalize process
>
> Q2: implement new process across all components involved in shipping Firefox
> Q3: all newly triaged bugs following the new process
>
> -- Emma Humphries, Bugmaster
> _______________________________________________
> dev-platform mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-platform

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Daniel Veditz-2
On Thu, Mar 31, 2016 at 12:28 PM, Milan Sreckovic <[hidden email]>
wrote:

> I’m going to start and keep arguing that we do not want to have an
> explicit name for that largest bucket of “wishlist” bugs, and should
> instead have it marked by the absence of a tag.


​What distinguishes a bug that has not been triaged from a bug that has
been triaged and (mentally?) put into the third bucket if there's no
explicit marker?​

This is about people that are reporting their first bugs, that are
> occasionally involved, who are vital to our success.  If I was one of
> those, and I started seeing that most, if not all, of my bugs are marked as
> wishlist or deferred or in-copious-spare-time, I’m going to get discouraged
> and stop doing it.  After all, I don’t seem to be valuably contributing,
> because I’m just telling you things you don’t care about.  Or, I could
> start arguing in the bug, that this should be higher priority, and fill up
> the comments with non-technical information.
>

​We get that now, with no marker at all. The only real difference I see
with a marker is that people will catch on sooner whereas now it takes a
while until they realize they are being ignored. They eventually get
discouraged​ or upset either way. Might even be better with a marker (for
some people) because at least they have some acknowledgement that someone
has looked at the bug even if they disagree about the importance. We may
have misunderstood, and thus mis-triaged, the bug and an explicit marker
might trigger the attempt to clarify sooner.

The people who are going to demand attention for their super important
critical blocker bug are likely to do that either way. A triage marker
might help lance the boil sooner rather than let frustration build up to
explosive levels.

-
​Dan Veditz​
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Emma Humphries
In reply to this post by Milan Sreckovic
On Thu, Mar 31, 2016 at 12:28 PM, Milan Sreckovic <[hidden email]>
wrote:

> Or, I could start arguing in the bug, that this should be higher priority,
> and fill up the comments with non-technical information.



​This reminds me, we have filtering tools to mute abusive and unproductive
comments in bugs.

https://wiki.mozilla.org/BMO/comment_tagging

-- Emma


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Emma Humphries
In reply to this post by Milan Sreckovic
What to call these has been a long thread in the document comments, but I'm
starting to lean towards:

FIX-FOR-RELEASE
FIX-SOON
and
BACKLOG

as well as adding (see the proposed edit) a FEATURE resolution for bugs
which are feature tracking and individual features. I'd consider adding a
consistency rule requiring a User Story field for bugs in that status.


On Thu, Mar 31, 2016 at 12:28 PM, Milan Sreckovic <[hidden email]>
wrote:

> This may be somewhat long winded.  I will make it in the context of Gecko
> graphics, because that’s where I have the most data and experience.  It may
> or may not apply to other components.
>
> Reviewing all the incoming bugs, in a timely matter, is very important to
> us.  Over the past few years, the graphics team fixed about half the bugs
> that came in (roughly, we fix a thousand out of the two thousand that come
> in.)  The main goal of any kind of a bug triage process is thus to choose
> the right half of the bugs to fix, and spend as little time as possible on
> the rest.
>
> With that in mind we started a much less documented and much more minimal
> process in 2015.  I don’t know that we have the data to back the “avoid
> issues falling between the cracks”, but it certainly seems that way.  One
> data point we do have - the average amount of time it took to fix the bugs
> triaged in 2015 was almost half of that for 2014 and the previous four
> years.
>
> This to illustrate that I believe in the bug triage, looking at bugs as
> they come in, and making some quick decisions how to proceed.
>
> I’m also a big believer in lightweight processes and minimal
> documentation, so there are a few comments I have on what the document
> below describes, and in general.
>
> The more states we have, the more not-so-useful conversation we’ll have
> about assigning those states.  I’m glad to see that we have a small number,
> currently named fix-now, non-urgent and wishlist (the naming is in flux and
> being argued in the document.)  I’m mentally mapping these to “blocking the
> release”, “can’t ship too many releases with this” and “I hope we
> eventually fix this”, but perhaps there is a different interpretation.
>
> I would expect to see the majority of the graphics bugs end up in the last
> group.  Why?  Since you never plan for the full capacity, as that actually
> reduces your throughput, and since we only fix half the bugs that come in,
> it stands to reason that we should not have even close to half of the bugs
> in the first two categories.  In other words, we want to be fixing some of
> the “wishlist” bugs.  And we need to be very careful about not making the
> fix-now turn into “we really should fix this”, and only allow the “team
> works on nothing else while there are fix-now bugs open”.  Otherwise, well,
> it loses its value.
>
> Step aside - my thoughts on the “How Triage Will Work in Bugzilla”
> section.  I would stick with the definition of the states and have
> dashboards that show the bug counts in different categories for different
> teams.  The current description is a bit too detailed and inflexible.  It
> suggests that we can figure out the best way to do this before we’ve even
> started doing it (the pilot program non-withstanding), and it mixes the
> mechanics with the goals.
>
> I’m going to start and keep arguing that we do not want to have an
> explicit name for that largest bucket of “wishlist” bugs, and should
> instead have it marked by the absence of a tag.  This is not about the
> heavily involved community, those that will stick around regardless of what
> we do to them, and anybody that reads this e-mail.  This is about people
> that are reporting their first bugs, that are occasionally involved, who
> are vital to our success.  If I was one of those, and I started seeing that
> most, if not all, of my bugs are marked as wishlist or deferred or
> in-copious-spare-time, I’m going to get discouraged and stop doing it.
> After all, I don’t seem to be valuably contributing, because I’m just
> telling you things you don’t care about.  Or, I could start arguing in the
> bug, that this should be higher priority, and fill up the comments with
> non-technical information.
>
> Getting close to a full page, I’ll stop now.  I’m available for live
> conversations on the topic :)
>
> —
> - Milan
>
>
>
> > On Mar 29, 2016, at 16:07 , Emma Humphries <[hidden email]> wrote:
> >
> > tl;dr
> >
> > In Quarter Two I'm implementing the work we’ve been doing to improve
> > triage, make actionable decisions on new bugs, and prevent us from
> shipping
> > regressions in Firefox.
> >
> > Today I’m asking for feedback on the plan which is posted at:
> >
> >
> https://docs.google.com/document/d/1FFrtS0u6gNBE1mxsGJA9JLseJ_U6tW-1NJvHMq551ko
> >
> > Allowing bugs to sit around without a decision on what we will do about
> > them sends the wrong message to Mozillans about how we treat bugs, how we
> > value their involvement, and reduces quality.
> >
> > The Firefox quality team (myself, Mike Hoye, Ryan VanderMeulen, Mark
> Cote,
> > and Benjamin Smedberg) want to make better assertions about the quality
> of
> > our releases by giving you tools to make clear decisions about which bugs
> > must be fixed for each release (urgent) and actively tracking those bugs.
> > What We Learned From The Pilot Program
> >
> > During the past 6 weeks, we have prototyped and tested a triage process
> > with the DOM, Hello, and Developer Tools teams.
> >
> > Andrew Overholt, who participated in the pilot for the DOM team, said, “A
> > consistent bug triage process can help us spread the load of watching
> > incoming bugs and help avoid issues falling through the cracks."
> >
> > During the pilot, the DOM team uncovered critical bugs quickly so that
> > people could be assigned to them.
> >
> > The pilot groups also found that the triage process needs to be fast and
> > have tooling to make going through bugs fast. It’s easy to fall behind on
> > triage for a component, but if you stay up to date it will take no more
> > than 15 minutes a day.
> >
> > You can find the bugs we triaged during the pilot by looking for
> whiteboard
> > tags containing ‘btpp-’.
> >
> > It is also important to have consistent, shared definitions for
> regression
> > across components so triagers do not waste effort on mis-labeled bugs.
> > Comments?
> >
> > I am posting this plan now for comment over the next week. I intend to
> > finalize the triage plan for implementation by Tuesday, April 5th.
> Feedback
> > and questions are welcome on the document, privately via email or IRC
> > (where I’m emceeaich) or on the [hidden email] mailing list.
> > Timeline
> >
> > January: finish finding component responsible parties
> >
> > February: pilot review of NEW bugs with four groups of components, draft
> > new process
> >
> > Now: comment period for new process, finalize process
> >
> > Q2: implement new process across all components involved in shipping
> Firefox
> > Q3: all newly triaged bugs following the new process
> >
> > -- Emma Humphries, Bugmaster
> > _______________________________________________
> > dev-platform mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-platform
>
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Milan Sreckovic
In reply to this post by Daniel Veditz-2

> On Mar 31, 2016, at 18:22 , Daniel Veditz <[hidden email]> wrote:
>
> On Thu, Mar 31, 2016 at 12:28 PM, Milan Sreckovic <[hidden email] <mailto:[hidden email]>> wrote:
> I’m going to start and keep arguing that we do not want to have an explicit name for that largest bucket of “wishlist” bugs, and should instead have it marked by the absence of a tag.
>
> ​What distinguishes a bug that has not been triaged from a bug that has been triaged and (mentally?) put into the third bucket if there's no explicit marker?​

I understood there was an “untriaged” status set by default.

On the other hand, the latest choice of “backlog” is non-confrontational enough that, for the purposes of the naming discussion, is close enough to “blank" for me not to argue.


- Milan

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Mike Hommey
In reply to this post by Emma Humphries
On Thu, Mar 31, 2016 at 06:47:32PM -0700, Emma Humphries wrote:

> What to call these has been a long thread in the document comments, but I'm
> starting to lean towards:
>
> FIX-FOR-RELEASE
> FIX-SOON
> and
> BACKLOG
>
> as well as adding (see the proposed edit) a FEATURE resolution for bugs
> which are feature tracking and individual features. I'd consider adding a
> consistency rule requiring a User Story field for bugs in that status.

If we're talking about a bmo-wide change, then a tri-state where one of
the state is bound to something being released, then that only leaves
two states for components that are not really attached to things making
it for a particular release, or are not released at all.

If we still get to use Pn and other related fields on top of that, why
are we adding another field? Not that I'm against adding fields, but as
someone else mentioned in the thread, we've done plenty of adding fields
in the past 18 years, and that's what kind of leads to the mess we are
in today. Adding a new field should be accompanied with removing
other(s).

On the other hand, I find it interesting that in some way, it overlaps
with the existing bug status (UNCONFIRMED, NEW, ASSIGNED), and I think
the document is dismissing it too quickly:

"because a bug can go from UNCONFIRMED to NEW, without us knowing how
important it is."

to which I want to answer with another question:

"then why not simply add status(es)?"

Bug status is currently, IMHO, completely misused and thus useless:
- people with editbug capability file as NEW by default. Why should a bug
  I file in a component I'm not working on (because I noticed a bug
  in Firefox) be NEW?
- there is a long tail of bugs assigned to people that are not being worked
  on (I should know, I have a lot of those, shame on me)

So it feels to me triage should replace/subsume it in some way.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Gervase Markham
On 01/04/16 15:51, Mike Hommey wrote:
> Bug status is currently, IMHO, completely misused and thus useless:
> - people with editbug capability file as NEW by default. Why should a bug
>   I file in a component I'm not working on (because I noticed a bug
>   in Firefox) be NEW?
> - there is a long tail of bugs assigned to people that are not being worked
>   on (I should know, I have a lot of those, shame on me)
>
> So it feels to me triage should replace/subsume it in some way.

I suspect they want to add a new field because changing bug statuses
seems like a massive change. Which it would be. However, not doing it
will leave us with two workflow widgets in Bugzilla instead of one, with
all the ambiguity that brings. In the long term, I see pain here.

If Bugzilla supported per-product workflow that might help. Is it worth
investing the BMO-hacking resources for this plan into that?

Gerv

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

glob
Gervase Markham wrote:

> On 01/04/16 15:51, Mike Hommey wrote:
>> Bug status is currently, IMHO, completely misused and thus useless:
>> - people with editbug capability file as NEW by default. Why should a bug
>>    I file in a component I'm not working on (because I noticed a bug
>>    in Firefox) be NEW?
>> - there is a long tail of bugs assigned to people that are not being worked
>>    on (I should know, I have a lot of those, shame on me)
>>
>> So it feels to me triage should replace/subsume it in some way.
> I suspect they want to add a new field because changing bug statuses
> seems like a massive change. Which it would be. However, not doing it
> will leave us with two workflow widgets in Bugzilla instead of one, with
> all the ambiguity that brings. In the long term, I see pain here.
right - adjusting the workflow is a silently breaking one so we're not
taking that step just (yet!).
eg. dashboards that use a hardcoded list of "open" states will silently
stop reporting on all bugs if we add a new state.
> If Bugzilla supported per-product workflow that might help. Is it worth
> investing the BMO-hacking resources for this plan into that?
the plan is to have per-product customisations in the UI with
per-product workflow enforcement via extensions.
"per-product workflows" isn't an option as we're not altering the BMO
workflow fields (status/resolution) in a way that would make this viable.

imho long term the list of status/resolutions needs to be updated; but
first we need to determine, document, and mandate the desired workflow.  
emma's work is the first step of this process, and i expect somewhere
near the end lives a "break all the things" change for the greater good.


-glob

--
byron jones - :glob - bugzilla.mozilla.org team lead -

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Onno Ekker-2
In reply to this post by Emma Humphries
Op 2-4-2016 om 0:51 schreef Mike Hommey:
> Bug status is currently, IMHO, completely misused and thus useless:
> - people with editbug capability file as NEW by default. Why should a bug
>   I file in a component I'm not working on (because I noticed a bug
>   in Firefox) be NEW?

I try to change it back to UNCONFIRMED whenever I file a "new" bug, but
there are times I forget. I think UNCONFIRMED would be a better default…

Onno

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Justin Dolske-2
In reply to this post by Emma Humphries
A few comments, although I think others have touched on some of this
already.

I think my biggest concern is that this is conflating triage and
prioritization, which complicates things and introduces ambiguity. For
example, what would it even mean to have a "triage: fix-now" bug that's
also a P5? I'd much rather leave prioritization to the priority field. I'm
similarly dubious about the value of "not-urgent" and "wishlist". In part
because those also imply priorities, but also because this phrasing seems
like just kicking the can down the road on "making a decision".

It would seem simpler to just have a triage +/-/? flag to indicate if a
decision on triage has been made. Although we're still kind of tap-dancing
around what to do with the inevitable pile of "we just won't be able to fix
this soon bugs" -- I'm not sure a subtle triage flag effectively
communicates intent to people that don't live inside Bugzilla like we do.
It wouldn't be any worse than the status quo (and so is fine to trial), but
it should really just be a new status/resolution... The existing "WONTFIX"
and "INVALID" often convey the wrong meaning, a more accurate (and
gentler!) "DEFERRED" or "INACTIVE" or $BIKESHED_HERE is something I'd like
to see for this common condition.

I don't really understand the 3-month auto-timeout of bugs back to an
untriaged state. This seems like it's just creating work and makes Bugzilla
annoying. Nor is a one-size-fits all solution right here -- large /
long-term and small / short-term project areas will have different needs.
Grooming the list of triaged, prioritized bugs is important for teams to
do, and we should encourage them to do so, but starting off with this being
automatic seems premature.

I'm also concerned about bolting on yet more UI, which makes Bugzilla even
_more_ obtuse and complicated. But given all the feedback on the current
proposal, I suppose it's not worth commenting on here because I presume
this would all change anyway. This also seems like it should just be a
separate project.

Justin


On Tue, Mar 29, 2016 at 1:07 PM, Emma Humphries <[hidden email]> wrote:

> tl;dr
>
> In Quarter Two I'm implementing the work we’ve been doing to improve
> triage, make actionable decisions on new bugs, and prevent us from shipping
> regressions in Firefox.
>
> Today I’m asking for feedback on the plan which is posted at:
>
>
> https://docs.google.com/document/d/1FFrtS0u6gNBE1mxsGJA9JLseJ_U6tW-1NJvHMq551ko
>
> Allowing bugs to sit around without a decision on what we will do about
> them sends the wrong message to Mozillans about how we treat bugs, how we
> value their involvement, and reduces quality.
>
> The Firefox quality team (myself, Mike Hoye, Ryan VanderMeulen, Mark Cote,
> and Benjamin Smedberg) want to make better assertions about the quality of
> our releases by giving you tools to make clear decisions about which bugs
> must be fixed for each release (urgent) and actively tracking those bugs.
> What We Learned From The Pilot Program
>
> During the past 6 weeks, we have prototyped and tested a triage process
> with the DOM, Hello, and Developer Tools teams.
>
> Andrew Overholt, who participated in the pilot for the DOM team, said, “A
> consistent bug triage process can help us spread the load of watching
> incoming bugs and help avoid issues falling through the cracks."
>
> During the pilot, the DOM team uncovered critical bugs quickly so that
> people could be assigned to them.
>
> The pilot groups also found that the triage process needs to be fast and
> have tooling to make going through bugs fast. It’s easy to fall behind on
> triage for a component, but if you stay up to date it will take no more
> than 15 minutes a day.
>
> You can find the bugs we triaged during the pilot by looking for
> whiteboard tags containing ‘btpp-’.
>
> It is also important to have consistent, shared definitions for regression
> across components so triagers do not waste effort on mis-labeled bugs.
> Comments?
>
> I am posting this plan now for comment over the next week. I intend to
> finalize the triage plan for implementation by Tuesday, April 5th. Feedback
> and questions are welcome on the document, privately via email or IRC
> (where I’m emceeaich) or on the [hidden email] mailing list.
> Timeline
>
> January: finish finding component responsible parties
>
> February: pilot review of NEW bugs with four groups of components, draft
> new process
>
> Now: comment period for new process, finalize process
>
> Q2: implement new process across all components involved in shipping
> Firefox
> Q3: all newly triaged bugs following the new process
>
> -- Emma Humphries, Bugmaster
>
> _______________________________________________
> firefox-dev mailing list
> [hidden email]
> https://mail.mozilla.org/listinfo/firefox-dev
>
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Triage Plan for Firefox Components

Boris Zbarsky
In reply to this post by Emma Humphries
On 4/1/16 6:51 PM, Mike Hommey wrote:
> - people with editbug capability file as NEW by default. Why should a bug
>    I file in a component I'm not working on (because I noticed a bug
>    in Firefox) be NEW?

Because the assumption is that you, as someone with editbugs, know
enough to write a decent bug report.

This may or may not be a good assumption; I've certainly seen lots of
crappy bug reports filed by people with editbugs, including in
components they _are_ working on...

-Boris

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
12