Anatomy of a regression

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Anatomy of a regression

Robert O'Callahan-3
Bug 723484 is a regression we shipped in Firefox 10 and 11. Not the worst
regression ever, but quite bad.

I think it's worth making an example of to determine what, if anything, we
could have done differently to avoid shipping the regression, or fix it
earlier. Unfortunately I don't have any good ideas myself, perhaps because
it's before 8am here. I do have a vague feeling that we should have
prioritized and started nagging over this bug earlier.

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Sheila Mooney
I suspect this is the kind of bug that should have been nominated for
tracking, correct? It would have at least ended up on a list that would
have been looked at more closely and likely it would have been addressed
during the release cycle it was found in. Something to help in the
short-term might be some repeat communication around the tracking flags,
how they work and the process. This is the second instance of this I have
heard of in recent weeks, and I am sure there are more issues we just
haven't discovered yet.

In the longer term, improved triage processes and such should really help
reduce the number of problems that fall through the cracks and flag big
regressions earlier on in the cycle.

On Mon, Mar 19, 2012 at 11:57 AM, Robert O'Callahan <[hidden email]>wrote:

> Bug 723484 is a regression we shipped in Firefox 10 and 11. Not the worst
> regression ever, but quite bad.
>
> I think it's worth making an example of to determine what, if anything, we
> could have done differently to avoid shipping the regression, or fix it
> earlier. Unfortunately I don't have any good ideas myself, perhaps because
> it's before 8am here. I do have a vague feeling that we should have
> prioritized and started nagging over this bug earlier.
>
> Rob
> --
> “You have heard that it was said, ‘Love your neighbor and hate your enemy.’
> But I tell you, love your enemies and pray for those who persecute you,
> that you may be children of your Father in heaven. ... If you love those
> who love you, what reward will you get? Are not even the tax collectors
> doing that? And if you greet only your own people, what are you doing more
> than others?" [Matthew 5:43-47]
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Alexander Keybl
In reply to this post by Robert O'Callahan-3
The normal escalation path for critical bugs is to nominate the bug for tracking in affected releases. We rely upon engineers and module owners to make that evaluation, preferably with a low bar for nomination so that bugs we're unsure about still get on the radar of release drivers. I think it's important to determine if there was a delay in determining the criticality of 723484, or whether the tracking process needs to be better communicated.

-Alex

On Mar 19, 2012, at 11:57 AM, Robert O'Callahan wrote:

> Bug 723484 is a regression we shipped in Firefox 10 and 11. Not the worst
> regression ever, but quite bad.
>
> I think it's worth making an example of to determine what, if anything, we
> could have done differently to avoid shipping the regression, or fix it
> earlier. Unfortunately I don't have any good ideas myself, perhaps because
> it's before 8am here. I do have a vague feeling that we should have
> prioritized and started nagging over this bug earlier.
>
> Rob
> --
> “You have heard that it was said, ‘Love your neighbor and hate your enemy.’
> But I tell you, love your enemies and pray for those who persecute you,
> that you may be children of your Father in heaven. ... If you love those
> who love you, what reward will you get? Are not even the tax collectors
> doing that? And if you greet only your own people, what are you doing more
> than others?" [Matthew 5:43-47]
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Dave Mandelin-2
In reply to this post by Robert O'Callahan-3
On Monday, March 19, 2012 11:57:04 AM UTC-7, Robert O&#39;Callahan wrote:
> Bug 723484 is a regression we shipped in Firefox 10 and 11. Not the worst
> regression ever, but quite bad.
>
> I think it's worth making an example of to determine what, if anything, we
> could have done differently to avoid shipping the regression, or fix it
> earlier. Unfortunately I don't have any good ideas myself, perhaps because
> it's before 8am here. I do have a vague feeling that we should have
> prioritized and started nagging over this bug earlier.

Did we know that bug was a serious problem? If not, that's the first step.

If we knew it was a serious problem, making sure it gets fixed is relatively easy. As in, easier than the identifying bad bugs ahead of time, but still a lot of work.

My practice for JS: I watch all new bugs filed against the JS component daily (unless out, traveling, or working on a special project). I especially look for user-facing regressions.

I assign user-facing regressions a priority (in my own records) based on how bad they are. That step is a judgment call, so I can't write down a decision tree or anything like that. Some of the things I look at are: platforms affected, number of users reporting the problem, whether it's reported for (popular sites, niche sites, obscure sites, sites under development), JS feature affected (if known).

If a bug is in the "most serious" category, I mark it tracking for the affected versions, which automatically gets extra attention from release drivers and others. I also set a target fix date (in my own to-do list), which defaults to 28 days after the report date (or 28 days after today's date if the bug arrived in my triage list late), but the important thing is that the target date is well before the release date. I very often miss target dates, but that's OK if the date is early enough. I check the to-do list frequently and fix or assign and follow through the bugs.

I don't really have a handle on the problem of actually identifying the worst bugs ahead of time. I think I can do an OK job of it based on intuition learned through experience, but I don't consider myself actually able to predict bug severity. One option would be to conservative: treat a bug as serious unless proven otherwise. I don't do that with user-facing regressions--there are just too many bugs and too many other things to do (but note that we do fix a lot of fuzz bugs of unknown significance, which I suspect is really valuable overall). I suspect doing well on prediction would take a concerted effort, with stats tracked over time. I haven't attempted it.

Dave
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Robert O'Callahan-3
In reply to this post by Alexander Keybl
On Tue, Mar 20, 2012 at 8:49 AM, Alex Keybl <[hidden email]> wrote:

> The normal escalation path for critical bugs is to nominate the bug for
> tracking in affected releases. We rely upon engineers and module owners to
> make that evaluation, preferably with a low bar for nomination so that bugs
> we're unsure about still get on the radar of release drivers. I think it's
> important to determine if there was a delay in determining the criticality
> of 723484, or whether the tracking process needs to be better communicated.
>

I'm not sure what we mean by "critical" here. The bug doesn't make Web
sites unsuable, but it affects enough users seriously enough that some Web
developers are trying to work around it. I think of that as "critical", but
does everyone else?

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Robert O'Callahan-3
On Tue, Mar 20, 2012 at 11:33 AM, Robert O'Callahan <[hidden email]>wrote:

> The bug doesn't make Web sites unsuable, but it affects enough users
> seriously enough that some Web developers are trying to work around it. I
> think of that as "critical"
>

To be clear, I think of *regressions* serious enough for Web developers to
work around as "critical".

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Scott Johnson-22
In reply to this post by Dave Mandelin-2
Dave:

Just a question from an inexperienced team member - why 28 days? What's
significant about that number that you choose to make it your to-do
list "achieve by" date? Is it just an arbitrary number, or have you
found that there's some reason to choose 28?

~Scott

On Mon 19 Mar 2012 02:50:32 PM CDT, Dave Mandelin wrote:

> On Monday, March 19, 2012 11:57:04 AM UTC-7, Robert O&#39;Callahan wrote:
>> Bug 723484 is a regression we shipped in Firefox 10 and 11. Not the worst
>> regression ever, but quite bad.
>>
>> I think it's worth making an example of to determine what, if anything, we
>> could have done differently to avoid shipping the regression, or fix it
>> earlier. Unfortunately I don't have any good ideas myself, perhaps because
>> it's before 8am here. I do have a vague feeling that we should have
>> prioritized and started nagging over this bug earlier.
>
> Did we know that bug was a serious problem? If not, that's the first step.
>
> If we knew it was a serious problem, making sure it gets fixed is relatively easy. As in, easier than the identifying bad bugs ahead of time, but still a lot of work.
>
> My practice for JS: I watch all new bugs filed against the JS component daily (unless out, traveling, or working on a special project). I especially look for user-facing regressions.
>
> I assign user-facing regressions a priority (in my own records) based on how bad they are. That step is a judgment call, so I can't write down a decision tree or anything like that. Some of the things I look at are: platforms affected, number of users reporting the problem, whether it's reported for (popular sites, niche sites, obscure sites, sites under development), JS feature affected (if known).
>
> If a bug is in the "most serious" category, I mark it tracking for the affected versions, which automatically gets extra attention from release drivers and others. I also set a target fix date (in my own to-do list), which defaults to 28 days after the report date (or 28 days after today's date if the bug arrived in my triage list late), but the important thing is that the target date is well before the release date. I very often miss target dates, but that's OK if the date is early enough. I check the to-do list frequently and fix or assign and follow through the bugs.
>
> I don't really have a handle on the problem of actually identifying the worst bugs ahead of time. I think I can do an OK job of it based on intuition learned through experience, but I don't consider myself actually able to predict bug severity. One option would be to conservative: treat a bug as serious unless proven otherwise. I don't do that with user-facing regressions--there are just too many bugs and too many other things to do (but note that we do fix a lot of fuzz bugs of unknown significance, which I suspect is really valuable overall). I suspect doing well on prediction would take a concerted effort, with stats tracked over time. I haven't attempted it.
>
> Dave
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Alexander Keybl
In reply to this post by Robert O'Callahan-3
> I'm not sure what we mean by "critical" here. The bug doesn't make Web sites unsuable, but it affects enough users seriously enough that some Web developers are trying to work around it. I think of that as "critical", but does everyone else?

A safe bar for tracking nominations should be whatever you as a subject-matter expert feel is critical to fix in a specific release - we're OK with greater bug volume in release management triage as long as the reasoning for nomination is explicit. We typically track recent regressions, bugs affecting a large population, bugs greatly affecting a smaller population, sg:high/sg:crit security bugs, top crashers, startup crashers, etc.

If tracked, we continually weigh risk vs reward and decide whether or not it makes sense to still uplift a fix depending on where we are in the schedule.

-Alex

On Mar 19, 2012, at 3:33 PM, Robert O'Callahan wrote:

> On Tue, Mar 20, 2012 at 8:49 AM, Alex Keybl <[hidden email]> wrote:
> The normal escalation path for critical bugs is to nominate the bug for tracking in affected releases. We rely upon engineers and module owners to make that evaluation, preferably with a low bar for nomination so that bugs we're unsure about still get on the radar of release drivers. I think it's important to determine if there was a delay in determining the criticality of 723484, or whether the tracking process needs to be better communicated.
>
> I'm not sure what we mean by "critical" here. The bug doesn't make Web sites unsuable, but it affects enough users seriously enough that some Web developers are trying to work around it. I think of that as "critical", but does everyone else?
>
> Rob
> --
> “You have heard that it was said, ‘Love your neighbor and hate your enemy.’ But I tell you, love your enemies and pray for those who persecute you, that you may be children of your Father in heaven. ... If you love those who love you, what reward will you get? Are not even the tax collectors doing that? And if you greet only your own people, what are you doing more than others?" [Matthew 5:43-47]
>

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Dave Mandelin-2
In reply to this post by Dave Mandelin-2
Scott,

It's semi-arbitrary.

On the demand side, I think ideally everything's supposed to be fixed before the Aurora->Beta merge, which means 6 weeks from report date. 4 weeks gives everything at least 2 weeks to bake on Aurora (or 2 weeks slack to get it done before the merge).

On the supply side, it takes a week or two min to fix a hard bug, so it doesn't make sense to go lower than that. And historically (over the last 6-12 months or so), 90% of JS engine bugs that get fixed, get fixed within a month, so it seemed like a good and achievable target for critical bugs.

Dave

On Monday, March 19, 2012 3:53:02 PM UTC-7, Scott Johnson wrote:

> Dave:
>
> Just a question from an inexperienced team member - why 28 days? What's
> significant about that number that you choose to make it your to-do
> list "achieve by" date? Is it just an arbitrary number, or have you
> found that there's some reason to choose 28?
>
> ~Scott
>
> On Mon 19 Mar 2012 02:50:32 PM CDT, Dave Mandelin wrote:
> > On Monday, March 19, 2012 11:57:04 AM UTC-7, Robert O&#39;Callahan wrote:
> >> Bug 723484 is a regression we shipped in Firefox 10 and 11. Not the worst
> >> regression ever, but quite bad.
> >>
> >> I think it's worth making an example of to determine what, if anything, we
> >> could have done differently to avoid shipping the regression, or fix it
> >> earlier. Unfortunately I don't have any good ideas myself, perhaps because
> >> it's before 8am here. I do have a vague feeling that we should have
> >> prioritized and started nagging over this bug earlier.
> >
> > Did we know that bug was a serious problem? If not, that's the first step.
> >
> > If we knew it was a serious problem, making sure it gets fixed is relatively easy. As in, easier than the identifying bad bugs ahead of time, but still a lot of work.
> >
> > My practice for JS: I watch all new bugs filed against the JS component daily (unless out, traveling, or working on a special project). I especially look for user-facing regressions.
> >
> > I assign user-facing regressions a priority (in my own records) based on how bad they are. That step is a judgment call, so I can't write down a decision tree or anything like that. Some of the things I look at are: platforms affected, number of users reporting the problem, whether it's reported for (popular sites, niche sites, obscure sites, sites under development), JS feature affected (if known).
> >
> > If a bug is in the "most serious" category, I mark it tracking for the affected versions, which automatically gets extra attention from release drivers and others. I also set a target fix date (in my own to-do list), which defaults to 28 days after the report date (or 28 days after today's date if the bug arrived in my triage list late), but the important thing is that the target date is well before the release date. I very often miss target dates, but that's OK if the date is early enough. I check the to-do list frequently and fix or assign and follow through the bugs.
> >
> > I don't really have a handle on the problem of actually identifying the worst bugs ahead of time. I think I can do an OK job of it based on intuition learned through experience, but I don't consider myself actually able to predict bug severity. One option would be to conservative: treat a bug as serious unless proven otherwise. I don't do that with user-facing regressions--there are just too many bugs and too many other things to do (but note that we do fix a lot of fuzz bugs of unknown significance, which I suspect is really valuable overall). I suspect doing well on prediction would take a concerted effort, with stats tracked over time. I haven't attempted it.
> >
> > Dave
> > _______________________________________________
> > dev-planning mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Matthias Versen
In reply to this post by Robert O'Callahan-3
Robert O'Callahan wrote:
> Bug 723484 is a regression we shipped in Firefox 10 and 11. Not the worst
> regression ever, but quite bad.
>
> I think it's worth making an example of to determine what, if anything, we
> could have done differently to avoid shipping the regression, or fix it
> earlier. Unfortunately I don't have any good ideas myself, perhaps because
> it's before 8am here. I do have a vague feeling that we should have
> prioritized and started nagging over this bug earlier.

I triaged that bug and I should requested tracking for this.
I don't want to spam the tracking guys with nominations and that could
be the reason why I didn't requested it.
Should I be a little bit more aggressive with requesting tracking ?

Based on the number of dupes i knew that this bug is a serious regression.
Many regressions like in this case are only reported for beta/release
builds and that makes it difficult to fix this regressions in time for
the next release.

Matthias
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Robert O'Callahan-3
On Wed, Mar 21, 2012 at 1:39 PM, Matthias Versen <[hidden email]> wrote:

> I triaged that bug and I should requested tracking for this.
> I don't want to spam the tracking guys with nominations and that could be
> the reason why I didn't requested it.
> Should I be a little bit more aggressive with requesting tracking ?
>

Yes please! :-)

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Boris Zbarsky
In reply to this post by Matthias Versen
On 3/20/12 8:39 PM, Matthias Versen wrote:
> I don't want to spam the tracking guys with nominations and that could
> be the reason why I didn't requested it.
> Should I be a little bit more aggressive with requesting tracking ?

Yes.  A good rule of thumb is that all regressions should request
tracking, imo.

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Robert O'Callahan-3
On Wed, Mar 21, 2012 at 2:00 PM, Boris Zbarsky <[hidden email]> wrote:

> On 3/20/12 8:39 PM, Matthias Versen wrote:
>
>> I don't want to spam the tracking guys with nominations and that could
>> be the reason why I didn't requested it.
>> Should I be a little bit more aggressive with requesting tracking ?
>>
>
> Yes.  A good rule of thumb is that all regressions should request
> tracking, imo.
>

Yes, I agree.

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Sheila Mooney
In reply to this post by Boris Zbarsky
+1

For any regression, even if you aren't quite sure how serious it is, I
would nominate the bug anyhow - adding as much detail as you can provide.
In many cases we set the tracking flag simply until some further
investigation is done and we can make a call on the impact. I nominate
almost all crash regressions although they don't all get tracked or get
fixed. At least they get in front of a set of stakeholders and important
items are less likely to fall through the cracks.

On Tue, Mar 20, 2012 at 6:00 PM, Boris Zbarsky <[hidden email]> wrote:

> On 3/20/12 8:39 PM, Matthias Versen wrote:
>
>> I don't want to spam the tracking guys with nominations and that could
>> be the reason why I didn't requested it.
>> Should I be a little bit more aggressive with requesting tracking ?
>>
>
> Yes.  A good rule of thumb is that all regressions should request
> tracking, imo.
>
> -Boris
>
> ______________________________**_________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/**listinfo/dev-planning<https://lists.mozilla.org/listinfo/dev-planning>
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Bugzilla from jruderman@gmail.com
Could release drivers keep an eye on a regression query instead of
hoping that "someone" nominates each regression? There are many
reasons a bug reporter might not think to nominate a specific bug for
a specific release.

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=kw%3Aregression+prod%3Acore%2Ctoolkit%2Cfirefox
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Alexander Keybl
We're not asking bug reporters to nominate for tracking, but rather for module triage and developers to do so.

I don't think it makes sense for release drivers to spend any large amount of time evaluating bugs that subject matter experts could quickly evaluate for criticality. We'd end up just pinging developers for their opinions regardless - why not front load that action?

If we had the resources, we would of course dive deeper into Bugzilla than we already do. But I'd rather devote any extra resources to untriaged bugs as opposed to bugs that have already had Mozilla eyes on them. We should instead make sure that all developers know about the tracking flags.

-Alex

Jesse Ruderman <[hidden email]> wrote:

Could release drivers keep an eye on a regression query instead of
hoping that "someone" nominates each regression? There are many
reasons a bug reporter might not think to nominate a specific bug for
a specific release.

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=kw%3Aregression+prod%3Acore%2Ctoolkit%2Cfirefox
_____________________________________________

dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Johnathan Nightingale
In reply to this post by Robert O'Callahan-3
On 2012-03-21, at 1:10 AM, Robert O'Callahan wrote:

> On Wed, Mar 21, 2012 at 2:00 PM, Boris Zbarsky <[hidden email]> wrote:
>
>> On 3/20/12 8:39 PM, Matthias Versen wrote:
>>
>>> I don't want to spam the tracking guys with nominations and that could
>>> be the reason why I didn't requested it.
>>> Should I be a little bit more aggressive with requesting tracking ?
>>>
>>
>> Yes.  A good rule of thumb is that all regressions should request
>> tracking, imo.
>>
>
> Yes, I agree.


Yep. And to Sheila's original call for a re-statement of tracking principles (which this thread is, de facto, already doing):

tracking-firefoxN flags are how we put things on the release team's radar for a given release. They don't block the release, but they ARE things we want to track, and some of them might be severe enough that we won't ship without resolution. In a train-based model, it's rarely the case that something blocks the release because either a) it was a problem in prior releases and didn't block them, or b) it's newly introduced and we can identify and backout the offender. That is not universally true, but describes the vast majority of tracked bugs.

Because tracking is a "put on radar for a specific release" flag, not a "stop shipment of the product" flag, I think a couple of pieces of general guidance fall out:

1) As an expert in your domain, if it concerns you for a particular release, nominate it with a clear comment describing why. Better to over-nom than to under-nom, within reason.

2) Bias heavily towards release-specific bugs. Nominating long-standing bugs as a way to "get more attention on them" is rarely a good idea.

Blocking flags used to double as "important bug" flags and were used that way, but in a model where trains leave regularly, we only want to tracking+ stuff that is specific to this train. We might, as a project, need an "important bug" flag with its own module-owner driven triage process, but that is decidedly another thread.

Alex runs channel meetings Tuesdays and Thursdays at 2pm PT where the channels are checked, nominations processed, and tracked bugs prodded. Please feel invited to attend if you want to develop your own intuition for how we manage those lists.

J

---
Johnathan Nightingale
Sr. Director of Firefox Engineering
[hidden email]




_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

beltzner
On Wed, Mar 21, 2012 at 6:51 AM, Johnathan Nightingale
<[hidden email]> wrote:
> 1) As an expert in your domain, if it concerns you for a particular release, nominate it with a clear comment describing why. Better to over-nom than to under-nom, within reason.

As a former triager (oh, the triage!) I also wish to emphasize: "with
a clear comment describing why." Things that are obvious to you may be
non-obvious to others, so as a guideline, include information about:

 - how many users does it affect? (all OSes? specific hardware? common
patterns in use on the web or a really hot demo site?)
 - what's the complexity of the fix (backout offending regressor? big rewrite?)
 - what's the available resource to fix it (OMG everyone's busy? get
to it this afternoon?)

Even if the answer to those questions is "unknown, but looking into
it" that's extremely valuable information to have.

> Alex runs channel meetings Tuesdays and Thursdays at 2pm PT where the channels are checked, nominations processed, and tracked bugs prodded. Please feel invited to attend if you want to develop your own intuition for how we manage those lists.

Also, it's entirely possible to generate queries of "what changed,
tracking-bug-wise, on a specific Tuesday or Thursday between 2pm and
3pm PT" - if you're unable to attend, Alex could whip those together
and publish them somewhere so you can see the changes at your leisure.

cheers,
mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

Michael Lefevre-3
In reply to this post by Johnathan Nightingale
On 21/03/2012 13:26, beltzner wrote:

> On Wed, Mar 21, 2012 at 6:51 AM, Johnathan Nightingale
> <[hidden email]>  wrote:
>> 1) As an expert in your domain, if it concerns you for a particular release, nominate it with a clear comment describing why. Better to over-nom than to under-nom, within reason.
>
> As a former triager (oh, the triage!) I also wish to emphasize: "with
> a clear comment describing why." Things that are obvious to you may be
> non-obvious to others, so as a guideline, include information about:
>
>   - how many users does it affect? (all OSes? specific hardware? common
> patterns in use on the web or a really hot demo site?)
>   - what's the complexity of the fix (backout offending regressor? big rewrite?)
>   - what's the available resource to fix it (OMG everyone's busy? get
> to it this afternoon?)
>
> Even if the answer to those questions is "unknown, but looking into
> it" that's extremely valuable information to have.

Going back to the bug in the example given, all this would seem to be a
bit of an obstacle. It looks like not only were the answers to all those
questions unknown, there was no "expert in the domain" looking at, only
QA volunteers, who were not "looking into it" because they could not
reproduce the bug, so they were waiting for the reporter (or someone
else who could reproduce) in order to get some answers.

So, if I'm not an expert in the domain, but a QA volunteer, and I don't
have answers to any of those questions but suspect that the bug may be
significant, do I put a tracking flag on it in the hopes that the
tracking flag will get some more eyes on it (like, someone who is an
expert in the area) so it can be effectively triaged? Or do I give it
2-3 weeks in the hopes that more information will be available (which
seems to be broadly what happened here)?

Michael
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Anatomy of a regression

beltzner
On Wed, Mar 21, 2012 at 4:36 PM, Michael Lefevre
<[hidden email]> wrote:
> So, if I'm not an expert in the domain, but a QA volunteer, and I don't have
> answers to any of those questions but suspect that the bug may be
> significant, do I put a tracking flag on it in the hopes that the tracking
> flag will get some more eyes on it (like, someone who is an expert in the
> area) so it can be effectively triaged? Or do I give it 2-3 weeks in the
> hopes that more information will be available (which seems to be broadly
> what happened here)?

You cc a domain expert (ask on IRC if you're unsure) and if you can't
find that person, you darned-tootin' put tracking+ on it and explain
why. Even a "I put tracking+ on this because it looks bad and I need
someone to help evaluate that."

Flags can be unset if it turns out that it was a false alarm.

cheers,
mike

(ps: Jesse's suggestion that component / module owners should do their
own triage of all new bugs was also good, but I don't think we're
there yet, discipline-wise, as an organization)
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
12