Rethinking TBPL and intermittent orange bug comments

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Rethinking TBPL and intermittent orange bug comments

Ed Morley
Hi all

As part of my new role on the Automation & Tools team, I'd like to make a number of improvements to TBPL, sheriffing & the way we deal with intermittent oranges.

The first of these is this:

At the moment, whenever anyone stars an intermittent orange on TBPL, the TBPL Robot posts a bug comment to the relevant orange bug (example: https://bugzilla.mozilla.org/show_bug.cgi?id=677841#c1) & also stores the star message in the database.

I propose that we no longer comment in the bug, for the following reasons:

* Some orange bugs have over 1000 of these automated comments, making the bug page ridiculously slow to load & meaning it's extremely hard to spot the actual comments discussing cause/fix. In some cases, devs even go so far as to create dependant bugs in which to do the work, since the original bug is unusable due to noise.

* Sheriffs tend to see the resultant bugspam as a means to nag developers to fix the oranges, but in reality many of them filter the bugmail to trash - and what we should really be doing instead, is deciding as a community if fixing oranges is important (I personally think it is, but I'm slightly biased! ;-) & if so, get managers to prioritise appropriately.

* The prioritisation of what oranges we should be tackling first should be done using OrangeFactor (http://brasstacks.mozilla.com/orangefactor/), however it does not use/need the TBPL Robot bug comments. At present, the 'spam devs with TBPL bot mail' approach can result in less common oranges receiving attention from well-meaning devs who happen to be CC'd, whilst other more common oranges who don't have anyone CC'd get sidelined. It's as daft as doing platform performance optimisation work without first looking at a profiler.

* The bug comments do serve a few other purposes, such as indicating which oranges still occur vs those that can be eventually resolved WFM - however I feel this would be much better suited as a say weekly post by OrangeFactor to each bug if it has occurred during that week, with: "This orange has occurred X times in the last 7 days, <platform breakdown>, <link to orange factor stats/details page etc>". OrangeFactor already reports the week's top oranges to dev.tree-managements & this could perhaps be extended to indicate which top-oranges are unassigned & need owners (now that devs won't mind setting themselves as the assignee, since there won't be TBPL bot spam).

Mark Côté has offered to make the necessary changes to OrangeFactor, but before we do anything we wanted to run this by everyone. Comments as to suggested interval of OrangeFactor posting to bugs and also contents of the message are welcome :-)

Many thanks,


Ed
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking TBPL and intermittent orange bug comments

Jonathan Kew-3
On 23/4/12 12:33, Ed Morley wrote:

> Hi all
>
> As part of my new role on the Automation&  Tools team, I'd like to
> make a number of improvements to TBPL, sheriffing&  the way we deal
> with intermittent oranges.
>
> The first of these is this:
>
> At the moment, whenever anyone stars an intermittent orange on TBPL,
> the TBPL Robot posts a bug comment to the relevant orange bug
> (example: https://bugzilla.mozilla.org/show_bug.cgi?id=677841#c1)&
> also stores the star message in the database.
>
> I propose that we no longer comment in the bug, for the following
> reasons:
 > ....

While I'm generally in agreement with the suggestions here, I do
sometimes find the auto-comments useful: when looking at an orange on
TBPL, it's sometimes not immediately clear whether (one of) the
suggested bug(s) is actually appropriate, and in that case I'll often
look at some of the earlier comments - at least the summary that TBPL
has posted, and sometimes even loading the linked logs - to compare with
the current orange and help in deciding how to star.

I'm not sure this is enough to justify spamming the bugs with hundreds
(or even thousands) of such comments, but if it becomes less convenient
to re-examine earlier examples of a given orange, it seems like we'll
have lost something useful, and correctly starring oranges as they
appear may become a little harder.

JK
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking TBPL and intermittent orange bug comments

Ed Morley
In reply to this post by Ed Morley
On Monday, April 23, 2012 1:05:46 PM UTC+1, Jonathan Kew wrote:
> While I'm generally in agreement with the suggestions here, I do
> sometimes find the auto-comments useful: when looking at an orange on
> TBPL, it's sometimes not immediately clear whether (one of) the
> suggested bug(s) is actually appropriate, and in that case I'll often
> look at some of the earlier comments - at least the summary that TBPL
> has posted, and sometimes even loading the linked logs - to compare with
> the current orange and help in deciding how to star.

We could make sure that the TBPL summary/bug suggestion box also links to the OrangeFactor page [1] for each suggestion (which would then have details of logs etc), thereby preserving the ability to do that?

[1] eg: http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=746876&entireHistory=true&tree=mozilla-central
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking TBPL and intermittent orange bug comments

Andrew  McCreight
In reply to this post by Ed Morley


This sounds like a good idea, but I have a few suggestions.

> * The bug comments do serve a few other purposes, such as indicating
> which oranges still occur vs those that can be eventually resolved
> WFM - however I feel this would be much better suited as a say
> weekly post by OrangeFactor to each bug if it has occurred during
> that week, with: "This orange has occurred X times in the last 7
> days, <platform breakdown>, <link to orange factor stats/details
> page etc>".

This should also have breakdown by tree, for cases where a random orange has been fixed on trunk, but not on Aurora or earlier.

> OrangeFactor already reports the week's top oranges to
> dev.tree-managements & this could perhaps be extended to indicate
> which top-oranges are unassigned & need owners (now that devs won't
> mind setting themselves as the assignee, since there won't be TBPL
> bot spam).

A week seems like too long of a delay for a resurgent intermittent orange.  It would be nice if there could be some kind of logic like "We haven't seen this random orange in more than a week, and we've seen it twice today already", and then you'd get a summary for, say, one day.  If a patch causes a random orange to return, the sooner we realize that the better, as it is harder to investigate things further back.

Thanks,

Andrew

>
> Mark Côté has offered to make the necessary changes to OrangeFactor,
> but before we do anything we wanted to run this by everyone.
> Comments as to suggested interval of OrangeFactor posting to bugs
> and also contents of the message are welcome :-)
>
> Many thanks,
>
>
> Ed
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking TBPL and intermittent orange bug comments

Phil Ringnalda
In reply to this post by Ed Morley
On 4/23/2012 4:33 AM, Ed Morley wrote:
> Mark Côté has offered to make the necessary changes to OrangeFactor

Where's the bug on "Completely rewrite Orange Factor to only use things
which IT is willing to support, and then move it onto tier 1 hardware"?
I'd like to cc myself on it, since there's no point whatsoever in
talking about "Orange Factor will do everything we need" until it will
do the most basic thing we need, actually be available (it's also sort
of difficult to see what things we need it to do that it doesn't
currently do based on vague memories of the time before the most recent
extended outage).

There's one thing I'm sure it will not do, even without having seen it
for a week or so, though:

https://bugzilla.mozilla.org/show_bug.cgi?id=734554#c71 and
https://bugzilla.mozilla.org/show_bug.cgi?id=738652#c144
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking TBPL and intermittent orange bug comments

Ehsan Akhgari
In reply to this post by Ed Morley
I'm not sure if this is a good idea, unless Orange Factor grows to provide
the exact same data as can be currently found in the bug comments.  Once it
does (and it also satisfies the availability requirements that philor
points out, I'd be happy to make the robot stop spamming the bugs.

Cheers,
--
Ehsan
<http://ehsanakhgari.org/>


On Mon, Apr 23, 2012 at 7:33 AM, Ed Morley
<[hidden email]>wrote:

> Hi all
>
> As part of my new role on the Automation & Tools team, I'd like to make a
> number of improvements to TBPL, sheriffing & the way we deal with
> intermittent oranges.
>
> The first of these is this:
>
> At the moment, whenever anyone stars an intermittent orange on TBPL, the
> TBPL Robot posts a bug comment to the relevant orange bug (example:
> https://bugzilla.mozilla.org/show_bug.cgi?id=677841#c1) & also stores the
> star message in the database.
>
> I propose that we no longer comment in the bug, for the following reasons:
>
> * Some orange bugs have over 1000 of these automated comments, making the
> bug page ridiculously slow to load & meaning it's extremely hard to spot
> the actual comments discussing cause/fix. In some cases, devs even go so
> far as to create dependant bugs in which to do the work, since the original
> bug is unusable due to noise.
>
> * Sheriffs tend to see the resultant bugspam as a means to nag developers
> to fix the oranges, but in reality many of them filter the bugmail to trash
> - and what we should really be doing instead, is deciding as a community if
> fixing oranges is important (I personally think it is, but I'm slightly
> biased! ;-) & if so, get managers to prioritise appropriately.
>
> * The prioritisation of what oranges we should be tackling first should be
> done using OrangeFactor (http://brasstacks.mozilla.com/orangefactor/),
> however it does not use/need the TBPL Robot bug comments. At present, the
> 'spam devs with TBPL bot mail' approach can result in less common oranges
> receiving attention from well-meaning devs who happen to be CC'd, whilst
> other more common oranges who don't have anyone CC'd get sidelined. It's as
> daft as doing platform performance optimisation work without first looking
> at a profiler.
>
> * The bug comments do serve a few other purposes, such as indicating which
> oranges still occur vs those that can be eventually resolved WFM - however
> I feel this would be much better suited as a say weekly post by
> OrangeFactor to each bug if it has occurred during that week, with: "This
> orange has occurred X times in the last 7 days, <platform breakdown>, <link
> to orange factor stats/details page etc>". OrangeFactor already reports the
> week's top oranges to dev.tree-managements & this could perhaps be extended
> to indicate which top-oranges are unassigned & need owners (now that devs
> won't mind setting themselves as the assignee, since there won't be TBPL
> bot spam).
>
> Mark Côté has offered to make the necessary changes to OrangeFactor, but
> before we do anything we wanted to run this by everyone. Comments as to
> suggested interval of OrangeFactor posting to bugs and also contents of the
> message are welcome :-)
>
> Many thanks,
>
>
> Ed
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking TBPL and intermittent orange bug comments

Chris Pearce-3
In reply to this post by Ed Morley
On 24/04/2012 6:55 a.m., Ehsan Akhgari wrote:
> I'm not sure if this is a good idea, unless Orange Factor grows to provide
> the exact same data as can be currently found in the bug comments.  Once it
> does (and it also satisfies the availability requirements that philor
> points out, I'd be happy to make the robot stop spamming the bugs.

+1. I usually find orange factor times out when I try access it (that
didn't use to be the norm), making it too unreliable to be relied upon.

Plus, people who find tbpl comments overwhelming in bugs should use the
Bugzilla Tweaks extension to hide tbpl robot comments.

I filter the tbpl robot's emails and what look like manual nag comments  
(sorry Philor!) into a separate folder. I periodically go through and
using this data plus search filters I can quickly and roughly get a feel
for the number of failures per day we're hitting in the tests I care
about (i.e. those written/maintained by me). I find this quite adequate
for filtering the things I care about. Please don't break it until
you've got a better and more reliable system.

Cheers,
Chris P.

> Cheers,
> --
> Ehsan
> <http://ehsanakhgari.org/>
>
>
> On Mon, Apr 23, 2012 at 7:33 AM, Ed Morley
> <[hidden email]>wrote:
>
>> Hi all
>>
>> As part of my new role on the Automation&  Tools team, I'd like to make a
>> number of improvements to TBPL, sheriffing&  the way we deal with
>> intermittent oranges.
>>
>> The first of these is this:
>>
>> At the moment, whenever anyone stars an intermittent orange on TBPL, the
>> TBPL Robot posts a bug comment to the relevant orange bug (example:
>> https://bugzilla.mozilla.org/show_bug.cgi?id=677841#c1)&  also stores the
>> star message in the database.
>>
>> I propose that we no longer comment in the bug, for the following reasons:
>>
>> * Some orange bugs have over 1000 of these automated comments, making the
>> bug page ridiculously slow to load&  meaning it's extremely hard to spot
>> the actual comments discussing cause/fix. In some cases, devs even go so
>> far as to create dependant bugs in which to do the work, since the original
>> bug is unusable due to noise.
>>
>> * Sheriffs tend to see the resultant bugspam as a means to nag developers
>> to fix the oranges, but in reality many of them filter the bugmail to trash
>> - and what we should really be doing instead, is deciding as a community if
>> fixing oranges is important (I personally think it is, but I'm slightly
>> biased! ;-)&  if so, get managers to prioritise appropriately.
>>
>> * The prioritisation of what oranges we should be tackling first should be
>> done using OrangeFactor (http://brasstacks.mozilla.com/orangefactor/),
>> however it does not use/need the TBPL Robot bug comments. At present, the
>> 'spam devs with TBPL bot mail' approach can result in less common oranges
>> receiving attention from well-meaning devs who happen to be CC'd, whilst
>> other more common oranges who don't have anyone CC'd get sidelined. It's as
>> daft as doing platform performance optimisation work without first looking
>> at a profiler.
>>
>> * The bug comments do serve a few other purposes, such as indicating which
>> oranges still occur vs those that can be eventually resolved WFM - however
>> I feel this would be much better suited as a say weekly post by
>> OrangeFactor to each bug if it has occurred during that week, with: "This
>> orange has occurred X times in the last 7 days,<platform breakdown>,<link
>> to orange factor stats/details page etc>". OrangeFactor already reports the
>> week's top oranges to dev.tree-managements&  this could perhaps be extended
>> to indicate which top-oranges are unassigned&  need owners (now that devs
>> won't mind setting themselves as the assignee, since there won't be TBPL
>> bot spam).
>>
>> Mark Côté has offered to make the necessary changes to OrangeFactor, but
>> before we do anything we wanted to run this by everyone. Comments as to
>> suggested interval of OrangeFactor posting to bugs and also contents of the
>> message are welcome :-)
>>
>> Many thanks,
>>
>>
>> Ed
>> _______________________________________________
>> dev-planning mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-planning
>>

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking TBPL and intermittent orange bug comments

Byron Jones-5
Chris Pearce wrote:
> Plus, people who find tbpl comments overwhelming in bugs should use
> the Bugzilla Tweaks extension to hide tbpl robot comments.
there's no need to install tweaks for this; it's been rolled into bmo
(look for "collapse TinderboxPushlog Comments").


--
byron - irc:glob - bugzilla.mozilla.org team -

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking TBPL and intermittent orange bug comments

Jonathan Kew-3
On 24/4/12 08:18, Byron Jones wrote:
> Chris Pearce wrote:
>> Plus, people who find tbpl comments overwhelming in bugs should use
>> the Bugzilla Tweaks extension to hide tbpl robot comments.
> there's no need to install tweaks for this; it's been rolled into bmo
> (look for "collapse TinderboxPushlog Comments").

Oh, cool - I hadn't noticed that! Though I find myself wishing it could
hide them altogether (including the comment header), not just collapse
the content. Or even collapse consecutive TBPL comments into a single
header that just says "TBPL Robot - <number> comments". Any chance? :)

JK
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking TBPL and intermittent orange bug comments

Ehsan Akhgari
In reply to this post by Byron Jones-5
On Tue, Apr 24, 2012 at 3:18 AM, Byron Jones <[hidden email]> wrote:

> Chris Pearce wrote:
>
>> Plus, people who find tbpl comments overwhelming in bugs should use the
>> Bugzilla Tweaks extension to hide tbpl robot comments.
>>
> there's no need to install tweaks for this; it's been rolled into bmo
> (look for "collapse TinderboxPushlog Comments").


I filed https://bugzilla.mozilla.org/show_bug.cgi?id=748629 in order to
make those comments collapsed by default.  This will hopefully help people
deal with those bugs easier.

--
Ehsan
<http://ehsanakhgari.org/>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning