Goals for "landing code" experience?

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Goals for "landing code" experience?

Christian Legnitto

On Dec 29, 2011, at 10:05 AM, Matt Brubeck wrote:

> On 12/29/2011 09:03 AM, Ehsan Akhgari wrote:
>>> I partially disagree, based on my experience in fixing random oranges in
>>> the "old" system. It was not really different from what happens today,
>>> there were just less changesets, but random failures were still not burning
>>> the tree and hard to detect in time.
>
>> I should also say that I strongly disagree with the assertion of the "old"
>> system being better because we had fewer tests and fewer code changes!
>
> And fewer platforms!
>
> At one point we -- mostly Ehsan :) -- had reduced the average orange count to less than 2 per push.  Since then it's climbed back up to 4 or more.  During that time, in addition to adding new tests and code, we've also added OS X 10.7 builds, native Android builds, non-PGO builds on Linux and Windows, etc.   So even if the number of intermittently failing tests had stayed the same, we would still expect to see the number of failures per push growing.
>
> Hopefully this part of the trend will level off now since I'm not aware of any imminent new platforms.  Eventually we will even retire some, like Android XUL and OS X 10.5.


Also, for those that don't know where we track this:

    http://brasstacks.mozilla.com/orangefactor/

I've also looked at what webkit does a little. I really like this view:

   http://test-results.appspot.com/dashboards/flakiness_dashboard.html (warning, large table)

If we had a view like that I could see "oh, this test has been flaky, probably not worth bugging people" vs "oh, this test has been green forever and it's now orange, I should look into it". The key part is that flakiness is not really a number...the trend matters.

Thanks,
Christian

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Goals for "landing code" experience?

Ehsan Akhgari
On Thu, Dec 29, 2011 at 1:31 PM, Christian Legnitto
<[hidden email]>wrote:

>
> On Dec 29, 2011, at 10:05 AM, Matt Brubeck wrote:
>
> > On 12/29/2011 09:03 AM, Ehsan Akhgari wrote:
> >>> I partially disagree, based on my experience in fixing random oranges
> in
> >>> the "old" system. It was not really different from what happens today,
> >>> there were just less changesets, but random failures were still not
> burning
> >>> the tree and hard to detect in time.
> >
> >> I should also say that I strongly disagree with the assertion of the
> "old"
> >> system being better because we had fewer tests and fewer code changes!
> >
> > And fewer platforms!
> >
> > At one point we -- mostly Ehsan :) -- had reduced the average orange
> count to less than 2 per push.  Since then it's climbed back up to 4 or
> more.  During that time, in addition to adding new tests and code, we've
> also added OS X 10.7 builds, native Android builds, non-PGO builds on Linux
> and Windows, etc.   So even if the number of intermittently failing tests
> had stayed the same, we would still expect to see the number of failures
> per push growing.
> >
> > Hopefully this part of the trend will level off now since I'm not aware
> of any imminent new platforms.  Eventually we will even retire some, like
> Android XUL and OS X 10.5.
>
>
> Also, for those that don't know where we track this:
>
>    http://brasstacks.mozilla.com/orangefactor/
>
> I've also looked at what webkit does a little. I really like this view:
>
>   http://test-results.appspot.com/dashboards/flakiness_dashboard.html(warning, large table)
>
> If we had a view like that I could see "oh, this test has been flaky,
> probably not worth bugging people" vs "oh, this test has been green forever
> and it's now orange, I should look into it". The key part is that flakiness
> is not really a number...the trend matters.
>

We do have something similar to that, see <
http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=482975&startday=2011-12-01&endday=2011-12-29&tree=mozilla-central>
for example.

--
Ehsan
<http://ehsanakhgari.org/>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Goals for "landing code" experience?

Nicholas Nethercote
One thing that hasn't been said:  the fact that (a) TBPL auto-matches
oranges against bugs, and (b) people like philor are very good at
filing bugs for oranges, means that things are much better than they
used to be.  90% of the oranges I get have a bug suggested for them
that is clearly a match.  A year or two ago we didn't have these
features and every orange was a head-scratcher.

Furthermore, inbound is nice because (a) you don't have to watch the
whole time (though I usually do) and (b) there's no "oh my god you
can't break the tree" pressure the way there is with
mozilla-central/aurora/beta.

And TBPL is *so* much better than the old tinderbox.

Overall, in the 3 years I've been working for Mozilla, the situation
is far better now than it was in the past, IMO.

I wonder if the auto-coalescing should be turned off, though.

Nick
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Goals for "landing code" experience?

Marco Bonardo-2
In reply to this post by Christian Legnitto
On 29/12/2011 20:07, Ehsan Akhgari wrote:
> On Thu, Dec 29, 2011 at 1:31 PM, Christian Legnitto
> <[hidden email]>wrote:

>> If we had a view like that I could see "oh, this test has been flaky,
>> probably not worth bugging people" vs "oh, this test has been green forever
>> and it's now orange, I should look into it". The key part is that flakiness
>> is not really a number...the trend matters.
>>
>
> We do have something similar to that, see<
> http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=482975&startday=2011-12-01&endday=2011-12-29&tree=mozilla-central>
> for example.

My complain is that this is a system that people have to look at
explicitly, but unfortunately it's not at the level of things developers
look at daily or weekly. I suppose if we'd do a poll very few developers
have looked at the orange factor in the last 2 weeks (let alone it was
christmas, my point is more generic).
So as a first step we may find a way to expose this to developers
periodically in places they usually look at. Thus my suggestion to
publish an oranges digest on planet, like once a week. In future when
everyone is more used to care about oranges maybe the orange factor will
get more attention by itself.

-m
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Goals for "landing code" experience?

Mark Côté
On 11-12-29 03:34 PM, Marco Bonardo wrote:

> On 29/12/2011 20:07, Ehsan Akhgari wrote:
>> On Thu, Dec 29, 2011 at 1:31 PM, Christian Legnitto
>> <[hidden email]>wrote:
>
>>> If we had a view like that I could see "oh, this test has been flaky,
>>> probably not worth bugging people" vs "oh, this test has been green
>>> forever
>>> and it's now orange, I should look into it". The key part is that
>>> flakiness
>>> is not really a number...the trend matters.
>>>
>>
>> We do have something similar to that, see<
>> http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=482975&startday=2011-12-01&endday=2011-12-29&tree=mozilla-central>
>>
>> for example.
>
> My complain is that this is a system that people have to look at
> explicitly, but unfortunately it's not at the level of things developers
> look at daily or weekly. I suppose if we'd do a poll very few developers
> have looked at the orange factor in the last 2 weeks (let alone it was
> christmas, my point is more generic).
> So as a first step we may find a way to expose this to developers
> periodically in places they usually look at. Thus my suggestion to
> publish an oranges digest on planet, like once a week. In future when
> everyone is more used to care about oranges maybe the orange factor will
> get more attention by itself.
>
> -m

Top 10 weekly oranges *should* have been going to the
dev.tree-management mailing list for some time now. I still have to hook
up NNTP support so that it goes to the actual newsgroup, though.  A blog
post is also a possibility.

Mark

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Goals for "landing code" experience?

Mark Côté
Okay, OrangeFactor reports should now be sent to the dev.tree-management
group every Tuesday morning.  I also ran it manually just now; the first
such report is at

http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/thread/62e5415f403f68b4#

(NB: There was a hiccup, or actually a few hiccups, in the last couple
weeks in which we lost the test-run count, hence the "?" OF from the
previous week.)

Is it still worthwhile to have it sent to Planet as well?  Or perhaps
cross-posted to other groups?

Mark


On 12-01-03 02:16 PM, Mark Cote wrote:

> On 11-12-29 03:34 PM, Marco Bonardo wrote:
>> On 29/12/2011 20:07, Ehsan Akhgari wrote:
>>> On Thu, Dec 29, 2011 at 1:31 PM, Christian Legnitto
>>> <[hidden email]>wrote:
>>
>>>> If we had a view like that I could see "oh, this test has been flaky,
>>>> probably not worth bugging people" vs "oh, this test has been green
>>>> forever
>>>> and it's now orange, I should look into it". The key part is that
>>>> flakiness
>>>> is not really a number...the trend matters.
>>>>
>>>
>>> We do have something similar to that, see<
>>> http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=482975&startday=2011-12-01&endday=2011-12-29&tree=mozilla-central>
>>>
>>>
>>> for example.
>>
>> My complain is that this is a system that people have to look at
>> explicitly, but unfortunately it's not at the level of things developers
>> look at daily or weekly. I suppose if we'd do a poll very few developers
>> have looked at the orange factor in the last 2 weeks (let alone it was
>> christmas, my point is more generic).
>> So as a first step we may find a way to expose this to developers
>> periodically in places they usually look at. Thus my suggestion to
>> publish an oranges digest on planet, like once a week. In future when
>> everyone is more used to care about oranges maybe the orange factor will
>> get more attention by itself.
>>
>> -m
>
> Top 10 weekly oranges *should* have been going to the
> dev.tree-management mailing list for some time now. I still have to hook
> up NNTP support so that it goes to the actual newsgroup, though. A blog
> post is also a possibility.
>
> Mark
>

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Goals for "landing code" experience?

Axel Hecht-2
In reply to this post by Ehsan Akhgari
On 29.12.11 21:10, Nicholas Nethercote wrote:

> One thing that hasn't been said:  the fact that (a) TBPL auto-matches
> oranges against bugs, and (b) people like philor are very good at
> filing bugs for oranges, means that things are much better than they
> used to be.  90% of the oranges I get have a bug suggested for them
> that is clearly a match.  A year or two ago we didn't have these
> features and every orange was a head-scratcher.
>
> Furthermore, inbound is nice because (a) you don't have to watch the
> whole time (though I usually do) and (b) there's no "oh my god you
> can't break the tree" pressure the way there is with
> mozilla-central/aurora/beta.
>
> And TBPL is *so* much better than the old tinderbox.
>
> Overall, in the 3 years I've been working for Mozilla, the situation
> is far better now than it was in the past, IMO.
>
> I wonder if the auto-coalescing should be turned off, though.
>
> Nick

I did some research, and it seems that the last patch I landed for the
module I own was landed yesterday, 6 years ago. The glory of owning RDF.
Just to clarify which past I'm talking about.

I'm not debating that we had darker times since then for the tree, but
that's also not really relevant to how I feel about landing code today.

Axel
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Goals for "landing code" experience?

Robert O'Callahan-3
On Sat, Jan 7, 2012 at 1:15 AM, Axel Hecht <[hidden email]> wrote:

> I did some research, and it seems that the last patch I landed for the
> module I own was landed yesterday, 6 years ago. The glory of owning RDF.
> Just to clarify which past I'm talking about.
>
> I'm not debating that we had darker times since then for the tree, but
> that's also not really relevant to how I feel about landing code today.
>

Like others on this thread, I feel that the current environment for landing
code is the best it's ever been (for a non-inbound-wrangler), and I've been
committing code regularly for over ten years now.

Six years ago we had fewer platforms and hardly any tests so it's not even
close to a fair comparison, and even so watching the Tinderbox waterfall
was a real pain. And if you did regress a test, e.g. Tp, good luck with no
try-server.

Rob
--
"If we claim to be without sin, we deceive ourselves and the truth is not
in us. If we confess our sins, he is faithful and just and will forgive us
our sins and purify us from all unrighteousness. If we claim we have not
sinned, we make him out to be a liar and his word is not in us." [1 John
1:8-10]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
12