Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Chris AtLee-3
Hello interwebs,

We'll be having our first Go Fast meeting tomorrow morning at 8am
Pacific / 11am Eastern. The goal of Go Fast is to speed up our build and
test processes and infrastructure.

If you have ideas about how to make our build and test processes faster,
or have questions about how things work, this is the meeting for you!

Agenda, notes, and dial in information here:
https://wiki.mozilla.org/ReleaseEngineering/BuildFaster/Meetings/2011-07-07

Cheers,
Chris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Kyle Huey-2
On Wed, Jul 6, 2011 at 10:20 AM, Chris AtLee <[hidden email]> wrote:

> Hello interwebs,
>
> We'll be having our first Go Fast meeting tomorrow morning at 8am Pacific /
> 11am Eastern. The goal of Go Fast is to speed up our build and test
> processes and infrastructure.
>
> If you have ideas about how to make our build and test processes faster, or
> have questions about how things work, this is the meeting for you!
>
> Agenda, notes, and dial in information here: https://wiki.mozilla.org/**
> ReleaseEngineering/**BuildFaster/Meetings/2011-07-**07<https://wiki.mozilla.org/ReleaseEngineering/BuildFaster/Meetings/2011-07-07>
>
> Cheers,
> Chris
>

Minutes, or at least a summary, would be greatly appreciated!

- Kyle
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Chris AtLee-3
In reply to this post by Chris AtLee-3
Thanks to everybody who participated (and took notes!), we had a great
turnout and lots of great ideas.

Meeting notes are up here:
https://wiki.mozilla.org/ReleaseEngineering/BuildFaster/Meetings/2011-07-07#Notes

We're planning to have these meetings every 2 weeks. If you're
interested in attending regularly, please let me know so we can arrange
a day and time that works best.

Cheers,
Chris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Cameron McCormack-4
Go Fast meeting notes:
> * Some tests are slow due to Mochitest UI (big HTML table)

I was curious how much time not updating that table would save, so I
pushed a change to try that just set display:none on the results table
and kicked off a few mochitest-1 runs.  It seems it doesn’t make much
different for opt builds (both were ~18 minutes), but for debug builds
it shaves off a whole 20 minutes, down from ~55 mins to ~35 mins!

with table: http://tbpl.mozilla.org/?tree=Try&rev=ba432723ee1a
without table: http://tbpl.mozilla.org/?tree=Try&rev=5d4716c45f14

> * * Want to cleanup Mochitest UI for other reasons

Yeah…

--
Cameron McCormack ≝ http://mcc.id.au/
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Chris AtLee-3
In reply to this post by Chris AtLee-3
On 07/07/11 11:58 PM, Cameron McCormack wrote:
> Go Fast meeting notes:
>> * Some tests are slow due to Mochitest UI (big HTML table)
>
> I was curious how much time not updating that table would save, so I
> pushed a change to try that just set display:none on the results table
> and kicked off a few mochitest-1 runs.  It seems it doesn’t make much
> different for opt builds (both were ~18 minutes), but for debug builds
> it shaves off a whole 20 minutes, down from ~55 mins to ~35 mins!

Um...that's kind of awesome.

Can we turn this off permanently? Or for just our debug builds at least?

> with table: http://tbpl.mozilla.org/?tree=Try&rev=ba432723ee1a
> without table: http://tbpl.mozilla.org/?tree=Try&rev=5d4716c45f14
>
>> * * Want to cleanup Mochitest UI for other reasons
>
> Yeah…
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Clint Talbert-3
In reply to this post by Chris AtLee-3
Thanks for taking the initiative! I'm not too surprised.  Bug 479352 is
the bug tracking the explicit "not drawing the table".  There is quite a
bit of other optimization that could be done w.r.t. not updating said
table and not tying the actual test-running functionality of mochitest
to that UI as well.  (That's the larger scope of bug 661251).

If we can simply change this to display: none and save time today
without breaking any functionality while we do the larger refactoring,
then I'm all for it.

Clint
On 7/7/2011 8:58 PM, Cameron McCormack wrote:

> Go Fast meeting notes:
>> * Some tests are slow due to Mochitest UI (big HTML table)
>
> I was curious how much time not updating that table would save, so I
> pushed a change to try that just set display:none on the results table
> and kicked off a few mochitest-1 runs.  It seems it doesn’t make much
> different for opt builds (both were ~18 minutes), but for debug builds
> it shaves off a whole 20 minutes, down from ~55 mins to ~35 mins!
>
> with table: http://tbpl.mozilla.org/?tree=Try&rev=ba432723ee1a
> without table: http://tbpl.mozilla.org/?tree=Try&rev=5d4716c45f14
>
>> * * Want to cleanup Mochitest UI for other reasons
>
> Yeah…
>

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Zack Weinberg-2
On 2011-07-08 2:16 PM, Clint Talbert wrote:
> Thanks for taking the initiative! I'm not too surprised. Bug 479352 is
> the bug tracking the explicit "not drawing the table". There is quite a
> bit of other optimization that could be done w.r.t. not updating said
> table and not tying the actual test-running functionality of mochitest
> to that UI as well. (That's the larger scope of bug 661251).
>
> If we can simply change this to display: none and save time today
> without breaking any functionality while we do the larger refactoring,
> then I'm all for it.

The table is useful when you run mochitests locally, with the browser
window visible.  Is there a practical way to disable it only on the
build farm?

zw
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Cameron McCormack-4
Zack Weinberg:
> The table is useful when you run mochitests locally, with the
> browser window visible.  Is there a practical way to disable it only
> on the build farm?

What about making the table display:none by default and add a link or a
button to show it?

It certainly would be easy enough to add another command line argument
to runtests.py to hide it if we only want that to happen on the build
machines, though.

--
Cameron McCormack ≝ http://mcc.id.au/
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Robert O'Callahan-3
On Sat, Jul 9, 2011 at 2:54 PM, Cameron McCormack <[hidden email]> wrote:

> Zack Weinberg:
> > The table is useful when you run mochitests locally, with the
> > browser window visible.  Is there a practical way to disable it only
> > on the build farm?
>
> What about making the table display:none by default and add a link or a
> button to show it?
>
> It certainly would be easy enough to add another command line argument
> to runtests.py to hide it if we only want that to happen on the build
> machines, though.
>

Why don't we just figure out why updating the table is slow and fix it,
either by fixing Gecko or changing the way the table is updated?

Rob
--
"If we claim to be without sin, we deceive ourselves and the truth is not in
us. If we confess our sins, he is faithful and just and will forgive us our
sins and purify us from all unrighteousness. If we claim we have not sinned,
we make him out to be a liar and his word is not in us." [1 John 1:8-10]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Mike Shaver
On Sun, Jul 10, 2011 at 7:40 AM, Robert O'Callahan <[hidden email]> wrote:
> Why don't we just figure out why updating the table is slow and fix it,
> either by fixing Gecko or changing the way the table is updated?

We should do that, I agree, but I don't think we should leave an
extra, useless 20 mins in every mochi-1 debug cycle until that's
finished.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Boris Zbarsky
In reply to this post by Cameron McCormack-4
On 7/10/11 7:40 AM, Robert O'Callahan wrote:
> Why don't we just figure out why updating the table is slow and fix it

There are several different things there.

First of all, some tests in the test suite flip the "show numbers using
alternate codepoints" pref.  The pref is process-wide; when it's flipped
we have to rerender the entire table (which is full of numbers, if you
recall, and huge).  The tests flip this pref a bunch of times.  I'm not
sure whether there's a way to improve this on the Gecko side; on the
test side we may be able to combine some tests so there are fewer pref
flips, maybe.

Second, during a test run we update the table to contain the
pass/fail/whatever counts.  This almost certainly means a column
rebalance, because we're changing the content.  I'm not sure how to
address this on either the Gecko or the test side.

There may be other performance chokepoints, of course.  It's a bit hard
to measure because it's scattered over many tests...

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Robert O'Callahan-3
On Mon, Jul 11, 2011 at 3:18 AM, Boris Zbarsky <[hidden email]> wrote:

> First of all, some tests in the test suite flip the "show numbers using
> alternate codepoints" pref.  The pref is process-wide; when it's flipped we
> have to rerender the entire table (which is full of numbers, if you recall,
> and huge).  The tests flip this pref a bunch of times.  I'm not sure whether
> there's a way to improve this on the Gecko side; on the test side we may be
> able to combine some tests so there are fewer pref flips, maybe.
>

No point in improving perf on that on the Gecko side, I agree. Maybe we
could just make the pref flip be non-live.

Second, during a test run we update the table to contain the
> pass/fail/whatever counts.  This almost certainly means a column rebalance,
> because we're changing the content.  I'm not sure how to address this on
> either the Gecko or the test side.
>

Put the counts in fixed-width blocks and optimize Gecko so that we don't
have to rebalance in that case?

Rob
--
"If we claim to be without sin, we deceive ourselves and the truth is not in
us. If we confess our sins, he is faithful and just and will forgive us our
sins and purify us from all unrighteousness. If we claim we have not sinned,
we make him out to be a liar and his word is not in us." [1 John 1:8-10]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Boris Zbarsky
In reply to this post by Boris Zbarsky
On 7/10/11 6:42 PM, Robert O'Callahan wrote:

> On Mon, Jul 11, 2011 at 3:18 AM, Boris Zbarsky<[hidden email]>  wrote:
>
>> First of all, some tests in the test suite flip the "show numbers using
>> alternate codepoints" pref.  The pref is process-wide; when it's flipped we
>> have to rerender the entire table (which is full of numbers, if you recall,
>> and huge).  The tests flip this pref a bunch of times.  I'm not sure whether
>> there's a way to improve this on the Gecko side; on the test side we may be
>> able to combine some tests so there are fewer pref flips, maybe.
>>
>
> No point in improving perf on that on the Gecko side, I agree. Maybe we
> could just make the pref flip be non-live.
>
> Second, during a test run we update the table to contain the
>> pass/fail/whatever counts.  This almost certainly means a column rebalance,
>> because we're changing the content.  I'm not sure how to address this on
>> either the Gecko or the test side.
>>
>
> Put the counts in fixed-width blocks and optimize Gecko so that we don't
> have to rebalance in that case?

Hmm.  Those are both workable.  File bugs?

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Ehsan Akhgari
In reply to this post by Robert O'Callahan-3
On 11-07-10 6:42 PM, Robert O'Callahan wrote:

> On Mon, Jul 11, 2011 at 3:18 AM, Boris Zbarsky<[hidden email]>  wrote:
>
>> First of all, some tests in the test suite flip the "show numbers using
>> alternate codepoints" pref.  The pref is process-wide; when it's flipped we
>> have to rerender the entire table (which is full of numbers, if you recall,
>> and huge).  The tests flip this pref a bunch of times.  I'm not sure whether
>> there's a way to improve this on the Gecko side; on the test side we may be
>> able to combine some tests so there are fewer pref flips, maybe.
>>
>
> No point in improving perf on that on the Gecko side, I agree. Maybe we
> could just make the pref flip be non-live.

IINM that would make it impossible to test those prefs.

Ehsan
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Boris Zbarsky
In reply to this post by Robert O'Callahan-3
On 7/12/11 3:14 PM, Ehsan Akhgari wrote:
>> No point in improving perf on that on the Gecko side, I agree. Maybe we
>> could just make the pref flip be non-live.
>
> IINM that would make it impossible to test those prefs.

Non-live would just mean they only affect documents loaded after the
pref is changed, I assume.  So you could still test them: flip the pref,
then load the test document.

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Ehsan Akhgari
On 11-07-12 3:19 PM, Boris Zbarsky wrote:
> On 7/12/11 3:14 PM, Ehsan Akhgari wrote:
>>> No point in improving perf on that on the Gecko side, I agree. Maybe we
>>> could just make the pref flip be non-live.
>>
>> IINM that would make it impossible to test those prefs.
>
> Non-live would just mean they only affect documents loaded after the
> pref is changed, I assume. So you could still test them: flip the pref,
> then load the test document.

Oh, OK, I see.  But honestly, if we can just hide the table and get the
perf win, I don't think that is worth doing, since most users won't
toggle this pref anyways...

Ehsan
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Mike Shaver
And didn't the data show that it was only a noticeable performance
difference on debug builds? seems like we shouldn't get too worked up about
structural fixes for that if so.

Mike
On Jul 12, 2011 6:32 PM, "Ehsan Akhgari" <[hidden email]> wrote:

> On 11-07-12 3:19 PM, Boris Zbarsky wrote:
>> On 7/12/11 3:14 PM, Ehsan Akhgari wrote:
>>>> No point in improving perf on that on the Gecko side, I agree. Maybe we
>>>> could just make the pref flip be non-live.
>>>
>>> IINM that would make it impossible to test those prefs.
>>
>> Non-live would just mean they only affect documents loaded after the
>> pref is changed, I assume. So you could still test them: flip the pref,
>> then load the test document.
>
> Oh, OK, I see. But honestly, if we can just hide the table and get the
> perf win, I don't think that is worth doing, since most users won't
> toggle this pref anyways...
>
> Ehsan
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Cameron McCormack-4
In reply to this post by Chris AtLee-3
Chris AtLee:
> Can we turn this off permanently? Or for just our debug builds at least?

This landed earlier today for both debug and opt builds (although not
for Android builds, yet).  Without doing multiple runs of anything, so
bearing in the mind the usual variance you get with build times, here
are the results before and after:

http://mcc.id.au/temp/2011/times.html

So we are getting quite substantial time savings in some places, in
particular for debug builds (Linux64 mochitest-4 down from 72m16s to
17m48s!) and less so, but still some savings, for opt builds.

The three times that increased are increases of only a couple of
minutes, within the expected differences that you might normally find, I
think.

mochitest-other runs don’t get as much of a saving, since the
browser-chrome harness didn’t previously show a results table anyway.

--
Cameron McCormack ≝ http://mcc.id.au/
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Cameron McCormack-4
Cameron McCormack:
> This landed earlier today for both debug and opt builds

and backed out for causing (or unearthing) hangs-on-shutdown on Linux64
opt debug mochitest-1 runs.

--
Cameron McCormack ≝ http://mcc.id.au/
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Go Fast Meeting: Thursday July 7, 2011 @ 8am PDT / 11am EDT

Cameron McCormack-4
Cameron McCormack:
> and backed out for causing (or unearthing) hangs-on-shutdown on Linux64
> opt debug mochitest-1 runs.

jgriffin worked around the hang, and it landed again.  Updated times, if
anyone is interested: http://mcc.id.au/temp/2011/times2.html
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
12