Performance Test Reference Platforms

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance Test Reference Platforms

Robert Helmer-5
Hello,

We need to have performance test reference platforms, like we do for
build platforms (see http://wiki.mozilla.org/ReferencePlatforms).

Right now we have one standalone Fedora Linux test box (bl-bldlnx01)
and one standalone Windows XP test box (bl-bldxp01), the Linux box is
pretty vanilla while the Windows box has several third-party utilities
(mostly installed through Cygwin).

I am going to be working on that later this week, starting with a
Windows XP reference platform. If anyone has any comments please let me
know. I am going to try to stick to the same version of tools that the
build platforms use, but there will be some unneeded dependencies (like
a compiler) and some new deps for the test harness that Alice is
working on.

I imagine the XP and Mac installs will end up being pretty much
out-of-the-box, while we'll want to disable a lot of services on Linux.
If anyone has hints along these lines for getting more stable numbers,
I'd love to hear about it!


Thanks!
Rob

_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Boris Zbarsky
Robert Helmer wrote:
> We need to have performance test reference platforms, like we do for
> build platforms (see http://wiki.mozilla.org/ReferencePlatforms).

Except we want multiple such (different operating system versions, etc).
  Imo.  Far too often we've had perf issues due to interactions with
particular OS versions that were really hard to track down.

-Boris
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Boris Zbarsky
For what it's worth, this thread shows some serious issues with our
e-mail gateways -- Mike's response ended up going where it shouldn't
have (my e-mail) and not where it should have (_both_ of the newsgroups
the post he was replying to was posted to).

Mike Shaver wrote:
> Is this something that was primarily problematic on Linux, or have
> there been a lot of problems with inter-version effects between XP
> service pack levels, etc.?

There have for sure been differences between XP and 2K, not to even
mention between those and 9x.  Mostly GDI differences, but some that we
just never figured out.

I would be incredibly surprised if there were not differences between XP
and Vista, especially given the rendering arch changes in Vista.

There have also been changes between OS X versions, as I recall.

And as we use more platform stuff (Uniscribe, etc), I personally expect
the OS version performance skew to increase.

And yes, there are differences on Linux.  Probably by distribution, and
certainly by X version.  I agree that if we could get the Linux
distributors to contribute tinderboxen for this that would be most
excellent.  Note that we'd want such tinderboxen for more than just
their "latest and greatest" distro version, which might be a hard sell
given my experiences with distros.  ;)

If we do want this to happen, and I agree that we do, we should have a
point man for this; my last conversation with caillon (about pango and
such) indicated to me that we have close to 0 communication with the
distros on technical issues...

-Boris
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Robert Helmer-5
In reply to this post by Boris Zbarsky
Boris Zbarsky wrote:
> Robert Helmer wrote:
> > We need to have performance test reference platforms, like we do for
> > build platforms (see http://wiki.mozilla.org/ReferencePlatforms).
>
> Except we want multiple such (different operating system versions, etc).
>   Imo.  Far too often we've had perf issues due to interactions with
> particular OS versions that were really hard to track down.

Definitely, for Windows we should have a set of Vista, XP and Server
reference platforms. Not sure what we'll want to target for Mac and
Linux yet. My primary goal for the next few weeks is to get some test
machines that Alice can use to test her patches to the new test harness
and graph server up, and generate basic documentation for what went
into the OS (starting with XP SP2 most likely).

In the next few months, my goal is to make the deployment of these
images simpler; ideally we'll install machines from common images. I
think we need to get to the point where test machines are just
different pools of identical machines that we can farm test jobs out
to, as builds come in.

We need to have the documentation on how we are doing this as well, to
make it easier for anyone who wants to contribute by running their own
testing infrastructure. I don't think our Tinderbox docs or testing
code are really adequate right now.

We also need to get the build farm to this point, but I think it will
be easier and less disruptive to expand the test farm first, and then
take what we learn back to the build farm. I'll be publishing a
schedule to my wiki page soon, and setting aside some time to focus on
these issues.

_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Robert Helmer-5
I've put up a first draft of the ref platform for XP:
http://wiki.mozilla.org/ReferencePlatforms/Test/WinXP
(linked from http://wiki.mozilla.org/ReferencePlatforms)

Here is a rough schedule:
http://wiki.mozilla.org/User:Rhelmer:Test_Farm_schedule

We're using the current hardware as a staging area to get everything up
and running. Once it's working well I'll attach a more specific
timetable to the "Q1 2007" goals.

_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Rob Campbell-5
Robert Helmer wrote:
> I've put up a first draft of the ref platform for XP:
> http://wiki.mozilla.org/ReferencePlatforms/Test/WinXP
> (linked from http://wiki.mozilla.org/ReferencePlatforms)

Good stuff, Rob.

I'm looking forward to using one of the Linux reference platforms as a
buildbot master.

That said, in the context of performance, does it make sense to keep a
few different platforms on hand? Ubuntu, Fedora Core, SUSE, etc.
representing what different users might have on-hand? It might be
interesting to see some performance details across the different
platforms, Windows and Mac included.

These might become less of a factor with Gecko1.9+ as we move to Cairo
rendering, but it's still interesting to me.

Also, as performance platforms, does it make sense to include all the
build tools? Shouldn't we be running perf tests on "user-level" machines?

I'm not trying to throw off your schedule, I'm just asking. I think it
couldn't hurt to have a few other standard reference VMs kicking around.
It's not like we'd need to keep them active on the network at all times,
but after a major rev we could cycle through them and compare numbers.

Rob
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Boris Zbarsky
Rob Campbell wrote:
> These might become less of a factor with Gecko1.9+ as we move to Cairo
> rendering, but it's still interesting to me.

Actually, as we rely on the system pango more and more we'll get more and more
perf divergence between different distros, I suspect.

-Boris
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Chris Cooper-2
In reply to this post by Rob Campbell-5
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rob Campbell wrote:
> That said, in the context of performance, does it make sense to keep a
> few different platforms on hand? Ubuntu, Fedora Core, SUSE, etc.
> representing what different users might have on-hand? It might be
> interesting to see some performance details across the different
> platforms, Windows and Mac included.

Would we be targeting Linux because it's easier to instrument and get
performance results from?

Otherwise, from a purely market-share perspective, this sounds like a
maintenance headache that would better be spent on various flavors of
Windows.

cheers,
- --
coop
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFrZP2qdY6vdJz4M8RAtMcAJ9UGHNhaQAimcxM0jDRbDdSSZIivwCfT0i2
GGtncx7XS6uRwkx13kagsUk=
=cBU6
-----END PGP SIGNATURE-----
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Rob Campbell-5
In reply to this post by Boris Zbarsky
Boris Zbarsky wrote:
> Rob Campbell wrote:
>> These might become less of a factor with Gecko1.9+ as we move to Cairo
>> rendering, but it's still interesting to me.
>
> Actually, as we rely on the system pango more and more we'll get more
> and more perf divergence between different distros, I suspect.

Oh, that's depressing...
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Andrew Schultz-2
In reply to this post by Rob Campbell-5
Rob Campbell wrote:
> That said, in the context of performance, does it make sense to keep a
> few different platforms on hand? Ubuntu, Fedora Core, SUSE, etc.
> representing what different users might have on-hand? It might be
> interesting to see some performance details across the different
> platforms, Windows and Mac included.

Off hand, I'd expect that there would be more performance differences
between different versions of gtk/pango/gcc than between distros
released by different vendors with similar package sets.  The vendors do
have their own patches and build config options, but the differences
between gcc 4.1 and gcc 4.0 on any one distro are going to be bigger
than the differences between gcc 4.0 on Fedora and SUSE.

So it might be more interesting (to whatever extent this is done) to
test different versions of the same distro (or different vendors but
purposefully picking releases with different package versions).
Maintaining multiple versions of the same distro is also likely to be
less of a hassle than multiple distros.

--
Andrew Schultz
[hidden email]
http://www.sens.buffalo.edu/~ajs42/
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: Performance Test Reference Platforms

Robert Helmer-5
In reply to this post by Rob Campbell-5
Rob Campbell wrote:

> Robert Helmer wrote:
> > I've put up a first draft of the ref platform for XP:
> > http://wiki.mozilla.org/ReferencePlatforms/Test/WinXP
> > (linked from http://wiki.mozilla.org/ReferencePlatforms)
>
> Good stuff, Rob.
>
> I'm looking forward to using one of the Linux reference platforms as a
> buildbot master.
>
> That said, in the context of performance, does it make sense to keep a
> few different platforms on hand? Ubuntu, Fedora Core, SUSE, etc.
> representing what different users might have on-hand? It might be
> interesting to see some performance details across the different
> platforms, Windows and Mac included.
>
> These might become less of a factor with Gecko1.9+ as we move to Cairo
> rendering, but it's still interesting to me.
>
> Also, as performance platforms, does it make sense to include all the
> build tools? Shouldn't we be running perf tests on "user-level" machines?


Hey Rob,

The Linux test ref platform does not exist yet :) Doing Windows first.


>
> I'm not trying to throw off your schedule, I'm just asking. I think it
> couldn't hurt to have a few other standard reference VMs kicking around.
> It's not like we'd need to keep them active on the network at all times,
> but after a major rev we could cycle through them and compare numbers.


Right. Planning on doing Vista/XP/server2k3 for win32,
ubuntu/fedora/suse for linux, etc. etc.

_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance