tinderbox tests and clean profiles

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

tinderbox tests and clean profiles

Boris Zbarsky
(reposting due to newsgroup server bug)

So at the moment, not cleaning the profile between tests is apparently
considered to be a bug for tinderboxen (see prometheus Firefox tinderbox
and the places startup time regression thing).

But real users aren't running with a blank profile.  They're running
with a profile with a bunch of history, etc in it.  So it seems to me
that's how we should do our performance tests too, in addition to (or in
place of) the ones we do with clean profiles.

If the worry is data quality and reproducibility, then we should have a
stock nontrivial profile, and between test runs nuke the profile (as we
do now) and replace it with our stock profile (which we do not).

Thoughts?

-Boris
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: tinderbox tests and clean profiles

Justin Wood (Callek)-2
Boris Zbarsky wrote:

> (reposting due to newsgroup server bug)
>
> So at the moment, not cleaning the profile between tests is apparently
> considered to be a bug for tinderboxen (see prometheus Firefox tinderbox
> and the places startup time regression thing).
>
> But real users aren't running with a blank profile.  They're running
> with a profile with a bunch of history, etc in it.  So it seems to me
> that's how we should do our performance tests too, in addition to (or in
> place of) the ones we do with clean profiles.
>
> If the worry is data quality and reproducibility, then we should have a
> stock nontrivial profile, and between test runs nuke the profile (as we
> do now) and replace it with our stock profile (which we do not).
>
> Thoughts?
>
> -Boris

As a passive observer, it sounds like a good idea (stock profile
preferred, possibly a "few varieties" of official stock profiles to
choose from).

And I would choose "in addition to", as brand new users should not be
completely forgotten about ;-). (First impressions are important too).

~Justin Wood (Callek)
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: tinderbox tests and clean profiles

Boris Zbarsky
> And I would choose "in addition to", as brand new users should not be
> completely forgotten about ;-). (First impressions are important too).

The first startup is always slow, since no fastload is happening.

But yes, both blank profile and "profile has stuff in it" tests would be preferable.

-Boris
_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance
Reply | Threaded
Open this post in threaded view
|

Re: tinderbox tests and clean profiles

Vladimir Vukicevic-3
>> And I would choose "in addition to", as brand new users should not be
>> completely forgotten about ;-). (First impressions are important
>> too).
>>
> The first startup is always slow, since no fastload is happening.

Could we add a pre-created fastload file to the default profile?  (Just a
thought for this case.)

> But yes, both blank profile and "profile has stuff in it" tests would
> be preferable.

I agree; we should create some sort of nontrivial profile and use that for
tests.  It would be nice if the tinderbox tools had a way of pulling down
a specific URL to a local location, if it wasn't already there (or if the
remote had changed)... it would be useful to have that as tests start depending
on external files.

   - Vlad


_______________________________________________
dev-performance mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-performance