Updated Performance Regression Policy

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Updated Performance Regression Policy

jmaher
It has come to my attention that the policy we use for handling performance of mozilla-central (and related trees) is outdated:
http://www.mozilla.org/hacking/regression-policy.html

I have spent some time looking at what we do and could reasonably be doing with our current tools.  The result is a newly proposed policy:
https://etherpad.mozilla.org/perf-policy

Please reply to this thread if there are questions, concerns or praises about this new proposed policy.  Assuming all changes are voiced and added in the policy before November 15th, I will work on making this the official policy the week of November 18th.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Updated Performance Regression Policy

Gavin Sharp-3
Can you give a brief summary of the changes?

Gavin

On Mon, Nov 4, 2013 at 10:50 AM, Joel Maher <[hidden email]> wrote:

> It has come to my attention that the policy we use for handling performance of mozilla-central (and related trees) is outdated:
> http://www.mozilla.org/hacking/regression-policy.html
>
> I have spent some time looking at what we do and could reasonably be doing with our current tools.  The result is a newly proposed policy:
> https://etherpad.mozilla.org/perf-policy
>
> Please reply to this thread if there are questions, concerns or praises about this new proposed policy.  Assuming all changes are voiced and added in the policy before November 15th, I will work on making this the official policy the week of November 18th.
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Updated Performance Regression Policy

Stephen Pohl
In reply to this post by jmaher
Joel Maher wrote:
> It has come to my attention that the policy we use for handling performance of mozilla-central (and related trees) is outdated:
> http://www.mozilla.org/hacking/regression-policy.html
>
> I have spent some time looking at what we do and could reasonably be doing with our current tools.  The result is a newly proposed policy:
> https://etherpad.mozilla.org/perf-policy
>
The new policy mentions that " There will be a few instances where there
is a performance regression and we approve this. These types of
regressions are not common and will require documentation as to why we
are regressing performance." Would it be possible to be specific about
who's to make the call? The original document says to  "start a
discussion with [hidden email]". Is this still true? What kind of
documentation should be written to explain a performance regression? I'm
asking because this could come in very handy for bug 860493, which turns
on history swipe animations on OSX by default (bug 678392).

Thanks!

-Stephen
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Updated Performance Regression Policy

Nicholas Nethercote
In reply to this post by jmaher
On Tue, Nov 5, 2013 at 5:50 AM, Joel Maher <[hidden email]> wrote:
>
> I have spent some time looking at what we do and could reasonably be doing with our current tools.  The result is a newly proposed policy:
> https://etherpad.mozilla.org/perf-policy

"If a regression is posted to dev.tree-management  and has no action
taken for 2 business days, Mozilla will start backing  code out to
reduce the regression."

What happens in the (common) case where there's a regression and there
are a dozen patches with half a dozen authors in the regression
window?  I received such an email just this morning, and I concluded
that the regression -- assuming it was real and not just noise --
almost certainly wasn't my fault.  Though it's possible I was wrong...

Nick
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Updated Performance Regression Policy

jmaher
In reply to this post by jmaher
There is a scenario with our current tools which reports a regression no branch X, then when that merges into mozilla-central or mozilla-inbound, we see that regression again.  I believe this is the case that you are seeing- a merge causes a regression alert.

Let me modify the policy to account for merges, we should have identified the regression earlier on and have a bug on file or a discussion surrounding that.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Updated Performance Regression Policy

jmaher
In reply to this post by jmaher

> The new policy mentions that " There will be a few instances where there
>
> is a performance regression and we approve this. These types of
>
> regressions are not common and will require documentation as to why we
>
> are regressing performance." Would it be possible to be specific about
>
> who's to make the call? The original document says to  "start a
>
> discussion with [hidden email]". Is this still true? What kind of
>
> documentation should be written to explain a performance regression? I'm
>
> asking because this could come in very handy for bug 860493, which turns
>
> on history swipe animations on OSX by default (bug 678392).
>
>
>
> Thanks!
>
>
>
> -Stephen

Thanks for calling this out and being proactive about changes you are making.  Let me find a channel for notification of upcoming perf issues and a place to document this.  Most likely this will be a distribution list to email and a wiki that journals changes and dates landed.  A possible whiteboard tag to add to bugs as well.

-Joel
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Updated Performance Regression Policy

Andrew  McCreight
In reply to this post by Nicholas Nethercote


----- Original Message -----

> On Tue, Nov 5, 2013 at 5:50 AM, Joel Maher <[hidden email]> wrote:
> >
> > I have spent some time looking at what we do and could reasonably be doing
> > with our current tools.  The result is a newly proposed policy:
> > https://etherpad.mozilla.org/perf-policy
>
> "If a regression is posted to dev.tree-management  and has no action
> taken for 2 business days, Mozilla will start backing  code out to
> reduce the regression."
>
> What happens in the (common) case where there's a regression and there
> are a dozen patches with half a dozen authors in the regression
> window?  I received such an email just this morning, and I concluded
> that the regression -- assuming it was real and not just noise --
> almost certainly wasn't my fault.  Though it's possible I was wrong...

Also, the bit about "Mozilla will start backing out" is weird.  Who is the "Mozilla" actually responsible for doing this?  The patch writer?  The sheriffs?  Module owners?  Somebody else?

Andrew

>
> Nick
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Updated Performance Regression Policy

Lawrence Mandel-2
In reply to this post by jmaher
----- Original Message -----

> It has come to my attention that the policy we use for handling performance
> of mozilla-central (and related trees) is outdated:
> http://www.mozilla.org/hacking/regression-policy.html
>
> I have spent some time looking at what we do and could reasonably be doing
> with our current tools.  The result is a newly proposed policy:
> https://etherpad.mozilla.org/perf-policy
>
> Please reply to this thread if there are questions, concerns or praises about
> this new proposed policy.  Assuming all changes are voiced and added in the
> policy before November 15th, I will work on making this the official policy
> the week of November 18th.

I added a comments section to the etherpad in order to track the comments in one place. I also added a number of comments to the pad.

Lawrence
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning