Shipping Type Inference and Other "Irreversible" Changes

classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Shipping Type Inference and Other "Irreversible" Changes

Dave Mandelin-2
* Background: TI

Brian Hackett, of the JavaScript team, has been doing great applied
research on using a hybrid static/dynamic analysis to optimize compiled
JS. The project is called TI or JM+TI, for "type inference". Brian now
has it to the point where it runs in the browser and is green on
Tinderbox. I just measured this morning and on Windows it improves our
V8 score from 4980 to 5440 (1.1x) and our Kraken score from 4880 to 3365
(1.45x). Some V8 subscores are particularly big; e.g., crypto goes from
7110 to 15000. TI also gets top scores of any engine on some workloads.
So it's a big boost for our performance, pushes forward JS optimization
technology, and will be even more effective once we have IonMonkey, so
it's good to start using it and testing it now. There are a few
regressions still to be worked out, but Brian is close to being ready to
land.

* Background: Irreversible Changes

The difficulty of landing TI in the train model is that the landing is
"irreversible". By that, I mean that neither pref'ing it off nor backing
it out is practical.
  - Pref'ing off doesn't work because the project changes many things
throughout the JS engine, so some of those changes can not be disabled
at run time. (The TI optimizations themselves can be pref'd off, of
course.)
  - Backing out is also hard because it touches so many areas of the JS
engine. This means a standard backout would generate a big merge that
might conflict with changes that landed on top of it. Doing this a few
weeks before an Aurora or Beta merge sounds risky. Alternatively, we
could back out the following changes, then back out TI, then rebase and
reland those changes. But that's a ton of work, and also risky because
of the rebasing.

We've made other necessary irreversible changes, including fatvals and
compartments last year. I think Azure might be another case. So it seems
very likely to happen again, which means we need a general solution to
the problem. The proposal below is intended to set up a general scheme
we can use for any future irreversible change.

* Proposal for Shipping Irreversible Changes

In the train model, changes can only land if they are reversible, so my
proposed solution to irreversible changes is not to land them, but
rather to hold them in separate repos and ship from there. So instead of
"landing", we "switch users to an alternate repo". And instead of
"backing out", we "switch users back to the canonical repo". Once the
product ships as a final release, we land to the canonical repo. In more
detail:

In the standard train model, we have these repos:

 - mozilla-central
 - mozilla-aurora
 - mozilla-beta
 - mozilla-release

Changes propagate through these repos on a schedule:

 - at Aurora cut time, m-c merges to mozilla-aurora
 - at Beta cut time, m-a merges to mozilla-beta
 - at Release time, m-b merges to mozilla-release

And each channel is linked to exactly one repo:

 - Nightly builds from mozilla-central
 - Aurora builds from mozilla-aurora
 - Beta ships builds mozilla-beta
 - Release builds from mozilla-release

With irreversible changes, we add an alternate repo for each standard
one except release, so we have

 - mozilla-central
 - [project repo; TI in this case]: merges from mozilla-central
automatically every day
 - mozilla-aurora
 - mozilla-aurora-alt: merges from mozilla-aurora automatically every day
 - mozilla-beta
 - mozilla-beta-alt: merges from mozilla-beta automatically every day
 - mozilla-release

Most fixes land to the same repo they always did. Fixes to TI land to
the alt repos.

Because of the automatic daily merges, mozilla-x-alt is always a copy of
mozilla-x with the addition of the project changes; changes landed to
the normal repo quickly propagate to the "alt" repo.

Propagation is similar to before:

 - At Aurora cut time, m-c merges to mozilla-aurora and TI merges to
mozilla-aurora-alt
 - At Beta cut time, m-a merges to m-b and mozilla-aurora-alt merges to
mozilla-beta-alt
 - At Release cut time, mozilla beta-alt merges to mozilla-release
 - One extra thing is that after release, the alt branches must become
the normal branches. We can do this by merging TI to m-c and each alt
branch to its corresponding normal branch.

The other big change is that we use channel switching to activate the
alt repos. The schedule would be like this:

 - When TI is ready (should be very early (first week) in Nightly
cycle), point Nightly to TI repo.
 - If TI is too unstable, point Nightly back to m-c. Everything is fully
back to normal!
 - If TI restabilizes quickly (say one killer bug got fixes), point
Nightly back to TI.
 - At Aurora cut, point Aurora to mozilla-aurora-alt
 - At Beta cut, point Beta to mozilla-beta-alt

The main work items for the proposal are:

 - Set up the 2 alt repos
 - Set up scripts to automatically merge normal to alt repos
 - Redirect Nightly/Aurora/Beta users according to the schedule above

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Steve Wendt
On 8/9/2011 3:48 PM, David Mandelin wrote:

> With irreversible changes, we add an alternate repo for each standard
> one except release
>
>   - When TI is ready (should be very early (first week) in Nightly
> cycle), point Nightly to TI repo.
>   - If TI is too unstable, point Nightly back to m-c. Everything is fully
> back to normal!
>   - If TI restabilizes quickly (say one killer bug got fixes), point
> Nightly back to TI.
>   - At Aurora cut, point Aurora to mozilla-aurora-alt
>   - At Beta cut, point Beta to mozilla-beta-alt

So you only get one "irreversible change" per 18 week cycle?  Perhaps
there should not be a beta-alt at all; either the feature makes it out
of aurora-alt, or it waits for the next cycle?
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Jeff Muizelaar-4
In reply to this post by Dave Mandelin-2

On 2011-08-09, at 6:48 PM, David Mandelin wrote:

> We've made other necessary irreversible changes, including fatvals and
> compartments last year. I think Azure might be another case.

Just for the record Azure is currently still pref-able and I think the pain of having to handle an irreversible change like this is enough to make us try really hard to avoid a similar situation.

Just the same it does seem like a problem we're going to have deal with and your proposal, while painful, doesn't sound totally insane to me.

-Jeff

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Christian Legnitto
In reply to this post by Steve Wendt

On Aug 9, 2011, at 4:07 PM, Steve Wendt wrote:

> On 8/9/2011 3:48 PM, David Mandelin wrote:
>
>> With irreversible changes, we add an alternate repo for each standard
>> one except release
>>
>>  - When TI is ready (should be very early (first week) in Nightly
>> cycle), point Nightly to TI repo.
>>  - If TI is too unstable, point Nightly back to m-c. Everything is fully
>> back to normal!
>>  - If TI restabilizes quickly (say one killer bug got fixes), point
>> Nightly back to TI.
>>  - At Aurora cut, point Aurora to mozilla-aurora-alt
>>  - At Beta cut, point Beta to mozilla-beta-alt
>
> So you only get one "irreversible change" per 18 week cycle?  Perhaps there should not be a beta-alt at all; either the feature makes it out of aurora-alt, or it waits for the next cycle?

There are some problems we only find when have millions of users so we still need the go/no-go option there as well.

Thanks,
Christian

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Jonas Sicking-2
In reply to this post by Dave Mandelin-2
On Tue, Aug 9, 2011 at 3:48 PM, David Mandelin <[hidden email]> wrote:
> The main work items for the proposal are:
>
>  - Set up the 2 alt repos
>  - Set up scripts to automatically merge normal to alt repos
>  - Redirect Nightly/Aurora/Beta users according to the schedule above

Another approach to all this is to simply go with the "normal"
approach, keeping in mind that we have the no-go option available for
each release.

In other words, if this lands on mozilla-central right after a merge,
that leaves 6 weeks of nightlies and 6 weeks of aurora before we reach
beta audience. That's almost 3 months of time to fix any bugs bad
enough to prevent a beta. After that there's another 6 weeks of beta
before release.

If we any time after that realize it's not of high enough quality, we
can always make a no-go decision for that train. By just making one
no-go decision we get the total time between landing and shipping
almost 6 months.

Granted, if we after 6 months realize that there's simply too many
bugs, or too wrong architecture, we are in a pretty bad place since
the patch at that point lives in all branches and so backing it out is
going to introduce a lot of risk even on aurora and beta channels. But
it seems pretty unlikely to me that that would be the case.

/ Jonas
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Scott Johnson-22
In reply to this post by Christian Legnitto

On 08/10/2011 11:25 AM, thus spoke Christian Legnitto:

> On Aug 9, 2011, at 4:07 PM, Steve Wendt wrote:
>
>> On 8/9/2011 3:48 PM, David Mandelin wrote:
>>
>>> With irreversible changes, we add an alternate repo for each standard
>>> one except release
>>>
>>>   - When TI is ready (should be very early (first week) in Nightly
>>> cycle), point Nightly to TI repo.
>>>   - If TI is too unstable, point Nightly back to m-c. Everything is fully
>>> back to normal!
>>>   - If TI restabilizes quickly (say one killer bug got fixes), point
>>> Nightly back to TI.
>>>   - At Aurora cut, point Aurora to mozilla-aurora-alt
>>>   - At Beta cut, point Beta to mozilla-beta-alt
>> So you only get one "irreversible change" per 18 week cycle?  Perhaps there should not be a beta-alt at all; either the feature makes it out of aurora-alt, or it waits for the next cycle?
> There are some problems we only find when have millions of users so we still need the go/no-go option there as well.
Look on the bright side, though... we need names for these new channels.
Might I suggest 'mozilla-hyperion'? :)

J/K. It sounds like a good plan, but I do agree that the changes should
either make it out of aurora-alt, and be placed into beta, or not make
it into beta at all (i.e. my opinion would be to scrap the
mozilla-beta-alt). That said, I think it's going to be a large amount of
work, and it seems to me to be much more work than utilizing the 'no go'
alternative.

~Scott

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Nicholas Nethercote
In reply to this post by Jonas Sicking-2
On Tue, Aug 9, 2011 at 4:48 PM, Jonas Sicking <[hidden email]> wrote:
>
> Granted, if we after 6 months realize that there's simply too many
> bugs, or too wrong architecture, we are in a pretty bad place since
> the patch at that point lives in all branches and so backing it out is
> going to introduce a lot of risk even on aurora and beta channels. But
> it seems pretty unlikely to me that that would be the case.

Let's not bet the farm on that.

Nick
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Robert O'Callahan-3
In reply to this post by Dave Mandelin-2
On Wed, Aug 10, 2011 at 10:48 AM, David Mandelin <[hidden email]>wrote:

> - Pref'ing off doesn't work because the project changes many things
> throughout the JS engine, so some of those changes can not be disabled
> at run time. (The TI optimizations themselves can be pref'd off, of
> course.)
>

How hard have you looked at breaking up those changes into independent
pieces for landing?

Rob
--
"If we claim to be without sin, we deceive ourselves and the truth is not in
us. If we confess our sins, he is faithful and just and will forgive us our
sins and purify us from all unrighteousness. If we claim we have not sinned,
we make him out to be a liar and his word is not in us." [1 John 1:8-10]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Matt Brubeck-3
In reply to this post by Dave Mandelin-2
On 08/09/2011 03:48 PM, David Mandelin wrote:
>    - Backing out is also hard because it touches so many areas of the JS
> engine. This means a standard backout would generate a big merge that
> might conflict with changes that landed on top of it. [...]
>
>   - mozilla-aurora-alt: merges from mozilla-aurora automatically every day
>   - mozilla-beta-alt: merges from mozilla-beta automatically every day

Won't the merges, over the course of the release cycle, involve about as
many conflicts as an eventual backout  would?  Essentially it is
spreading the work of the backout over the 18-week cycle.

This is still useful, because it spreads out the work and also lets us
test continuously and find problems sooner.  But it seems wrong to
assume these merges can be "automatic" when we've also said that the
delta between repos is likely to cause conflicts with subsequent changes.

Sometimes the automatic merge will fail, and other times it will succeed
but produce broken code.  I'm sure the JS team understands this since
they have plenty of experience with long-lived branches.  But for the
sake of this discussion it should be explicit that someone will need to
own the manual work of dealing with occasional merge conflicts.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Karl Tomlinson-4
Matt Brubeck writes:

> Won't the merges, over the course of the release cycle, involve
> about as many conflicts as an eventual backout  would?
> Essentially it is spreading the work of the backout over the
> 18-week cycle.

Yes.
And ensuring time is allocated for / spent on the work,
even if it turns out to be unneeded.

> This is still useful, because [...] and also
> lets us test continuously and find problems sooner.

Let's us test continuously on tinderboxen, which is more than the
standard train model provides for back-out paths.

IIUC it doesn't give us user testing of the back-out path.
User-testing issues take the longest to show up, and so we may not
have the confidence to go with the back-out path.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Axel Hecht
In reply to this post by Dave Mandelin-2
On 10.08.11 00:48, David Mandelin wrote:

> * Background: TI
>
> Brian Hackett, of the JavaScript team, has been doing great applied
> research on using a hybrid static/dynamic analysis to optimize compiled
> JS. The project is called TI or JM+TI, for "type inference". Brian now
> has it to the point where it runs in the browser and is green on
> Tinderbox. I just measured this morning and on Windows it improves our
> V8 score from 4980 to 5440 (1.1x) and our Kraken score from 4880 to 3365
> (1.45x). Some V8 subscores are particularly big; e.g., crypto goes from
> 7110 to 15000. TI also gets top scores of any engine on some workloads.
> So it's a big boost for our performance, pushes forward JS optimization
> technology, and will be even more effective once we have IonMonkey, so
> it's good to start using it and testing it now. There are a few
> regressions still to be worked out, but Brian is close to being ready to
> land.
>
> * Background: Irreversible Changes
>
> The difficulty of landing TI in the train model is that the landing is
> "irreversible". By that, I mean that neither pref'ing it off nor backing
> it out is practical.
>    - Pref'ing off doesn't work because the project changes many things
> throughout the JS engine, so some of those changes can not be disabled
> at run time. (The TI optimizations themselves can be pref'd off, of
> course.)
>    - Backing out is also hard because it touches so many areas of the JS
> engine. This means a standard backout would generate a big merge that
> might conflict with changes that landed on top of it. Doing this a few
> weeks before an Aurora or Beta merge sounds risky. Alternatively, we
> could back out the following changes, then back out TI, then rebase and
> reland those changes. But that's a ton of work, and also risky because
> of the rebasing.
>
> We've made other necessary irreversible changes, including fatvals and
> compartments last year. I think Azure might be another case. So it seems
> very likely to happen again, which means we need a general solution to
> the problem. The proposal below is intended to set up a general scheme
> we can use for any future irreversible change.
>
> * Proposal for Shipping Irreversible Changes
>
> In the train model, changes can only land if they are reversible, so my
> proposed solution to irreversible changes is not to land them, but
> rather to hold them in separate repos and ship from there. So instead of
> "landing", we "switch users to an alternate repo". And instead of
> "backing out", we "switch users back to the canonical repo". Once the
> product ships as a final release, we land to the canonical repo. In more
> detail:
>
> In the standard train model, we have these repos:
>
>   - mozilla-central
>   - mozilla-aurora
>   - mozilla-beta
>   - mozilla-release
>
> Changes propagate through these repos on a schedule:
>
>   - at Aurora cut time, m-c merges to mozilla-aurora
>   - at Beta cut time, m-a merges to mozilla-beta
>   - at Release time, m-b merges to mozilla-release
>
> And each channel is linked to exactly one repo:
>
>   - Nightly builds from mozilla-central
>   - Aurora builds from mozilla-aurora
>   - Beta ships builds mozilla-beta
>   - Release builds from mozilla-release
>
> With irreversible changes, we add an alternate repo for each standard
> one except release, so we have
>
>   - mozilla-central
>   - [project repo; TI in this case]: merges from mozilla-central
> automatically every day
>   - mozilla-aurora
>   - mozilla-aurora-alt: merges from mozilla-aurora automatically every day
>   - mozilla-beta
>   - mozilla-beta-alt: merges from mozilla-beta automatically every day
>   - mozilla-release
>
> Most fixes land to the same repo they always did. Fixes to TI land to
> the alt repos.
>
> Because of the automatic daily merges, mozilla-x-alt is always a copy of
> mozilla-x with the addition of the project changes; changes landed to
> the normal repo quickly propagate to the "alt" repo.
>
> Propagation is similar to before:
>
>   - At Aurora cut time, m-c merges to mozilla-aurora and TI merges to
> mozilla-aurora-alt
>   - At Beta cut time, m-a merges to m-b and mozilla-aurora-alt merges to
> mozilla-beta-alt
>   - At Release cut time, mozilla beta-alt merges to mozilla-release
>   - One extra thing is that after release, the alt branches must become
> the normal branches. We can do this by merging TI to m-c and each alt
> branch to its corresponding normal branch.
>
> The other big change is that we use channel switching to activate the
> alt repos. The schedule would be like this:
>
>   - When TI is ready (should be very early (first week) in Nightly
> cycle), point Nightly to TI repo.
>   - If TI is too unstable, point Nightly back to m-c. Everything is fully
> back to normal!
>   - If TI restabilizes quickly (say one killer bug got fixes), point
> Nightly back to TI.
>   - At Aurora cut, point Aurora to mozilla-aurora-alt
>   - At Beta cut, point Beta to mozilla-beta-alt
>
> The main work items for the proposal are:
>
>   - Set up the 2 alt repos
>   - Set up scripts to automatically merge normal to alt repos
>   - Redirect Nightly/Aurora/Beta users according to the schedule above
>

Technical detail, we would need the feature repos and the main repos to
be the same l10n-wise. Not much of an issue in this case, but in the
general case, we should keep this on the radar.

Axel
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Ben Hearsum-2
In reply to this post by Dave Mandelin-2
On 08/09/11 06:48 PM, David Mandelin wrote:

> With irreversible changes, we add an alternate repo for each standard
> one except release, so we have
>
>   - mozilla-central
>   - [project repo; TI in this case]: merges from mozilla-central
> automatically every day
>   - mozilla-aurora
>   - mozilla-aurora-alt: merges from mozilla-aurora automatically every day
>   - mozilla-beta
>   - mozilla-beta-alt: merges from mozilla-beta automatically every day
>   - mozilla-release

So, under this plan, we would have all of our Aurora and Beta users on
the -alt variants?

If that's the case, I don't see the benefit to having the repositories.

If we end up shipping the -alt code, the original aurora and beta
repositories are not useful as anything except a stepping stone to the
"real" code we're shipping.

If we lose confidence in the -alt code, we can't ship the original code,
because we've had no Aurora or Beta users testing it(1). On this
assumption, we may as well just have the irreversible changes in plain
-aurora and -beta repositories because we can't ship their contents with
your plan.

- Ben

(1) Given that changes like this are large and invasive, I don't think
we can say that testing done from -alt repositories can carry over.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Ted Mielczarek-2
In reply to this post by Nicholas Nethercote
On Tue, Aug 9, 2011 at 8:36 PM, Nicholas Nethercote
<[hidden email]> wrote:
> On Tue, Aug 9, 2011 at 4:48 PM, Jonas Sicking <[hidden email]> wrote:
>>
>> Granted, if we after 6 months realize that there's simply too many
>> bugs, or too wrong architecture, we are in a pretty bad place since
>> the patch at that point lives in all branches and so backing it out is
>> going to introduce a lot of risk even on aurora and beta channels. But
>> it seems pretty unlikely to me that that would be the case.
>
> Let's not bet the farm on that.

Yeah. I think as painful as it might be, we need to think about how to
land large changes like this in a way that we can also disable or back
them out if need be. Look at the Firefox 4 cycle. We were stalled on
fallout from compartment changes that were necessary for JaegerMonkey
work, and that stretched our beta cycle out indefinitely. What's to
say that we won't find a series of hard-to-fix regressions from the TI
work that cause us to be unable to ship, and delay hundreds or
thousands of other fixes from reaching our users?

In short, if we're going to live on the faster release cycle, we have
to really believe it and not break the rules for anything, otherwise
the whole thing will blow up on us.

-Ted
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Brian William Hackett
In reply to this post by Robert O'Callahan-3

----- Original Message -----
> How hard have you looked at breaking up those changes into independent
> pieces for landing?
>
> Rob

The project can be broken up into a few (still large) pieces, but the problem is that the actual optimizations depend on all the rest being in place, and that remainder is mainly bookkeeping which gives no benefit and adds up to significant overhead on JS heavy pages (including 3-5% on the V8 and SunSpider benchmarks).

Brian
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Brian William Hackett
In reply to this post by Matt Brubeck-3

----- Original Message -----
> Sometimes the automatic merge will fail, and other times it will
> succeed
> but produce broken code. I'm sure the JS team understands this since
> they have plenty of experience with long-lived branches. But for the
> sake of this discussion it should be explicit that someone will need
> to
> own the manual work of dealing with occasional merge conflicts.

I would be managing conflicts when merging into the alternate branch, I've been doing this already since the project's inception.

Brian
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Robert Kaiser
In reply to this post by Dave Mandelin-2
David Mandelin schrieb:
> With irreversible changes, we add an alternate repo for each standard
> one except release

We have no reasonable plan for analyzing crashes/stability on an
alternate branch, and currently no good tooling to count apart different
aurora or beta repos in terms of significant crashes. It's hard enough
already to get tooling in place to be able to watch different beta
releases, and we already have too low user volumes on Aurora and
probably also beta to get really good crash data.
Further splitting the audience would only lead to needing a lot more
manpower to create tooling and watch the differences, as well as less
reliable numbers due to low audience numbers on those different builds.

 From that point of view, I don't really like that proposal and would
like it better to just land that work at the beginning of a Nightly
cycle so we can work out as many problems as possible in the 5-6 weeks
until this hits aurora.

Robert Kaiser


--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community should think about. And most of the
time, I even appreciate irony and fun! :)
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Johnathan Nightingale
In reply to this post by Ted Mielczarek-2
On 2011-08-10, at 9:12 AM, Ted Mielczarek wrote:
> Yeah. I think as painful as it might be, we need to think about how to
> land large changes like this in a way that we can also disable or back
> them out if need be. Look at the Firefox 4 cycle. We were stalled on
> fallout from compartment changes that were necessary for JaegerMonkey
> work, and that stretched our beta cycle out indefinitely. What's to
> say that we won't find a series of hard-to-fix regressions from the TI
> work that cause us to be unable to ship, and delay hundreds or
> thousands of other fixes from reaching our users?

I'm terrified that your first paragraph here will spin this thread off into a debate about the sources of FF4 schedule slippage. If you, gentle reader, are tempted to point out that there were other causes for FF4's schedule being what it was, my fervent hope is that you will let it sit. The reasons for FF4's elongated delivery schedule are manifest and complex. Monolithic development has schedule slip in its core. It's true that compartments landed late, and needed follow up work, but so did a couple of other big pieces.

Please, god, let's avoid that rathole that I'm sure Ted didn't intend.

> In short, if we're going to live on the faster release cycle, we have
> to really believe it and not break the rules for anything, otherwise
> the whole thing will blow up on us.

Modulo a tiny bit of human judgement in that "not break the rules for anything" clause, I agree. We took a post-aurora merge for JS last release because of a communication failure that resulting in them missing the cut off. That won't be a habit, but we broke the rules in a minor way for the right reasons.

Still, though - I agree. Rapid release means you have more chances to get it right, that we never need to ship code we're not happy with (or hold back a release) because otherwise we'll have to wait a year. If TI is so invasive that a killswitch is impossible, then backing out is the killswitch we have left. If other code builds on top of that in ways that are harmed by a backout, that code may *also* have to wait 6 more weeks, or there might be some hand-merging needed to strain the other bugs through the TI mesh.

Either way, though, I think that for code *on a release train*, it either works well enough to ship it, or it gets killed by whatever means we have available.

There is still an important question here around how to get widespread testing for code not yet on a release train. Ideas like portioning off 20% of our nightly users to hammer on a TI-Nightly somewhere have been floated in the past, and might accomplish much of what Dave and Brian want here (finding TI bugs early) without needing to fight against our tree rules on release trains. If I'm right in thinking that might address part of the concern here, then we should revisit those conversations that were scoped out of the initial rapid release discussion, ideally in a separate thread.

I don't believe we should accept irreversible changes. I do believe that some changes are very expensive to reverse, though, and we should find ways to assess and reduce the risk of having to pay that cost.

J

---
Johnathan Nightingale
Director of Firefox Engineering
[hidden email]



_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Ted Mielczarek-2
On Wed, Aug 10, 2011 at 11:06 AM, Johnathan Nightingale
<[hidden email]> wrote:
> I'm terrified that your first paragraph here will spin this thread off into a debate about the sources of FF4 schedule slippage. If you, gentle reader, are tempted to point out that there were other causes for FF4's schedule being what it was, my fervent hope is that you will let it sit. The reasons for FF4's elongated delivery schedule are manifest and complex. Monolithic development has schedule slip in its core. It's true that compartments landed late, and needed follow up work, but so did a couple of other big pieces.

Indeed, it was a reductionist view of things, I just remember the
compartment fallout very vividly. Don't take it as gospel. I believe
the point stands. Holding releases hostage for any one feature, no
matter how valuable, will just lead us back to the pain of our old
release process.

-Ted
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Benjamin Smedberg
In reply to this post by Dave Mandelin-2
On 8/9/2011 6:48 PM, David Mandelin wrote:

>
> With irreversible changes, we add an alternate repo for each standard
> one except release, so we have
>
>   - mozilla-central
>   - [project repo; TI in this case]: merges from mozilla-central
> automatically every day
>   - mozilla-aurora
>   - mozilla-aurora-alt: merges from mozilla-aurora automatically every day
>   - mozilla-beta
>   - mozilla-beta-alt: merges from mozilla-beta automatically every day
>   - mozilla-release
>
> Most fixes land to the same repo they always did. Fixes to TI land to
> the alt repos.
>
> Because of the automatic daily merges, mozilla-x-alt is always a copy of
> mozilla-x with the addition of the project changes; changes landed to
> the normal repo quickly propagate to the "alt" repo.
Having read lots of this discussion, I think there is probably an
alternate way to achieve these goals. The basic principle of your
proposal is to maintain a codebase without the TI code. I think we can
do this today:

When we believe it is ready for nightly users, merge TI into nightly.
Maintain a separate project branch "nightly-without-ti" where TI is
immediately "backed out" again.

The TI team will be responsible for maintaining the nightly-without-ti
branch. This probably means that other large JS landings which might
conflict should be avoided during this time period.

When we get to Aurora, release drivers and the TI team decide whether we
believe the feature is ready for the aurora train. If it is, we keep an
aurora-without-ti branch active so that we can flip TI off if blocking
issues are found. Similar for the beta branch.

This avoids a fair bit of release engineering headache switching users
between release channels and dealing with mechanics issues such as
download links, localization repos, etc, while keeping options open for
disabling TI throughout the release cycle.

The downside is that the TI team will need to deal with merge conflicts
on the *-without-ti branches, but it seems to me that this is roughly
the same amount of work as in your proposal, where you merge into the
-alt branch.

--BDS

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Shipping Type Inference and Other "Irreversible" Changes

Asa Dotzler-2
In reply to this post by Dave Mandelin-2


Benjamin Smedberg wrote:

> The downside is that the TI team will need to deal with merge conflicts
> on the *-without-ti branches, but it seems to me that this is roughly
> the same amount of work as in your proposal, where you merge into the
> -alt branch.
>
> --BDS

The other downside is we won't have our 60K nightly testers making sure
that the not-TI branch is actually working as well as the TI branch
should there be any differences that manifest as a result of merges or
TI-dependent differences.

- A
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
12