Rushing to meet deadlines

classic Classic list List threaded Threaded
58 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Rushing to meet deadlines

Matt Brubeck-3
There was a big traffic jam in mozilla-central last night as people rushed to get features and fixes landed before the m-c to aurora migration.  I strongly suggest that we should not be doing this.  As a guideline, I propose that if you would not request to land a patch on Aurora the day after the migration, you should probably not land it on mozilla-central the day before the migration.  Risky changes should not be going to Aurora with zero Nightly coverage.

I speak from experience: The mobile team rushed in a number of features right before the Firefox 5 migration to Aurora.  We got lucky with many of them, but a few had problems and needed multiple fixes and backouts.  They did not end up shipping with Firefox 5, and rushing them in did not benefit users, testers, or developers.  It just wasted time spent preparing patches for diverging branches, watching two different trees as we double-landed changes, verifying fixes in multiple nightly builds, and revising plans and release notes.  We would have been better off just letting all the questionable last-minute changes wait for the next train.

Holding onto a patch to land it early in the next release cycle might seem like a wasted opportunity, but it will actually give you a lot more freedom to iterate and tweak and fix things on trunk without the restrictions of the Aurora branch and the extra work and process involved.  (On Aurora, you will be forced to back out or take extremely low-risk fixes, rather than the fix that you might think is ideal.)
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Aakash Desai-2
+1.

-- Aakash

----- Original Message -----
From: "Matt Brubeck" <[hidden email]>
To: [hidden email]
Sent: Tuesday, May 24, 2011 9:01:07 AM
Subject: Rushing to meet deadlines

There was a big traffic jam in mozilla-central last night as people rushed to get features and fixes landed before the m-c to aurora migration.  I strongly suggest that we should not be doing this.  As a guideline, I propose that if you would not request to land a patch on Aurora the day after the migration, you should probably not land it on mozilla-central the day before the migration.  Risky changes should not be going to Aurora with zero Nightly coverage.

I speak from experience: The mobile team rushed in a number of features right before the Firefox 5 migration to Aurora.  We got lucky with many of them, but a few had problems and needed multiple fixes and backouts.  They did not end up shipping with Firefox 5, and rushing them in did not benefit users, testers, or developers.  It just wasted time spent preparing patches for diverging branches, watching two different trees as we double-landed changes, verifying fixes in multiple nightly builds, and revising plans and release notes.  We would have been better off just letting all the questionable last-minute changes wait for the next train.

Holding onto a patch to land it early in the next release cycle might seem like a wasted opportunity, but it will actually give you a lot more freedom to iterate and tweak and fix things on trunk without the restrictions of the Aurora branch and the extra work and process involved.  (On Aurora, you will be forced to back out or take extremely low-risk fixes, rather than the fix that you might think is ideal.)
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Curtis Koenig
+
I think there should be a 24 hour moratorium (at the minimum) on
landings before cut over.

On 2011/05/24 9:09 AM, Aakash Desai wrote:

> +1.
>
> -- Aakash
>
> ----- Original Message -----
> From: "Matt Brubeck"<[hidden email]>
> To: [hidden email]
> Sent: Tuesday, May 24, 2011 9:01:07 AM
> Subject: Rushing to meet deadlines
>
> There was a big traffic jam in mozilla-central last night as people rushed to get features and fixes landed before the m-c to aurora migration.  I strongly suggest that we should not be doing this.  As a guideline, I propose that if you would not request to land a patch on Aurora the day after the migration, you should probably not land it on mozilla-central the day before the migration.  Risky changes should not be going to Aurora with zero Nightly coverage.
>
> I speak from experience: The mobile team rushed in a number of features right before the Firefox 5 migration to Aurora.  We got lucky with many of them, but a few had problems and needed multiple fixes and backouts.  They did not end up shipping with Firefox 5, and rushing them in did not benefit users, testers, or developers.  It just wasted time spent preparing patches for diverging branches, watching two different trees as we double-landed changes, verifying fixes in multiple nightly builds, and revising plans and release notes.  We would have been better off just letting all the questionable last-minute changes wait for the next train.
>
> Holding onto a patch to land it early in the next release cycle might seem like a wasted opportunity, but it will actually give you a lot more freedom to iterate and tweak and fix things on trunk without the restrictions of the Aurora branch and the extra work and process involved.  (On Aurora, you will be forced to back out or take extremely low-risk fixes, rather than the fix that you might think is ideal.)
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Kyle Huey-2
All that does is move the problem 24 hours earlier ...

Also, the point of the new release train schedule is that m-c never closes.

- Kyle

On Tue, May 24, 2011 at 9:33 AM, Curtis Koenig <[hidden email]> wrote:

> +
> I think there should be a 24 hour moratorium (at the minimum) on landings
> before cut over.
>
>
> On 2011/05/24 9:09 AM, Aakash Desai wrote:
>
>> +1.
>>
>> -- Aakash
>>
>> ----- Original Message -----
>> From: "Matt Brubeck"<[hidden email]>
>> To: [hidden email]
>> Sent: Tuesday, May 24, 2011 9:01:07 AM
>> Subject: Rushing to meet deadlines
>>
>> There was a big traffic jam in mozilla-central last night as people rushed
>> to get features and fixes landed before the m-c to aurora migration.  I
>> strongly suggest that we should not be doing this.  As a guideline, I
>> propose that if you would not request to land a patch on Aurora the day
>> after the migration, you should probably not land it on mozilla-central the
>> day before the migration.  Risky changes should not be going to Aurora with
>> zero Nightly coverage.
>>
>> I speak from experience: The mobile team rushed in a number of features
>> right before the Firefox 5 migration to Aurora.  We got lucky with many of
>> them, but a few had problems and needed multiple fixes and backouts.  They
>> did not end up shipping with Firefox 5, and rushing them in did not benefit
>> users, testers, or developers.  It just wasted time spent preparing patches
>> for diverging branches, watching two different trees as we double-landed
>> changes, verifying fixes in multiple nightly builds, and revising plans and
>> release notes.  We would have been better off just letting all the
>> questionable last-minute changes wait for the next train.
>>
>> Holding onto a patch to land it early in the next release cycle might seem
>> like a wasted opportunity, but it will actually give you a lot more freedom
>> to iterate and tweak and fix things on trunk without the restrictions of the
>> Aurora branch and the extra work and process involved.  (On Aurora, you will
>> be forced to back out or take extremely low-risk fixes, rather than the fix
>> that you might think is ideal.)
>> _______________________________________________
>> dev-planning mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-planning
>> _______________________________________________
>> dev-planning mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-planning
>>
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Matt Brubeck-3
In reply to this post by Matt Brubeck-3
On Tuesday, May 24, 2011 9:35:40 AM UTC-7, Kyle Huey wrote:
> All that does is move the problem 24 hours earlier ...

No, it leaves m-c open for regression fixes and low-risk changes, while ensuring that higher-risk changes land sooner so that any fixes and backouts can be done on trunk where it requires less effort and churn for everyone.

> Also, the point of the new release train schedule is that m-c never closes.

I'm not proposing that m-c should be closed; I'm proposing that while *some* changes can land at any time, *other* changes are better landing a few days earlier or waiting for the next train.

There's a sliding scale of riskiness, and I think we all have some sense of it.  Really complicated changes (anything likely to result in multiple followup bugs) should probably be timed to land in the first week or two of a new development cycle, even if it means sitting on a finished patch for a while.  Slightly less risky changes could land later in the cycle, but maybe not in the last week, or the last day, or whatever the engineers and QA agree is a reasonable time for testing and stabilization.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Matt Brubeck-3
On Tuesday, May 24, 2011 9:41:11 AM UTC-7, Matt Brubeck wrote:
> On Tuesday, May 24, 2011 9:35:40 AM UTC-7, Kyle Huey wrote:
> > All that does is move the problem 24 hours earlier ...
>
> No, it leaves m-c open for regression fixes and low-risk changes, while ensuring that higher-risk changes land sooner so that any fixes and backouts can be done on trunk where it requires less effort and churn for everyone.

Oops, I didn't notice when I wrote this that Kyle was responding to Curtis's message rather than to my original post.  Yes, I agree with Kyle (and disagree with Curtis) that actually closing m-c would not solve the problem.

[Also, sorry for constantly double-posting - Google Groups posts to both the newsgroup and mailing list by default, and I keep forgetting to fix it.]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Johnathan Nightingale
In reply to this post by Kyle Huey-2
On 2011-05-24, at 12:35 PM, Kyle Huey wrote:

> All that does is move the problem 24 hours earlier ...
>
> Also, the point of the new release train schedule is that m-c never closes.

This. I've heard a couple people talking about this lately, and I think it requires some basic judgement:

- Don't last-minute-land something that you know you'll immediately have to back out or that you affirmatively want nightly bake time for (why are you last-minute landing that anyhow?)
- Otherwise treat pre-merge as any other day, and only land code you believe to be product-quality, but don't accord it particularly special status.

As Kyle says, a key aspect of the rapid release model is that aurora and beta allow us to stabilize, and central is allowed to keep being fast. Your features should have clear kill switches, or be easily backed out, but we should not let an impending aurora merge chill central. The exception being, as I say, some basic judgement around the likelihood you'll have to back your change out immediately or that it's not aurora-ready without some nightly bake time.

J

---
Johnathan Nightingale
Director of Firefox Engineering
[hidden email]



_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

L. David Baron
In reply to this post by Matt Brubeck-3
On Tuesday 2011-05-24 09:41 -0700, Matt Brubeck wrote:
> I'm not proposing that m-c should be closed; I'm proposing that
> while *some* changes can land at any time, *other* changes are
> better landing a few days earlier or waiting for the next train.

I think the idea of the new cycle is that we sort this stuff out on
Aurora, not on mozilla-central, so mozilla-central is always open.
That said, those who land right before the pull will be responsible
for landing fixes in two places, but that's their problem.

I think the standards for Aurora right after the pull should be and
are reasonably lenient in terms of allowing fixes to things that
landed recently (but not allowing new features), but then get
stricter over time.

-David

--
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Matt Brubeck-3
On 05/24/2011 09:46 AM, L. David Baron wrote:
> I think the idea of the new cycle is that we sort this stuff out on
> Aurora, not on mozilla-central, so mozilla-central is always open.
> That said, those who land right before the pull will be responsible
> for landing fixes in two places, but that's their problem.

Yes, absolutely.  Please take my suggestions as "advice for developers
who want to save themselves time in the long run" rather than "new rules
that we need to enforce on everyone."
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

John O'Duinn
In reply to this post by Aakash Desai-2
+1.

Thanks for posting this Matt. Excellent summary of why not to rush in
last minute - its exactly what this rapid release cadence is to help
avoid...

tc
John.
=====
On 5/24/11 9:09 AM, Aakash Desai wrote:

> +1.
>
> -- Aakash
>
> ----- Original Message -----
> From: "Matt Brubeck" <[hidden email]>
> To: [hidden email]
> Sent: Tuesday, May 24, 2011 9:01:07 AM
> Subject: Rushing to meet deadlines
>
> There was a big traffic jam in mozilla-central last night as people rushed to get features and fixes landed before the m-c to aurora migration.  I strongly suggest that we should not be doing this.  As a guideline, I propose that if you would not request to land a patch on Aurora the day after the migration, you should probably not land it on mozilla-central the day before the migration.  Risky changes should not be going to Aurora with zero Nightly coverage.
>
> I speak from experience: The mobile team rushed in a number of features right before the Firefox 5 migration to Aurora.  We got lucky with many of them, but a few had problems and needed multiple fixes and backouts.  They did not end up shipping with Firefox 5, and rushing them in did not benefit users, testers, or developers.  It just wasted time spent preparing patches for diverging branches, watching two different trees as we double-landed changes, verifying fixes in multiple nightly builds, and revising plans and release notes.  We would have been better off just letting all the questionable last-minute changes wait for the next train.
>
> Holding onto a patch to land it early in the next release cycle might seem like a wasted opportunity, but it will actually give you a lot more freedom to iterate and tweak and fix things on trunk without the restrictions of the Aurora branch and the extra work and process involved.  (On Aurora, you will be forced to back out or take extremely low-risk fixes, rather than the fix that you might think is ideal.)
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Shaver
In reply to this post by Curtis Koenig
On Tue, May 24, 2011 at 12:33 PM, Curtis Koenig <[hidden email]> wrote:
> +
> I think there should be a 24 hour moratorium (at the minimum) on landings
> before cut over.

That just kicks the can down (up?) the road.  It doesn't matter when a
hard cut-off is, if there's a hard cut-off.  We can fix (remove)
things on Aurora before we push updates, and AIUI that's the plan of
record.  One such option is just to roll back to an earlier day of m-c
than the cut-off, if the commons got too tragic.

I think that, given the smaller number of things that can be ready on
each release clock tick, we'll see this be much better than the
stampedes of old; we are likely still working through a
non-representative backlog from the long march to FF4.

We need to see what we actually do, with more than a single data
point, before we start revising our policies if we do that at all.  We
had a good (great) plan for this model, but the map is not the
territory.

I think we've learned, usually painfully, that policy works best when
it reinforces behaviour rather than imposes a change.  Social
pressure/alignment and paving cowpaths are the way we have seen change
happen most effectively here.

All that said, kudos for making a specific, actionable suggestion --
even if it's one I disagree with.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Shaver
In reply to this post by John O'Duinn
On Tue, May 24, 2011 at 2:01 PM, John O'Duinn <[hidden email]> wrote:
> Thanks for posting this Matt. Excellent summary of why not to rush in
> last minute - its exactly what this rapid release cadence is to help
> avoid...

I think the opposite, actually.

- you can rush to m-c any day you want
- aurora cut-over should be just another day in the life of m-c, a
mere release mechanic
- we deal with messy rushes that day like any other: piecemeal or
wholesale backout, etc.

Corollary: people should stop thinking about the calendar in the very
general case.  It'll take time to break that (previously adaptive)
habit, but we'll get there.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Justin Lebar-3
In reply to this post by Matt Brubeck-3
>> All that does is move the problem 24 hours earlier ...
> No, it leaves m-c open for regression fixes and low-risk changes

I thought the point of Aurora was to take regression fixes and low-risk changes?

IOW, if this isn't the same as "closing the tree", maybe it's the same as "making m-c into Aurora-lite" for X hours.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Matt Brubeck-3
In reply to this post by Matt Brubeck-3
On Tuesday, May 24, 2011 11:44:28 AM UTC-7, Mike Shaver wrote:
> I think the opposite, actually.
>
> - you can rush to m-c any day you want

On any other day, people pushing to m-c are not "rushing."

> - aurora cut-over should be just another day in the life of m-c, a
> mere release mechanic

Treating it as "just another day" would be a great step forward.  Right now it's the opposite - people are clearly treating the Aurora migration date as a time to make more pushes, with more haste and more urgency.

In the 24 hours before the migration, we had 37 pushes to mozilla-central including 10 backouts or bustage fixes.  In the same period one week earlier, we had a 22 pushes and only 1 backout.

If we were actually treating this time like any other, I wouldn't have seen any need to start this thread.  But what I see are people trying to land things as late in the cycle as possible, which I know from experience leads to extra work and wasted time in the long run.  This "mere release mechanic" affects how we spend our development time, and what (not just when) we deliver to our users.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Shaver
On Tue, May 24, 2011 at 4:17 PM, Matt Brubeck <[hidden email]> wrote:
> On Tuesday, May 24, 2011 11:44:28 AM UTC-7, Mike Shaver wrote:
>> I think the opposite, actually.
>>
>> - you can rush to m-c any day you want
>
> On any other day, people pushing to m-c are not "rushing."

After prolonged tree closures, often after all-hands or summits,
sometimes after major holidays -- we see spikes sometimes for reasons
other than (perceived or real) deadlines.  Or, we certainly have in
the history of the project.

> If we were actually treating this time like any other, I wouldn't have seen any need to start this thread.  But what I see are people trying to land things as late in the cycle as possible, which I know from experience leads to extra work and wasted time in the long run.  This "mere release mechanic" affects how we spend our development time, and what (not just when) we deliver to our users.

Yes, as I said, we're not there yet.  But I think it's premature to
try and solve this with policy (esp since it's not clear what the
actual problematic result was, so far, other than a frustrating day on
trunk).

Some human-to-human direct contact about "why did you feel you needed
to land that day? did you feel like you had to take shortcuts?" is
probably more productive than this thread, but I'm not sufficiently
informed to know who to have those conversations with.  These are
people making decisions with good intentions, not abstract predators
attacking the sanctity of our tree.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Marco Bonardo-2
In reply to this post by Matt Brubeck-3
Il 24/05/2011 22:17, Matt Brubeck ha scritto:
> But what I see are people trying to land things as late in the cycle as possible

While I share part of your thoughts, about the fact it should not work
like this, I don't think anybody is trying or willing to land things as
late as possible.

What happens (among other things) imo is that:
- We are not yet used to the new process. Especially to the 6 weeks
timeframe.
- we are passionate about releasing the best browser we can. We should
do better at thinking it's just matter of 6 weeks before users will get
the next code.
- some review path can take a lot of time, so much that you attach patch
at the beginning of the 6 weeks and you get a review the day before the
merge. While this should not be a problem (you can just take the next
train) it's still frustrating when weeks before you planned that as part
of a certain release.

None of these is a compelling reason or excuse to rush, but I think all
of them are things we are not seeing the right way, yet.

And, as someone who slept 4 hours to try helping keep the tree sane, and
sheriffed most of the rush, I can ensure you nobody gave me the
impression to be willing to destroy the tree, or to land as late as
possible just to do it.

Cheers,
Marco
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Robert O'Callahan-3
If the sheriff metered checkins all day, and we required people to be
merging from green trees or from green tryserver pushes, we wouldn't have
problems, right? Which of those did we not do this time?

Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Marco Bonardo-2
In reply to this post by Marco Bonardo-2
Il 24/05/2011 23:49, Robert O'Callahan ha scritto:
> If the sheriff metered checkins all day, and we required people to be
> merging from green trees or from green tryserver pushes, we wouldn't have
> problems, right? Which of those did we not do this time?

Mostly was fine, but not all patches had full green tryruns, some had
partial runs that ended up missing exactly what failed. Like the change
that caused Windows Talos regressions, had green tryrun, but not Windows
Talos.

A partial issue regarding Try is that in the last 2 days getting results
from it was taking hours, someone got a build after 645 minutes. So
putting more hardware on Try may help.

Another issue was the xpcshell.ini late change in the cycle, it caused
at least 2-3 burnings (and I was among those with a merge from a green
tree, just not enough up-to-date), and lots of patches had to be
modified last-second.

Apart these, I think it didn't work that bad, it was just a long long
queue (that is what we are discussing), with some not-completely-tested
patch in the middle (that we may improve). But it ended up with a tree
in a decent status (last 6 pushes before the merge were green), it just
took some time to get all talos and verify the perf regression was fixed.

Cheers,
Marco
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Axel Hecht
In reply to this post by Matt Brubeck-3
Hi,

ignoring the data about our tree status in the past few, I'd also like
to chime in with an observation I made myself over the past few days:

There are actually a good deal of people that are not rushing their
fixes in. They were close, some of them actually had r+ with comments,
but they didn't land yet because those comments weren't in yet.

I'm saying that because I've had a bunch of bugs in my inbox that I
cared about but couldn't dedicate the brain cycles to that I wanted. And
not one of them got rushed in into the tree over the weekend. Reviewers
were helpful, and I generally didn't observe a "must not miss milestone"
panic.

To me, this feels much better than other code freezes. Independent of
the tree status.

Axel

On 24.05.11 18:01, Matt Brubeck wrote:
> There was a big traffic jam in mozilla-central last night as people rushed to get features and fixes landed before the m-c to aurora migration.  I strongly suggest that we should not be doing this.  As a guideline, I propose that if you would not request to land a patch on Aurora the day after the migration, you should probably not land it on mozilla-central the day before the migration.  Risky changes should not be going to Aurora with zero Nightly coverage.
>
> I speak from experience: The mobile team rushed in a number of features right before the Firefox 5 migration to Aurora.  We got lucky with many of them, but a few had problems and needed multiple fixes and backouts.  They did not end up shipping with Firefox 5, and rushing them in did not benefit users, testers, or developers.  It just wasted time spent preparing patches for diverging branches, watching two different trees as we double-landed changes, verifying fixes in multiple nightly builds, and revising plans and release notes.  We would have been better off just letting all the questionable last-minute changes wait for the next train.
>
> Holding onto a patch to land it early in the next release cycle might seem like a wasted opportunity, but it will actually give you a lot more freedom to iterate and tweak and fix things on trunk without the restrictions of the Aurora branch and the extra work and process involved.  (On Aurora, you will be forced to back out or take extremely low-risk fixes, rather than the fix that you might think is ideal.)

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Justin Wood (Callek)-2
In reply to this post by Marco Bonardo-2
On 5/24/2011 6:08 PM, Marco Bonardo wrote:
> Another issue was the xpcshell.ini late change in the cycle,

This, imo, is the type of change that shouldn't land in last week or two
of a cycle, since it affects so many people with no direct product
benefit, (has a nice benefit, but has the potential to stall, harm
everyone else getting things done).

Also this wasn't announced publically as a "we'll be doing <x> this day"
First I heard announced of this (other than SeaMonkey trees burning) was
like 2 or so days after it landed.

If it was announced, it wasn't public enough, and this from someone who
watches many places, but does have to skim information sometimes.

We need to do better on large-scale changes like this.

--
~Justin Wood (Callek)
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
123