Rushing to meet deadlines

classic Classic list List threaded Threaded
58 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Shaver
On Tue, May 24, 2011 at 3:41 PM, Justin Wood (Callek) <[hidden email]> wrote:
> On 5/24/2011 6:08 PM, Marco Bonardo wrote:
>>
>> Another issue was the xpcshell.ini late change in the cycle,
>
> This, imo, is the type of change that shouldn't land in last week or two of
> a cycle,

Disagree.  There are no cycles, they are an illusion perpetrated by
the release team.  If things land a few days later because of a
disruptive change, they might miss a cut-over, or a weekend of
nightly-bake or whatever.

Maybe we should slightly randomize the dates of cut-over.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Asa Dotzler-2
In reply to this post by Axel Hecht
On 5/24/2011 3:40 PM, Axel Hecht wrote:

> Hi,
>
> ignoring the data about our tree status in the past few, I'd also like
> to chime in with an observation I made myself over the past few days:
>
> There are actually a good deal of people that are not rushing their
> fixes in. They were close, some of them actually had r+ with comments,
> but they didn't land yet because those comments weren't in yet.
>
> I'm saying that because I've had a bunch of bugs in my inbox that I
> cared about but couldn't dedicate the brain cycles to that I wanted. And
> not one of them got rushed in into the tree over the weekend. Reviewers
> were helpful, and I generally didn't observe a "must not miss milestone"
> panic.
>
> To me, this feels much better than other code freezes. Independent of
> the tree status.
>
> Axel

This is something I tweeted earlier today that I wanted to second here.
A lot more people were not rushing in at the last minute than were.
Maybe it was a bit of a mess the last 18 hours of the cycle but it was
only a mess for a handful of people that for what ever reason were up
against the deadline.

That seems like a decent feedback loop to have. If you're really late,
expect to crash up against the handful of other people that are really
late. If you get in early or you decide to wait a day or two and land
for the next cycle, you avoid that pain.

Or put another way, I don't think we should be optimizing, or even
tuning very much, (and certainly not setting policy) for this
pathological case. Most people did the right thing and got in early
enough or held off and the system worked for those people.

- A
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Asa Dotzler-2
In reply to this post by Justin Wood (Callek)-2
On 5/24/2011 4:06 PM, Mike Shaver wrote:

> On Tue, May 24, 2011 at 3:41 PM, Justin Wood (Callek)<[hidden email]>  wrote:
>> On 5/24/2011 6:08 PM, Marco Bonardo wrote:
>>>
>>> Another issue was the xpcshell.ini late change in the cycle,
>>
>> This, imo, is the type of change that shouldn't land in last week or two of
>> a cycle,
>
> Disagree.  There are no cycles, they are an illusion perpetrated by
> the release team.  If things land a few days later because of a
> disruptive change, they might miss a cut-over, or a weekend of
> nightly-bake or whatever.
>
> Maybe we should slightly randomize the dates of cut-over.
>
> Mike

I proposed this, jokingly, on IRC -- that we just pick a nice green
changeset from some arbitrary time in the last three or four days and
migrate that to Aurora.

- A (illusion perpetrator)
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Shawn Wilsher-3
In reply to this post by Mike Shaver
On 5/24/2011 4:06 PM, Mike Shaver wrote:
> Maybe we should slightly randomize the dates of cut-over.
Just because our migration date is a Tuesday doesn't mean the release
team has to take everything right up until then.  They could very easily
pick a known-good state before then, and take up to that, right?

It does make it harder for people to properly set the Target Milestone
in bugzilla, but then the release team could go fix bugs they didn't
take too.

Cheers,

Shawn


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Asa Dotzler-2
In reply to this post by Mike Shaver
On 5/24/2011 4:24 PM, Shawn Wilsher wrote:

> On 5/24/2011 4:06 PM, Mike Shaver wrote:
>> Maybe we should slightly randomize the dates of cut-over.
> Just because our migration date is a Tuesday doesn't mean the release
> team has to take everything right up until then. They could very easily
> pick a known-good state before then, and take up to that, right?
>
> It does make it harder for people to properly set the Target Milestone
> in bugzilla, but then the release team could go fix bugs they didn't
> take too.
>
> Cheers,
>
> Shawn
>

We discussed this at this morning's migration meeting and agreed that we
should abide by the published plan this time. I think this is worth
considering for next time, but not as a counter to the day before chaos
so much as a way around the "we migrate even if the tree is red" implied
by that published plan.

- A
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Robert O'Callahan-3
In reply to this post by Mike Shaver
On Wed, May 25, 2011 at 11:06 AM, Mike Shaver <[hidden email]> wrote:

> Disagree.  There are no cycles, they are an illusion perpetrated by
> the release team.
>

I don't agree with that. Six weeks is still a significant amount of time,
and I think it's a good thing for people to try to get something into
release N instead of N + 1 --- when that can be done without breaking stuff.

Maybe we should slightly randomize the dates of cut-over.
>

I like this plan!

Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Axel Hecht
In reply to this post by Justin Wood (Callek)-2
On 25.05.11 01:06, Mike Shaver wrote:

> On Tue, May 24, 2011 at 3:41 PM, Justin Wood (Callek)<[hidden email]>  wrote:
>> On 5/24/2011 6:08 PM, Marco Bonardo wrote:
>>>
>>> Another issue was the xpcshell.ini late change in the cycle,
>>
>> This, imo, is the type of change that shouldn't land in last week or two of
>> a cycle,
>
> Disagree.  There are no cycles, they are an illusion perpetrated by
> the release team.  If things land a few days later because of a
> disruptive change, they might miss a cut-over, or a weekend of
> nightly-bake or whatever.
>
> Maybe we should slightly randomize the dates of cut-over.

I disagree. The main benefit of having a strict calendar is that people
can plan for it. Apparently not everybody's plan worked out this time,
but the learning curve isn't that bad, I think.

The ability to plan is not only good for developers on mozilla-central,
it's also a great feature for localizers. It's only a few hours after
the migration, and we already have 8 good localizations on aurora.

Randomizing the cut-over date can be done in a ton of ways, but I didn't
come up with a way that wouldn't cut into the benefits of the new model
for l10n.

Axel
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Wes Garland
In reply to this post by Mike Shaver
On 24 May 2011 19:06, Mike Shaver <[hidden email]> wrote:

> There are no cycles, they are an illusion perpetrated by
> the release team.


Cycles may be an illusion, but I have found that nothing sharpens the mind
like a deadline.

A slight twist on Asa's idea might be interesting - pick the last green
changeset before the cut-over time; otherwise treat Aurora day as you would
any other (busy?) day.

This offers the benefits of the illusory cycle, while putting social
pressure on those who might otherwise rush-push patches that risk green.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Shawn Wilsher-3
In reply to this post by Axel Hecht
On 5/24/2011 11:20 PM, Axel Hecht wrote:
> The ability to plan is not only good for developers on mozilla-central,
> it's also a great feature for localizers. It's only a few hours after
> the migration, and we already have 8 good localizations on aurora.
>
> Randomizing the cut-over date can be done in a ton of ways, but I didn't
> come up with a way that wouldn't cut into the benefits of the new model
> for l10n.
I'm failing to see how this would impact l10n in any meaningful way.
Nobody is suggesting that the migration happen on a random day, but just
that the time we choose to take changesets up to be somewhat random
instead of "must be in m-c by Monday 11:59 PM PDT".  Migration would
still happen at it's regularly scheduled time.

Cheers,

Shawn


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Robert Kaiser
In reply to this post by Matt Brubeck-3
L. David Baron schrieb:
> I think the idea of the new cycle is that we sort this stuff out on
> Aurora, not on mozilla-central, so mozilla-central is always open.

My understanding as well.

> That said, those who land right before the pull will be responsible
> for landing fixes in two places, but that's their problem.

Well, on Aurora we should mostly disable them or back them out eagerly,
at least if the fix doesn't look to be damn easy and low risk by itself.
And I would be more eager with those backouts or disabling on stuff that
landed a short time before the cutover, as that didn't have time to bake
and could contain even more problems than the immediate one.

If you have trains leaving regularly, you will always have a few people
race in the last minute to catch one. That's the case in the physical
world (observed it often enough) and the virtual world as well. And in
the end, in our world, it's not too much of a problem, we'll just throw
out the one fast from Aurora that cause problems (to be fixed on trunk
for the next train) and leave the good ones in. I think we just need to
be relaxed about that and just deal with in terms of the train model
instead of trying to fight it happening. Having a bit of a last minute
race is human and just shows that people are eager to see their work go
out to users. As long as it doesn't get too bad, let's just sit back and
enjoy the show and clean up in the first day of Aurora, and we'll be fine.

That said, the race this time was very small compared to what I've seen
in freezes for previous stable versions, so I think the new model works
out fine and a lot of people are aware that they'll just make the next
train. After all, things that didn't make this cutover are still well on
schedule to be shipped this year - which is a major shift compared to
previous freeze deadlines. ;-)

Robert Kaiser


--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community should think about. And most of the
time, I even appreciate irony and fun! :)
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Axel Hecht
In reply to this post by Axel Hecht
On 25.05.11 14:19, Shawn Wilsher wrote:

> On 5/24/2011 11:20 PM, Axel Hecht wrote:
>> The ability to plan is not only good for developers on mozilla-central,
>> it's also a great feature for localizers. It's only a few hours after
>> the migration, and we already have 8 good localizations on aurora.
>>
>> Randomizing the cut-over date can be done in a ton of ways, but I didn't
>> come up with a way that wouldn't cut into the benefits of the new model
>> for l10n.
> I'm failing to see how this would impact l10n in any meaningful way.
> Nobody is suggesting that the migration happen on a random day, but just
> that the time we choose to take changesets up to be somewhat random
> instead of "must be in m-c by Monday 11:59 PM PDT". Migration would
> still happen at it's regularly scheduled time.

Well, you're restricting shaver's comment quite significantly.

So in that scenario, (merge war room happens on schedule, and picks a
changeset from the past):

If there are l10n-impact changes between the picked changeset and the
current state of the tree, localizers could end up landing content on
l10n-central that's not good for aurora. That did happen when we cut
aurora the first time in a way. Pretty naughty it was.

We can further restrict the randomness, of course:

- limit the timewindow that we're willing to go back during warroom.

Either by date, or by saying you're not crossing an l10n-impact landing.

The first will basically put an l10n embargo for the maximum time window
on central, I don't think that's a win.

Limiting the random to not cross an l10n-impact landing is going to work
technically, but it might just make the window in which you can choose a
changeset for aurora rather small.

Axel
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mark Finkle
In reply to this post by John O'Duinn
On May 24, 2:44 pm, Mike Shaver <[hidden email]> wrote:

> On Tue, May 24, 2011 at 2:01 PM, John O'Duinn <[hidden email]> wrote:
> > Thanks for posting this Matt. Excellent summary of why not to rush in
> > last minute - its exactly what this rapid release cadence is to help
> > avoid...
>
> I think the opposite, actually.
>
> - you can rush to m-c any day you want
> - aurora cut-over should be just another day in the life of m-c, a
> mere release mechanic
> - we deal with messy rushes that day like any other: piecemeal or
> wholesale backout, etc.

I agree with this and would add the following: Benjamin Smedberg made
a great decision to just pick a changeset that was green, _not_ the
tip of mozilla-central, to use as the migration point. I think this
approach will also help us break our bad habit of rushing to the
deadline. The "deadline" will just move back to a green point anyway.

> Corollary: people should stop thinking about the calendar in the very
> general case.  It'll take time to break that (previously adaptive)
> habit, but we'll get there.

It is very hard to break. I wish we had a few simple ways to try to
break the habit. Going cold turkey isn't easy
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Shaver
On Wed, May 25, 2011 at 9:18 AM, Mark Finkle <[hidden email]> wrote:
> It is very hard to break. I wish we had a few simple ways to try to
> break the habit. Going cold turkey isn't easy

We're not going cold turkey, really, and I think the first drop in our
dose went pretty well, actually.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Dave Townsend
In reply to this post by Mark Finkle
On 25 May 2011 09:18, Mark Finkle <[hidden email]> wrote:

> On May 24, 2:44 pm, Mike Shaver <[hidden email]> wrote:
> > On Tue, May 24, 2011 at 2:01 PM, John O'Duinn <[hidden email]>
> wrote:
> > > Thanks for posting this Matt. Excellent summary of why not to rush in
> > > last minute - its exactly what this rapid release cadence is to help
> > > avoid...
> >
> > I think the opposite, actually.
> >
> > - you can rush to m-c any day you want
> > - aurora cut-over should be just another day in the life of m-c, a
> > mere release mechanic
> > - we deal with messy rushes that day like any other: piecemeal or
> > wholesale backout, etc.
>
> I agree with this and would add the following: Benjamin Smedberg made
> a great decision to just pick a changeset that was green, _not_ the
> tip of mozilla-central, to use as the migration point. I think this
> approach will also help us break our bad habit of rushing to the
> deadline. The "deadline" will just move back to a green point anyway.
>

We actually did pick the tip of m-c even though tests hadn't completed on it
yet, because that is what we had announced we were going to do.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Marco Bonardo-2
In reply to this post by Mark Finkle
Il 25/05/2011 18:31, Dave Townsend ha scritto:
> We actually did pick the tip of m-c even though tests hadn't completed on it
> yet, because that is what we had announced we were going to do.

And I think, based on the announcement, this was the best choice, it was
clearly said "if you want your stuff in Firefox 6 land it before 10AM".

I suggest for next announcement to generically say "We will choose a
green changeset in this day", without specifying any time.

Cheers,
Marco
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Shaver
In reply to this post by Robert O'Callahan-3
On Tue, May 24, 2011 at 6:21 PM, Robert O'Callahan <[hidden email]> wrote:
> On Wed, May 25, 2011 at 11:06 AM, Mike Shaver <[hidden email]> wrote:
>>
>> Disagree.  There are no cycles, they are an illusion perpetrated by
>> the release team.
>
> I don't agree with that. Six weeks is still a significant amount of time,
> and I think it's a good thing for people to try to get something into
> release N instead of N + 1 --- when that can be done without breaking stuff.

I think we should be trying to optimize globally, not locally
per-feature, and I believe that we get more great stuff to more people
in a shorter time if we don't spend 5 weekdays of every 30 pushing for
faster review, crowding try, not re-testing after other big landings,
stepping on each other's backouts, etc.  Slow is smooth and smooth is
fast.

>> Maybe we should slightly randomize the dates of cut-over.
>
> I like this plan!

Well, liking that plan means that things can still miss for 6 weeks
because they landed a day after the revealed cut-over, so I don't know
how to reconcile that with your first point, tbh.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Christian Legnitto
In reply to this post by Marco Bonardo-2

On May 25, 2011, at 9:46 AM, Marco Bonardo wrote:

> Il 25/05/2011 18:31, Dave Townsend ha scritto:
>> We actually did pick the tip of m-c even though tests hadn't completed on it
>> yet, because that is what we had announced we were going to do.
>
> And I think, based on the announcement, this was the best choice, it was clearly said "if you want your stuff in Firefox 6 land it before 10AM".
>
> I suggest for next announcement to generically say "We will choose a green changeset in this day", without specifying any time.

I don't like the uncertainty that gives devs about their fix making it in. We have been pretty consistent saying we would migrate on red and deal with the fallout on mozilla-aurora (the benefit of having multiple repos, yay!). Pulling a green changeset is a big shift from the previously announce plan and I really don't feel the benefits outweigh the costs.

Christian



_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Benjamin Smedberg
On 5/25/2011 2:19 PM, Christian Legnitto wrote:
> On May 25, 2011, at 9:46 AM, Marco Bonardo wrote:
>
>> Il 25/05/2011 18:31, Dave Townsend ha scritto:
>>> We actually did pick the tip of m-c even though tests hadn't completed on it
>>> yet, because that is what we had announced we were going to do.
>> And I think, based on the announcement, this was the best choice, it was clearly said "if you want your stuff in Firefox 6 land it before 10AM".
>>
>> I suggest for next announcement to generically say "We will choose a green changeset in this day", without specifying any time.
> I don't like the uncertainty that gives devs about their fix making it in. We have been pretty consistent saying we would migrate on red and deal with the fallout on mozilla-aurora (the benefit of having multiple repos, yay!). Pulling a green changeset is a big shift from the previously announce plan and I really don't feel the benefits outweigh the costs.
I strongly disagree. It didn't hurt too badly this time, but I think
that we should be importing "known good" code as measured by our tests.
It's easy to do, and means that the -central tree can be managed
normally without people trying to get in under the wire and dealing with
backouts across multiple locations.

Or we should just pick the changeset which was chosen for the Tuesday
nightly.

--BDS

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Connor-4
In reply to this post by Christian Legnitto

On 2011-05-25, at 2:19 PM, Christian Legnitto wrote:

>
> On May 25, 2011, at 9:46 AM, Marco Bonardo wrote:
>
>> Il 25/05/2011 18:31, Dave Townsend ha scritto:
>>> We actually did pick the tip of m-c even though tests hadn't completed on it
>>> yet, because that is what we had announced we were going to do.
>>
>> And I think, based on the announcement, this was the best choice, it was clearly said "if you want your stuff in Firefox 6 land it before 10AM".
>>
>> I suggest for next announcement to generically say "We will choose a green changeset in this day", without specifying any time.
>
> I don't like the uncertainty that gives devs about their fix making it in. We have been pretty consistent saying we would migrate on red and deal with the fallout on mozilla-aurora (the benefit of having multiple repos, yay!). Pulling a green changeset is a big shift from the previously announce plan and I really don't feel the benefits outweigh the costs.

I don't think the uncertainty is substantially different.  If I'm trying to make a cutoff, I'm already not certain that the tree will be in a good state at the last minute.  Unless I'm going to just land on red to make the cutoff, I don't think this has as much cost as you're implying here.

I think that if a developer really wants to make a cutoff they should be landing well in advance of the merge, not in the last 12 hours.

-- Mike


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Chris Cooper-2
In reply to this post by Benjamin Smedberg
On 2011-05-25 2:24 PM, Benjamin Smedberg wrote:
> Or we should just pick the changeset which was chosen for the Tuesday
> nightly.

This seems like a good solution to me. The automation has already found
a known-good changeset for you.

cheers,
--
coop


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

signature.asc (299 bytes) Download Attachment
123