Rushing to meet deadlines

classic Classic list List threaded Threaded
58 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Dao-6
On 25.05.2011 20:24, Benjamin Smedberg wrote:
> Or we should just pick the changeset which was chosen for the Tuesday
> nightly.

That changeset was chosen because it was successfully built across
platforms, not because tests were passing.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Christian Legnitto
In reply to this post by Chris Cooper-2
On May 25, 2011, at 12:22 PM, Chris Cooper wrote:

> On 2011-05-25 2:24 PM, Benjamin Smedberg wrote:
>> Or we should just pick the changeset which was chosen for the Tuesday
>> nightly.
>
> This seems like a good solution to me. The automation has already found
> a known-good changeset for you.

Will this prevent the last minute rush, which is the whole point of this thread? I bet it won't.

Christian
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Benjamin Smedberg
On 5/25/2011 3:27 PM, Christian Legnitto wrote:
> On May 25, 2011, at 12:22 PM, Chris Cooper wrote:
>
>> On 2011-05-25 2:24 PM, Benjamin Smedberg wrote:
>>> Or we should just pick the changeset which was chosen for the Tuesday
>>> nightly.
>> This seems like a good solution to me. The automation has already found
>> a known-good changeset for you.
> Will this prevent the last minute rush, which is the whole point of this thread? I bet it won't.
It will be an incentive to land a day or two earlier, certainly. And the
only people it would hurt are the people who actually landed in the
middle of a non-good tree, which isn't really supposed to happen anyway.
So I think it's a win-win.

--BDS

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Connor-4

On 2011-05-25, at 3:34 PM, Benjamin Smedberg wrote:

> On 5/25/2011 3:27 PM, Christian Legnitto wrote:
>> On May 25, 2011, at 12:22 PM, Chris Cooper wrote:
>>
>>> On 2011-05-25 2:24 PM, Benjamin Smedberg wrote:
>>>> Or we should just pick the changeset which was chosen for the Tuesday
>>>> nightly.
>>> This seems like a good solution to me. The automation has already found
>>> a known-good changeset for you.
>> Will this prevent the last minute rush, which is the whole point of this thread? I bet it won't.
> It will be an incentive to land a day or two earlier, certainly. And the only people it would hurt are the people who actually landed in the middle of a non-good tree, which isn't really supposed to happen anyway. So I think it's a win-win.

If there is a last minute, there will be a last minute rush.  Going back to the last green changeset means that people who tried to beat the buzzer will frequently miss out.  That should be enough to change behaviours, but if not it at least means that the Aurora merge doesn't have to incur that overhead.

-- Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Armen Zambrano G.
In reply to this post by Benjamin Smedberg
On 11-05-25 3:40 PM, Mike Connor wrote:

>
> On 2011-05-25, at 3:34 PM, Benjamin Smedberg wrote:
>
>> On 5/25/2011 3:27 PM, Christian Legnitto wrote:
>>> On May 25, 2011, at 12:22 PM, Chris Cooper wrote:
>>>
>>>> On 2011-05-25 2:24 PM, Benjamin Smedberg wrote:
>>>>> Or we should just pick the changeset which was chosen for the Tuesday
>>>>> nightly.
>>>> This seems like a good solution to me. The automation has already found
>>>> a known-good changeset for you.
>>> Will this prevent the last minute rush, which is the whole point of this thread? I bet it won't.
>> It will be an incentive to land a day or two earlier, certainly. And the only people it would hurt are the people who actually landed in the middle of a non-good tree, which isn't really supposed to happen anyway. So I think it's a win-win.
>
> If there is a last minute, there will be a last minute rush.  Going back to the last green changeset means that people who tried to beat the buzzer will frequently miss out.  That should be enough to change behaviours, but if not it at least means that the Aurora merge doesn't have to incur that overhead.
>
> -- Mike
If we use the changeset of the last nightly we would know that the
aurora build will have the same issues that would be reported during the
day of the merge by m-c users. If we take any changeset after the
previous nightly cutoff we will be testing something slightly different.

What do you think?

cheers,
Armen
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Axel Hecht
In reply to this post by Christian Legnitto
On 25.05.11 20:24, Benjamin Smedberg wrote:

> On 5/25/2011 2:19 PM, Christian Legnitto wrote:
>> On May 25, 2011, at 9:46 AM, Marco Bonardo wrote:
>>
>>> Il 25/05/2011 18:31, Dave Townsend ha scritto:
>>>> We actually did pick the tip of m-c even though tests hadn't
>>>> completed on it
>>>> yet, because that is what we had announced we were going to do.
>>> And I think, based on the announcement, this was the best choice, it
>>> was clearly said "if you want your stuff in Firefox 6 land it before
>>> 10AM".
>>>
>>> I suggest for next announcement to generically say "We will choose a
>>> green changeset in this day", without specifying any time.
>> I don't like the uncertainty that gives devs about their fix making it
>> in. We have been pretty consistent saying we would migrate on red and
>> deal with the fallout on mozilla-aurora (the benefit of having
>> multiple repos, yay!). Pulling a green changeset is a big shift from
>> the previously announce plan and I really don't feel the benefits
>> outweigh the costs.
> I strongly disagree. It didn't hurt too badly this time, but I think
> that we should be importing "known good" code as measured by our tests.
> It's easy to do, and means that the -central tree can be managed
> normally without people trying to get in under the wire and dealing with
> backouts across multiple locations.
>
> Or we should just pick the changeset which was chosen for the Tuesday
> nightly.
>

What's the exact algorithm the nightlies use?

Axel
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Marco Bonardo-2
In reply to this post by Christian Legnitto
Il 25/05/2011 20:24, Benjamin Smedberg ha scritto:
> Or we should just pick the changeset which was chosen for the Tuesday
> nightly.

While would be good to have nightly testing on a known changeset (taking
another changeset means nightly testers are using something with more or
less fixes), this won't prevent taking a bad changeset.
In this specific case, the nightly contained a fix that was backed out 3
pushes later, due to causing random crashes in CC. Then this is nor much
different from the current cherrypicking.

Taking the nightly, while better for testing, would also not solve the
rush "problem", saying "push before 3:30AM to be in the release" is not
different from "push before 10AM to be in the release".

Personally I think that, more than finding a solution to randomize the
time of the merge, would be better to concentrate on finding reasons
that bring patches and features to be ready on the last day, in the last
hours.
I don't think that I'd stay awake till the morning to push a patch, if I
could have it reviewed and green on try a week, or even just a day
before. I want to push when things are ready, so why things get ready
exactly in those last hours?
Faster Try, earlier feedback on approaches/UI/patches, more reviewers
for strategic components, may make the difference here.

When I get last-minute feedback, last-minute reviews, and 10 hours to
get results from Try, I find myself ready just some hour before the merge.
I could just skip it and take the next train, but that looks like a
failure for me and I feel sorry for my project manager, my reviewer and
also users. Most likely this psychological thing matters too, we still
fail thinking that it's just matter of 6 weeks. I'm not yet used to
this, honestly, but I think it will get better.

Cheers,
Marco
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Robert O'Callahan-3
In reply to this post by Mike Shaver
On Thu, May 26, 2011 at 4:58 AM, Mike Shaver <[hidden email]> wrote:

> I think we should be trying to optimize globally, not locally
> per-feature, and I believe that we get more great stuff to more people
> in a shorter time if we don't spend 5 weekdays of every 30 pushing for
> faster review, crowding try, not re-testing after other big landings,
> stepping on each other's backouts, etc.  Slow is smooth and smooth is
> fast.
>

Some of those are obvious bad things that we should never do.

But saying "hey, if you review my feature patch today and my big refactoring
patch tomorrow instead of the other way around, users may get this feature
six weeks earlier" seems perfectly rational to me. Always ignoring the
release cycle would not be rational.

Less hypothetically, I interrupted some of my bigger projects to do a batch
of small regression fixes that I wanted to be in FF6 instead of FF7. I don't
think that was wrong. The fixes landed via cedar a couple of days before the
cutoff, and no process violations occurred.

>> Maybe we should slightly randomize the dates of cut-over.
> >
> > I like this plan!
>
> Well, liking that plan means that things can still miss for 6 weeks
> because they landed a day after the revealed cut-over, so I don't know
> how to reconcile that with your first point, tbh.
>

"Optimizing for the timing of the release cycle" can mean more than
"everything gets checked in at the last minute", and "sometimes unexpected
events will make you wait for the next train" will always be true
regardless.

FWIW, on reflection I like "most recent green-ish changeset" more than
randomized choice.

Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Daniel Cater
In reply to this post by Matt Brubeck-3
On Wednesday, 25 May 2011 19:24:02 UTC+1, Benjamin Smedberg  wrote:
> Or we should just pick the changeset which was chosen for the Tuesday
> nightly.
>
> --BDS

I think this is a good idea. It means that you get all of the nightly testers running the same code that will be in the first Aurora build. It also slightly helps the "in before the deadline" problem as the changeset used for nightlies isn't entirely predictable. All builds have to go green and the build time of the longest build is long and varies quite a lot. If a changeset hasn't finished building across all platforms by the nightly start time then a previous changeset will be used.

In future, I expect the code to choose the nightly changeset will take into account the test runs as well.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Ehsan Akhgari
In reply to this post by Chris Cooper-2
On 11-05-25 3:22 PM, Chris Cooper wrote:
> On 2011-05-25 2:24 PM, Benjamin Smedberg wrote:
>> Or we should just pick the changeset which was chosen for the Tuesday
>> nightly.
>
> This seems like a good solution to me. The automation has already found
> a known-good changeset for you.

FWIW, the second sentence is actually false!  It has picked the last
changeset known to build successfully.  There is a huge difference
between the two.

Cheers,
Ehsan
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Chris Cooper-2
On 2011-05-25 9:27 PM, Ehsan Akhgari wrote:
> FWIW, the second sentence is actually false!  It has picked the last
> changeset known to build successfully.  There is a huge difference
> between the two.

Replace "known-good" with "known-to-build" and you have my intent.

It just means one less hurdle to be overcome in the first days of the a
new Aurora cycle, and drivers don't have to play favorites with
late-landing developer check-ins.

cheers,
--
coop


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

signature.asc (299 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Connor-4
In reply to this post by Armen Zambrano G.

On 2011-05-25, at 4:13 PM, Armen Zambrano Gasparnian wrote:

> On 11-05-25 3:40 PM, Mike Connor wrote:
>>
>> On 2011-05-25, at 3:34 PM, Benjamin Smedberg wrote:
>>
>>> On 5/25/2011 3:27 PM, Christian Legnitto wrote:
>>>> On May 25, 2011, at 12:22 PM, Chris Cooper wrote:
>>>>
>>>>> On 2011-05-25 2:24 PM, Benjamin Smedberg wrote:
>>>>>> Or we should just pick the changeset which was chosen for the Tuesday
>>>>>> nightly.
>>>>> This seems like a good solution to me. The automation has already found
>>>>> a known-good changeset for you.
>>>> Will this prevent the last minute rush, which is the whole point of this thread? I bet it won't.
>>> It will be an incentive to land a day or two earlier, certainly. And the only people it would hurt are the people who actually landed in the middle of a non-good tree, which isn't really supposed to happen anyway. So I think it's a win-win.
>>
>> If there is a last minute, there will be a last minute rush.  Going back to the last green changeset means that people who tried to beat the buzzer will frequently miss out.  That should be enough to change behaviours, but if not it at least means that the Aurora merge doesn't have to incur that overhead.
>>
>> -- Mike
> If we use the changeset of the last nightly we would know that the aurora build will have the same issues that would be reported during the day of the merge by m-c users. If we take any changeset after the previous nightly cutoff we will be testing something slightly different.
>
> What do you think?

I think I'd be okay with this if nightly builds actually built off all-green changesets.  Since they don't, I think the value is questionable.  I see very little argument presented for taking untested csets onto Aurora if we can simply go back to an actually-known-good changeset.

That we also provide a bigger disincentive for last-minute and/or risky landings (i.e. your change is not guaranteed to make Aurora unless you get a green run before the cutoff) is a strong bonus and fulfils the overall desire to minimize/remove this pain.

-- Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Rob Sayre-2
In reply to this post by Matt Brubeck-3
On Wednesday, May 25, 2011 6:06:52 PM UTC-7, Robert O&#39;Callahan wrote:
>
> But saying "hey, if you review my feature patch today and my big refactoring
> patch tomorrow instead of the other way around, users may get this feature
> six weeks earlier" seems perfectly rational to me. Always ignoring the
> release cycle would not be rational.

I agree. But I want to suggest that we give this process some time to sink in. I wonder if we will see frantic check-in behavior for Firefox 10. I think we should see whether developers react differently once they realize that these events happen often.

Also, we have a relatively green tree on mozilla-central today, so we didn't lose tons of time. Recall that our previous release tactics have resulted in multi-day tree closures.

- Rob
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Rob Sayre-2
In reply to this post by Matt Brubeck-3
On Tuesday, May 24, 2011 9:01:07 AM UTC-7, Matt Brubeck wrote:
>
> I speak from experience

I will be taking the advice Matt gave here. It's understandable for a development effort to miss a target release. It's only when code misses two or three or four release opportunities (depending on the complexity) that we'll have to look closer at what's happening.

2 release cycles: 12 weeks, 2.76 months
3 release cycles: 18 weeks, 4.14 months
4 release cycles: 24 weeks, 5.52 months

Some efforts will take 6 months or more than a year. Hopefully, we'll be able to spot those by the amount of planning and resources we're putting in. Or, we'll be up front about taking a risk that may not pan out.

- Rob
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Asa Dotzler-2
In reply to this post by Marco Bonardo-2
On 5/25/2011 4:21 PM, Marco Bonardo wrote:
> Il 25/05/2011 20:24, Benjamin Smedberg ha scritto:
>> Or we should just pick the changeset which was chosen for the Tuesday
>> nightly.
>
> While would be good to have nightly testing on a known changeset (taking
> another changeset means nightly testers are using something with more or
> less fixes), this won't prevent taking a bad changeset.

It's also practically not interesting. We're going to spend a few days
evaluating and turning off or backing out things that aren't ready on
Aurora before we deliver our first build to the Aurora channel so we're
probably never going to match the nightly anyway.

- A
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Robert Kaiser
In reply to this post by Mike Shaver
Robert O'Callahan schrieb:
> "Optimizing for the timing of the release cycle" can mean more than
> "everything gets checked in at the last minute"

Actually, for me, it means the exact opposite - good optimization for
the cycle means that the big things will land in the first two weeks and
smaller fixes as well as improvements on those bug things will land in
the latter four weeks of it. Somehow that reminds me of the "merge
windows" used for the Linux kernel at the beginning of each cycle...

Robert Kaiser

--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community should think about. And most of the
time, I even appreciate irony and fun! :)
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Mike Hommey-5
On Thu, May 26, 2011 at 06:11:05PM +0200, Robert Kaiser wrote:

> Robert O'Callahan schrieb:
> >"Optimizing for the timing of the release cycle" can mean more than
> >"everything gets checked in at the last minute"
>
> Actually, for me, it means the exact opposite - good optimization
> for the cycle means that the big things will land in the first two
> weeks and smaller fixes as well as improvements on those bug things
> will land in the latter four weeks of it. Somehow that reminds me of
> the "merge windows" used for the Linux kernel at the beginning of
> each cycle...

But then, the Linux kernel development cycle doesn't work with fixed
date, but simply waits for things to be good enough after the fixups
period following the merge window.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Rushing to meet deadlines

Robert Kaiser
In reply to this post by Robert Kaiser
Mike Hommey schrieb:

> On Thu, May 26, 2011 at 06:11:05PM +0200, Robert Kaiser wrote:
>> Robert O'Callahan schrieb:
>>> "Optimizing for the timing of the release cycle" can mean more than
>>> "everything gets checked in at the last minute"
>>
>> Actually, for me, it means the exact opposite - good optimization
>> for the cycle means that the big things will land in the first two
>> weeks and smaller fixes as well as improvements on those bug things
>> will land in the latter four weeks of it. Somehow that reminds me of
>> the "merge windows" used for the Linux kernel at the beginning of
>> each cycle...
>
> But then, the Linux kernel development cycle doesn't work with fixed
> date, but simply waits for things to be good enough after the fixups
> period following the merge window.

Sure, and they don't ship binaries to end users, etc. - What I meant was
just the roughly two weeks of merging the large changes at the beginning
of the cycle.

Robert Kaiser


--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community should think about. And most of the
time, I even appreciate irony and fun! :)
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
123