Coupled Train Model

classic Classic list List threaded Threaded
48 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Coupled Train Model

Alexander Keybl
The main purpose of Aurora as a channel is to get early feedback on new Firefox versions with users that better represent our Beta/Release populations. Aurora users knowingly accept a lower quality bar than Beta/Release, as well as more frequent updates.

Aurora is a fantastic tool, but it's my team's opinion that it doesn't make great use of the full six weeks allotted to it in each development cycle. We find most major Aurora issues in the first week or so after the merge, and critical Beta-blocking issues are typically resolved within days thereafter. We've verified this premise by taking a look at bugs tracked for Firefox 24, fixed in the Aurora timeframe.

We have an informal saying in Release Management that we should always "get new code in front of the most users as soon as possible". Given that goal, I'd like us to spend even more of the 18 weeks with the much larger Beta population, since that population is the most applicable to Release in diversity and quality on Aurora is already fairly high. Here's my proposed "Coupled Train Model" (thanks to curtisk for the naming):

* change the desktop/android train model to have two 9-week cycles rather than three 6-week cycles (still 18 weeks start to finish)
* enable Aurora updates the day after release, rather than at the end of the week
* merge mozilla-aurora to mozilla-beta as soon as critical quality issues have been resolved on both Aurora and Release (1-2 weeks after that)
** we'd utilize the 1-2 week overlap to push dot releases to the Beta population prior to Release (bonus functionality!)
* continue to have certain features only enabled on Aurora and lower
* continue to use mozilla-aurora and the Aurora population as a method of testing fixes prior to uplift to mozilla-beta
* allow developers to focus on two pre-release Gecko versions at a time, rather than three

This proposal sees the Aurora population as a shorter, intermediate step towards Beta testing rather than an equally weighted phase of development (in terms of time). A chart explaining the Coupled Train Model can be found at https://pbs.twimg.com/media/BV7r23WCMAAROMP.png:large

Things we're still considering, and we're meeting separately on:

* a start date - current proposal is starting with Firefox 30 in the new year
* a new string freeze date - current proposal is upon the merge to Beta, since features that aren't ready for release will be backed out by that point
* an API freeze date, for add-on/plugin authors - still need to determine a hard date, but somewhere shortly after the merge to Beta
* when (and from where) to branch future FxOS releases
* security update frequency and the bar for a security dot release (given platform updates every 9 weeks)
* how we would want to communicate the Aurora/Beta "releases" (perhaps together?)

Thanks to all those who have already provided feedback (and early support) at the Open Sessions on Releases in Santa Clara and Toronto. I'd love to hear even more feedback from those community members who weren't able to attend.

-Alex
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Clint Talbert-3
On 10/18/2013 11:10 AM, Alex Keybl wrote:

> * change the desktop/android train model to have two 9-week cycles rather than three 6-week cycles (still 18 weeks start to finish)
> * enable Aurora updates the day after release, rather than at the end of the week
> * merge mozilla-aurora to mozilla-beta as soon as critical quality issues have been resolved on both Aurora and Release (1-2 weeks after that)
> ** we'd utilize the 1-2 week overlap to push dot releases to the Beta population prior to Release (bonus functionality!)
> * continue to have certain features only enabled on Aurora and lower
> * continue to use mozilla-aurora and the Aurora population as a method of testing fixes prior to uplift to mozilla-beta
> * allow developers to focus on two pre-release Gecko versions at a time, rather than three
>
> This proposal sees the Aurora population as a shorter, intermediate step towards Beta testing rather than an equally weighted phase of development (in terms of time). A chart explaining the Coupled Train Model can be found at https://pbs.twimg.com/media/BV7r23WCMAAROMP.png:large
>
>
I could be confused here, but I don't see how this is going to work with
QA. The way I read the diagram, the two weeks of "fixing aurora" are
going to happen at the same time as "doing stabilization point releases
on beta for the new release".  I don't think we want to force two
conflicting priorities on the QA teams where they have to chose between
supporting release stabilization and ensuring that aurora has stabilized
to be uplifted into beta.

Furthermore, the b2g side of the story is really unclear. B2G is trying
hard to align with the train model, and I don't see how the b2g model
would adapt to this.

Clint
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Alexander Keybl
> I could be confused here, but I don't see how this is going to work with QA. The way I read the diagram, the two weeks of "fixing aurora" are going to happen at the same time as "doing stabilization point releases on beta for the new release".  I don't think we want to force two conflicting priorities on the QA teams where they have to chose between supporting release stabilization and ensuring that aurora has stabilized to be uplifted into beta.

I think that's a very narrow view of QA's bandwidth. Rarely do we have to choose between one investigation or another. We maintain 4 branches (3 pre-release), mind you. If we did have to choose, we'd of course prioritize Release though.

> Furthermore, the b2g side of the story is really unclear. B2G is trying hard to align with the train model, and I don't see how the b2g model would adapt to this.

That would be our work, so don't worry about us undoing it. The story is unclear because we haven't made final plans yet.

-Alex

On Oct 21, 2013, at 12:52 PM, Clint Talbert <[hidden email]> wrote:

> On 10/18/2013 11:10 AM, Alex Keybl wrote:
>> * change the desktop/android train model to have two 9-week cycles rather than three 6-week cycles (still 18 weeks start to finish)
>> * enable Aurora updates the day after release, rather than at the end of the week
>> * merge mozilla-aurora to mozilla-beta as soon as critical quality issues have been resolved on both Aurora and Release (1-2 weeks after that)
>> ** we'd utilize the 1-2 week overlap to push dot releases to the Beta population prior to Release (bonus functionality!)
>> * continue to have certain features only enabled on Aurora and lower
>> * continue to use mozilla-aurora and the Aurora population as a method of testing fixes prior to uplift to mozilla-beta
>> * allow developers to focus on two pre-release Gecko versions at a time, rather than three
>>
>> This proposal sees the Aurora population as a shorter, intermediate step towards Beta testing rather than an equally weighted phase of development (in terms of time). A chart explaining the Coupled Train Model can be found at https://pbs.twimg.com/media/BV7r23WCMAAROMP.png:large
>>
>>
> I could be confused here, but I don't see how this is going to work with QA. The way I read the diagram, the two weeks of "fixing aurora" are going to happen at the same time as "doing stabilization point releases on beta for the new release".  I don't think we want to force two conflicting priorities on the QA teams where they have to chose between supporting release stabilization and ensuring that aurora has stabilized to be uplifted into beta.
>
> Furthermore, the b2g side of the story is really unclear. B2G is trying hard to align with the train model, and I don't see how the b2g model would adapt to this.
>
> Clint

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

ashughes
In reply to this post by Alexander Keybl
On Monday, October 21, 2013 10:54:44 AM UTC-7, Alex Keybl wrote:
> > I could be confused here, but I don't see how this is going to work with QA. The way I read the diagram, the two weeks of "fixing aurora" are going to happen at the same time as "doing stabilization point releases on beta for the new release".  I don't think we want to force two conflicting priorities on the QA teams where they have to chose between supporting release stabilization and ensuring that aurora has stabilized to be uplifted into beta.
>
>
>
> I think that's a very narrow view of QA's bandwidth. Rarely do we have to choose between one investigation or another. We maintain 4 branches (3 pre-release), mind you. If we did have to choose, we'd of course prioritize Release though.

I cannot nor will I speak for the entire QA org; the following is purely my personal opinion. I'm not yet convinced that Aurora and the current train model are broken. I suspect it's how we are using them (or not) that is broken.

In my experience it's taken nearly everything Desktop QA has to keep up with the pace of uplifts and testing of Betas. I personally believe that QA should be living in Aurora but the current state of things does not afford us that opportunity. To get there we need better automation, tools for community involvement, and stricter, more clearly defined rules governing uplifts (especially on Beta).

I do not see how this proposal benefits the quality/stability of the product(s) we ship and I fear it might impose an even greater challenge on an already stretched QA team.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Karl Tomlinson-4
In reply to this post by Alexander Keybl
On Fri, 18 Oct 2013 14:10:52 -0400, Alex Keybl wrote:

> Aurora is a fantastic tool, but it's my team's opinion that it
> doesn't make great use of the full six weeks allotted to it in
> each development cycle. We find most major Aurora issues in the
> first week or so after the merge, and critical Beta-blocking
> issues are typically resolved within days thereafter. We've
> verified this premise by taking a look at bugs tracked for
> Firefox 24, fixed in the Aurora timeframe.
>
> We have an informal saying in Release Management that we should
> always "get new code in front of the most users as soon as
> possible". Given that goal, I'd like us to spend even more of
> the 18 weeks with the much larger Beta population, since that
> population is the most applicable to Release in diversity and
> quality on Aurora is already fairly high.

The benefits that I see from the proposal are:

1. Increase number of weeks of beta testing from 6 to hopefully 7.

2. Reduce time between m-c landing and release from 12-18 down to
   9-18.  Assuming evenly distributed landings the average
   reduction is 1.5 weeks.

The main risk/disadvantage is lowering the quality of aurora and
beta.  The Aurora channel would be almost as risky as Nightly at
merge time.  Beta would have 2 weeks of stabilization instead of 6.

My experience is that there are plenty of bugs on Nightly that
take more than 2 weeks to filter through triage, get a fix and
review.

Those fixes, and others that are resolved before 2 weeks,
currently have some time of testing before they need to appear on
Beta.  The new proposal either doesn't have testing time for these
fixes, or they don't make it in time for Beta users.

I would like to ask:

A. If most beta blocking issues are resolved within days, then how
   valuable is an extra week of beta testing?

B. Is it worth the cost of a lower quality beta?
   A lower quality beta would mean fewer users on that train, and
   therefore a narrower, less diverse, sample of testers and lower
   quality testing, IMHO more likely leading to a lower quality
   release product.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Chris Peterson-12
On 10/21/13 3:10 PM, Karl Tomlinson wrote:
> 1. Increase number of weeks of beta testing from 6 to hopefully 7.
>
> 2. Reduce time between m-c landing and release from 12-18 down to
>     9-18.  Assuming evenly distributed landings the average
>     reduction is 1.5 weeks.

Plus, the estimated 7 weeks of Beta assumes that, because we currently
stablize 6 weeks of Nightly development with 2 weeks on Aurora, we can
stablize 9 weeks of Nightly development with 2 weeks on Aurora.

Do we have reason to believe that Aurora stablization won't also
increase by 1.5x to 3 weeks? That would give us only 6 weeks of Beta
testing, i.e. no change from today's Beta duration, except with 3 weeks
of Aurora "bake time" (instead of 6) of 1.5x more code.


cpeterson
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Lawrence Mandel-2
----- Original Message -----

> On 10/21/13 3:10 PM, Karl Tomlinson wrote:
> > 1. Increase number of weeks of beta testing from 6 to hopefully 7.
> >
> > 2. Reduce time between m-c landing and release from 12-18 down to
> >     9-18.  Assuming evenly distributed landings the average
> >     reduction is 1.5 weeks.
>
> Plus, the estimated 7 weeks of Beta assumes that, because we currently
> stablize 6 weeks of Nightly development with 2 weeks on Aurora, we can
> stablize 9 weeks of Nightly development with 2 weeks on Aurora.
>
> Do we have reason to believe that Aurora stablization won't also
> increase by 1.5x to 3 weeks? That would give us only 6 weeks of Beta
> testing, i.e. no change from today's Beta duration, except with 3 weeks
> of Aurora "bake time" (instead of 6) of 1.5x more code.

This is an interesting thought. IIRC, the original proposal saw Nightly remain at 6 weeks, Aurora drop to 3 weeks and beta move to 9 weeks. With the 9/9 cycle we appear to be optimizing for development time. Do we need/want additional development time in each cycle? Does an additional week on beta really warrant the effort required to make this change?

Lawrence
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Eric Rescorla
Apologies if I have missed this, but I've been reading this thread and I
find myself wondering why we don't adopt a model more like Chrome's
(http://www.chromium.org/getting-involved/dev-channel#TOC-Channels)

- 6 weeks on Nightly ("Canary")
- 6 weeks on Beta
- 6 weeks on Stable

Chrome also has a "Dev" channel which is just the latest Nightly which
has received some testing. We could just rebrand Aurora in that model
and move forward. If we seriously think that Aurora isn't getting much
testing, anyway, wouldn't this be simplest?

Again, apologies if this has been discussed and rejected. If such a
discussion exists, I will gladly accept a pointer.

Thanks,
-Ekr





On Mon, Oct 21, 2013 at 4:30 PM, Lawrence Mandel <[hidden email]>wrote:

> ----- Original Message -----
> > On 10/21/13 3:10 PM, Karl Tomlinson wrote:
> > > 1. Increase number of weeks of beta testing from 6 to hopefully 7.
> > >
> > > 2. Reduce time between m-c landing and release from 12-18 down to
> > >     9-18.  Assuming evenly distributed landings the average
> > >     reduction is 1.5 weeks.
> >
> > Plus, the estimated 7 weeks of Beta assumes that, because we currently
> > stablize 6 weeks of Nightly development with 2 weeks on Aurora, we can
> > stablize 9 weeks of Nightly development with 2 weeks on Aurora.
> >
> > Do we have reason to believe that Aurora stablization won't also
> > increase by 1.5x to 3 weeks? That would give us only 6 weeks of Beta
> > testing, i.e. no change from today's Beta duration, except with 3 weeks
> > of Aurora "bake time" (instead of 6) of 1.5x more code.
>
> This is an interesting thought. IIRC, the original proposal saw Nightly
> remain at 6 weeks, Aurora drop to 3 weeks and beta move to 9 weeks. With
> the 9/9 cycle we appear to be optimizing for development time. Do we
> need/want additional development time in each cycle? Does an additional
> week on beta really warrant the effort required to make this change?
>
> Lawrence
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Asa Dotzler-2
In reply to this post by Alexander Keybl
On 10/18/2013 11:10 AM, Alex Keybl wrote:
> Thanks to all those who have already provided feedback (and early
> support) at the Open Sessions on Releases in Santa Clara and Toronto.
> I'd love to hear even more feedback from those community members who
> weren't able to attend.

I love that you had a session on this at the Summit and I'm sorry I
didn't get a chance to participate in that.

Was there any discussion at the Summit session about how the release
cadence maps not only to the efficient use of limited contributor
resources, but also how it impacts our end users?

I realize that you're not focused on the cadence of end user releases as
much as efficiently utilizing the pre-release channels, but if we're
going to shuffle the deck after two and a half years of rapid releases,
maybe it's worth bringing in more voices from the user-focused teams
like Product Management, Support/User Advocacy, UX/UR, Product
Marketing, etc.

My personal opinion is that our end users would love Firefox at least a
little bit more, possibly a lot more, if our release cadence was
quarterly.  But that's a very different discussion and perhaps one that
was already ruled out at the Summit sessions?

- A
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Ben Francis
In reply to this post by Eric Rescorla
On Tue, Oct 22, 2013 at 5:42 AM, Eric Rescorla <[hidden email]> wrote:

> I
> find myself wondering why we don't adopt a model more like Chrome's
> (http://www.chromium.org/getting-involved/dev-channel#TOC-Channels)
>
> - 6 weeks on Nightly ("Canary")
> - 6 weeks on Beta
> - 6 weeks on Stable


Are there any real blockers preventing us from doing this? If we think
Aurora isn't serving our needs well then removing it altogether in favour
of the 6/6/6 model Chrome uses would certainly equal out a competitive
disadvantage we currently have, with the time it takes to get a feature to
market. It also certainly helps with "getting new code in front of the most
users as soon as possible" and might provide an opportunity for us to
leverage the beta population more for testing and localisation, turning
more users into contributors and moving us closer to the goal of 1 million
Mozillians.

In addition to this it's been a huge effort to move to the 12 week train
model for Firefox OS and we're still only just getting started. It would be
challenging to change that again at the beginning of next year. A 12 week
cycle is already very aggressive from our partners' point of view but
"quarterly" releases is something we've talked quite loudly about as a
differentiator and is potentially disruptive in the mobile industry. A six
week heartbeat would mean we could continue running our 12 week Firefox OS
trains every other heartbeat.

In the long term I would love it if we could roll out silent 6 weekly
zero-rated Gecko updates to Firefox OS devices. We could split out many of
our Gaia apps (like media and productivity apps) into the Marketplace so
that partners have less to test and those apps can release directly to
users.

Let's continue to keep pushing the web forward, faster, on all fronts, but
keep everyone at Mozilla working to the same heartbeat.

My two cents.

Ben
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Francesco Lodolo [:flod]
2013/10/22 Ben Francis <[hidden email]>

> Are there any real blockers preventing us from doing this?


At what point do you plan to freeze strings, and let localizers translate
them?

Francesco
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Alexander Keybl
In reply to this post by Ben Francis
We definitely plan to keep Firefox IS on the same heartbeat as we originally implemented, it's just a matter of how we align.

FxOS should not prevent us from making positive change on desktop, although it's of course an important consideration.

-Alex

----- Original Message -----
From: Ben Francis <[hidden email]>
To: Eric Rescorla <[hidden email]>
Cc: Chris Peterson <[hidden email]>, [hidden email], Lawrence Mandel <[hidden email]>
Sent: Tue, 22 Oct 2013 07:34:55 -0700 (PDT)
Subject: Re: Coupled Train Model

On Tue, Oct 22, 2013 at 5:42 AM, Eric Rescorla <[hidden email]> wrote:

> I
> find myself wondering why we don't adopt a model more like Chrome's
> (http://www.chromium.org/getting-involved/dev-channel#TOC-Channels)
>
> - 6 weeks on Nightly ("Canary")
> - 6 weeks on Beta
> - 6 weeks on Stable


Are there any real blockers preventing us from doing this? If we think
Aurora isn't serving our needs well then removing it altogether in favour
of the 6/6/6 model Chrome uses would certainly equal out a competitive
disadvantage we currently have, with the time it takes to get a feature to
market. It also certainly helps with "getting new code in front of the most
users as soon as possible" and might provide an opportunity for us to
leverage the beta population more for testing and localisation, turning
more users into contributors and moving us closer to the goal of 1 million
Mozillians.

In addition to this it's been a huge effort to move to the 12 week train
model for Firefox OS and we're still only just getting started. It would be
challenging to change that again at the beginning of next year. A 12 week
cycle is already very aggressive from our partners' point of view but
"quarterly" releases is something we've talked quite loudly about as a
differentiator and is potentially disruptive in the mobile industry. A six
week heartbeat would mean we could continue running our 12 week Firefox OS
trains every other heartbeat.

In the long term I would love it if we could roll out silent 6 weekly
zero-rated Gecko updates to Firefox OS devices. We could split out many of
our Gaia apps (like media and productivity apps) into the Marketplace so
that partners have less to test and those apps can release directly to
users.

Let's continue to keep pushing the web forward, faster, on all fronts, but
keep everyone at Mozilla working to the same heartbeat.

My two cents.

Ben
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Alexander Keybl
In reply to this post by Karl Tomlinson-4
It's an extra 1-2 weeks of stabilization time, which will go a long way in stamping out amorphous beta-only issues.

We don't expect there to be an impact to Beta usage, given current quality of the Aurora channel. We do a good job of keeping Nightly in a usable state for the large majority of use cases. We have a discovery issue which we'd like to mitigate through more time with our largest pre-release population.

-Alex

----- Original Message -----
From: Karl Tomlinson <[hidden email]>
To: [hidden email]
Sent: Mon, 21 Oct 2013 15:10:09 -0700 (PDT)
Subject: Re: Coupled Train Model

On Fri, 18 Oct 2013 14:10:52 -0400, Alex Keybl wrote:

> Aurora is a fantastic tool, but it's my team's opinion that it
> doesn't make great use of the full six weeks allotted to it in
> each development cycle. We find most major Aurora issues in the
> first week or so after the merge, and critical Beta-blocking
> issues are typically resolved within days thereafter. We've
> verified this premise by taking a look at bugs tracked for
> Firefox 24, fixed in the Aurora timeframe.
>
> We have an informal saying in Release Management that we should
> always "get new code in front of the most users as soon as
> possible". Given that goal, I'd like us to spend even more of
> the 18 weeks with the much larger Beta population, since that
> population is the most applicable to Release in diversity and
> quality on Aurora is already fairly high.

The benefits that I see from the proposal are:

1. Increase number of weeks of beta testing from 6 to hopefully 7.

2. Reduce time between m-c landing and release from 12-18 down to
 9-18. Assuming evenly distributed landings the average
 reduction is 1.5 weeks.

The main risk/disadvantage is lowering the quality of aurora and
beta. The Aurora channel would be almost as risky as Nightly at
merge time. Beta would have 2 weeks of stabilization instead of 6.

My experience is that there are plenty of bugs on Nightly that
take more than 2 weeks to filter through triage, get a fix and
review.

Those fixes, and others that are resolved before 2 weeks,
currently have some time of testing before they need to appear on
Beta. The new proposal either doesn't have testing time for these
fixes, or they don't make it in time for Beta users.

I would like to ask:

A. If most beta blocking issues are resolved within days, then how
 valuable is an extra week of beta testing?

B. Is it worth the cost of a lower quality beta?
 A lower quality beta would mean fewer users on that train, and
 therefore a narrower, less diverse, sample of testers and lower
 quality testing, IMHO more likely leading to a lower quality
 release product.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Alexander Keybl
In reply to this post by Eric Rescorla
The main problem I see here is regularly secting a Nightly build that we can confidently push to Aurora users. May be too much churn for that population over the course of a cycle (as opposed to once at the beginning of the cycle).

It's obviously not a bad idea (it works for our competition) but we need to make sure our final solution fits our needs.

-Alex

----- Original Message -----
From: Eric Rescorla <[hidden email]>
To: Lawrence Mandel <[hidden email]>
Cc: Chris Peterson <[hidden email]>, [hidden email]
Sent: Mon, 21 Oct 2013 21:42:35 -0700 (PDT)
Subject: Re: Coupled Train Model

Apologies if I have missed this, but I've been reading this thread and I
find myself wondering why we don't adopt a model more like Chrome's
(http://www.chromium.org/getting-involved/dev-channel#TOC-Channels)

- 6 weeks on Nightly ("Canary")
- 6 weeks on Beta
- 6 weeks on Stable

Chrome also has a "Dev" channel which is just the latest Nightly which
has received some testing. We could just rebrand Aurora in that model
and move forward. If we seriously think that Aurora isn't getting much
testing, anyway, wouldn't this be simplest?

Again, apologies if this has been discussed and rejected. If such a
discussion exists, I will gladly accept a pointer.

Thanks,
-Ekr





On Mon, Oct 21, 2013 at 4:30 PM, Lawrence Mandel <[hidden email]>wrote:

> ----- Original Message -----
> > On 10/21/13 3:10 PM, Karl Tomlinson wrote:
> > > 1. Increase number of weeks of beta testing from 6 to hopefully 7.
> > >
> > > 2. Reduce time between m-c landing and release from 12-18 down to
> > > 9-18. Assuming evenly distributed landings the average
> > > reduction is 1.5 weeks.
> >
> > Plus, the estimated 7 weeks of Beta assumes that, because we currently
> > stablize 6 weeks of Nightly development with 2 weeks on Aurora, we can
> > stablize 9 weeks of Nightly development with 2 weeks on Aurora.
> >
> > Do we have reason to believe that Aurora stablization won't also
> > increase by 1.5x to 3 weeks? That would give us only 6 weeks of Beta
> > testing, i.e. no change from today's Beta duration, except with 3 weeks
> > of Aurora "bake time" (instead of 6) of 1.5x more code.
>
> This is an interesting thought. IIRC, the original proposal saw Nightly
> remain at 6 weeks, Aurora drop to 3 weeks and beta move to 9 weeks. With
> the 9/9 cycle we appear to be optimizing for development time. Do we
> need/want additional development time in each cycle? Does an additional
> week on beta really warrant the effort required to make this change?
>
> Lawrence
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Alexander Keybl
In reply to this post by ashughes
It puts a release in front of a much larger pre-release population for more of the cycle, allowing us to identify and fix issues farther from release.

The tweaked model also keeps developers focused in fewer versions, making sure recent changes are fresh in our memory.

It also allows for us to ship dot releases to our Beta users prior to release, a huge quality improvement from where we stand now.

It's not clear to me how spending less time with Aurora will impact QA bandwidth. It should instead help with focus (fewer Gecko versions to juggle).

-Alex

----- Original Message -----
From: anthony s hughes <[hidden email]>
To: [hidden email]
Sent: Mon, 21 Oct 2013 14:29:01 -0700 (PDT)
Subject: Re: Coupled Train Model

On Monday, October 21, 2013 10:54:44 AM UTC-7, Alex Keybl wrote:
> > I could be confused here, but I don't see how this is going to work with QA. The way I read the diagram, the two weeks of "fixing aurora" are going to happen at the same time as "doing stabilization point releases on beta for the new release". I don't think we want to force two conflicting priorities on the QA teams where they have to chose between supporting release stabilization and ensuring that aurora has stabilized to be uplifted into beta.
>
>
>
> I think that's a very narrow view of QA's bandwidth. Rarely do we have to choose between one investigation or another. We maintain 4 branches (3 pre-release), mind you. If we did have to choose, we'd of course prioritize Release though.

I cannot nor will I speak for the entire QA org; the following is purely my personal opinion. I'm not yet convinced that Aurora and the current train model are broken. I suspect it's how we are using them (or not) that is broken.

In my experience it's taken nearly everything Desktop QA has to keep up with the pace of uplifts and testing of Betas. I personally believe that QA should be living in Aurora but the current state of things does not afford us that opportunity. To get there we need better automation, tools for community involvement, and stricter, more clearly defined rules governing uplifts (especially on Beta).

I do not see how this proposal benefits the quality/stability of the product(s) we ship and I fear it might impose an even greater challenge on an already stretched QA team.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Adam Roach
In reply to this post by Ben Francis
On 10/22/13 09:34, Ben Francis wrote:

> On Tue, Oct 22, 2013 at 5:42 AM, Eric Rescorla <[hidden email]> wrote:
>
>> I
>> find myself wondering why we don't adopt a model more like Chrome's
>> (http://www.chromium.org/getting-involved/dev-channel#TOC-Channels)
>>
>> - 6 weeks on Nightly ("Canary")
>> - 6 weeks on Beta
>> - 6 weeks on Stable
>
> Are there any real blockers preventing us from doing this? If we think
> Aurora isn't serving our needs well then removing it altogether in favour
> of the 6/6/6 model Chrome uses would certainly equal out a competitive
> disadvantage we currently have, with the time it takes to get a feature to
> market.

Right. Especially when it comes to new and developing technologies
(WebRTC comes to mind), being handicapped by six weeks contributes to an
overall impression that Firefox is simply less nimble than Chrome. On
top of new features taking longer, we have the additional impact that
any flaws that aren't major enough to uplift take a substantial chunk of
time to get into the hands of users.

I think that there's also an argument that we have a non-trivial
incremental cost in maintaining four release-train trees as compared to
three. So, even if we were to re-engineer the current
Nightly/Aurora/Beta/Release schedules to get from Nightly to Release in
12 weeks, we're still paying an additional price for producing a nearly
un-used product.

As a final point: any argument about why this schedule won't work needs
to adequately explain which of the factors that differentiate the
Firefox team different than the Chrome team are responsible for making
it okay for them but not for us. I'm not saying there aren't any such
reasons; just that I can't easily come up with any that are particularly
compelling.

I think this is a place where being bold would pay large dividends.

--
Adam Roach
Principal Platform Engineer
[hidden email]
+1 650 903 0800 x863
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Ben Francis
In reply to this post by Francesco Lodolo [:flod]
On Tue, Oct 22, 2013 at 4:14 PM, Francesco Lodolo [:flod]
<[hidden email]>wrote:

> 2013/10/22 Ben Francis <[hidden email]>
>
> > Are there any real blockers preventing us from doing this?
>
> At what point do you plan to freeze strings, and let localizers translate
> them?


I don't have all the answers, I was just asking what the blockers are.
Thanks for bringing this one up, it was certainly something that was raised
at the session at the Summit in Toronto.

I think with Chromium they have a string freeze a couple of weeks before
moving a release to beta to provide a bit of a head start and then do the
bulk of the translations over the beta period, but it certainly reduces the
overall amount of time for localisations to happen. Perhaps we don't have
the level of resources needed to make this possible.

We could turn this problem into an opportunity with a campaign to recruit
volunteer translators from our beta testing community. "You are using beta
software, if you see untranslated strings and are able to translate them,
this is how you can help..." We'd certainly be taking a risk, but if we
successfully recruited more translators do you think we could speed up the
localisation process?
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Francesco Lodolo [:flod]
Il 22/10/13 18:59, Ben Francis ha scritto:
> I think with Chromium they have a string freeze a couple of weeks
> before moving a release to beta to provide a bit of a head start and
> then do the bulk of the translations over the beta period, but it
> certainly reduces the overall amount of time for localisations to
> happen. Perhaps we don't have the level of resources needed to make
> this possible.
Which means that beta releases will be low-quality, half-translated
products for the first part of the cycle.

That's far from optimal: low quality in localization means perceived low
quality of the product itself (poor Firefox in Italian = poor Firefox,
period), and as a localizer I will be less than happy to have something
like this available to the public (it would make my work look sloppy).
> We could turn this problem into an opportunity with a campaign to
> recruit volunteer translators from our beta testing community. "You
> are using beta software, if you see untranslated strings and are able
> to translate them, this is how you can help..." We'd certainly be
> taking a risk, but if we successfully recruited more translators do
> you think we could speed up the localisation process?
I personally think that this would just lower the quality of the product
we ship. There are team leaders and communities in place to ensure that
we ship the best product possible (quality obviously varies from locale
to locale), introducing occasional localizations/localizers would make
that almost impossible.

Francesco

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Daniel Veditz-2
In reply to this post by Asa Dotzler-2
On 10/21/2013 10:46 PM, Asa Dotzler wrote:
> Was there any discussion at the Summit session about how the release
> cadence maps not only to the efficient use of limited contributor
> resources, but also how it impacts our end users?

> My personal opinion is that our end users would love Firefox at least a
> little bit more, possibly a lot more, if our release cadence was
> quarterly.  But that's a very different discussion and perhaps one that
> was already ruled out at the Summit sessions?

we did talk about it. The proposal lengthens the time between
feature-changing updates which everyone (most?) agreed would be welcomed
by users who generally don't like change and interruption. My personal
concern is that we then might need to inject a mid-term security update
on a regular basis which doesn't help as much, although such an update
would at least not change any user-facing features or break add-ons.

-Dan Veditz
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Coupled Train Model

Ben Francis
In reply to this post by Francesco Lodolo [:flod]
On Tue, Oct 22, 2013 at 6:10 PM, Francesco Lodolo [:flod]
<[hidden email]>wrote:

> Il 22/10/13 18:59, Ben Francis ha scritto:
>
>  I think with Chromium they have a string freeze a couple of weeks before
>> moving a release to beta to provide a bit of a head start and then do the
>> bulk of the translations over the beta period, but it certainly reduces the
>> overall amount of time for localisations to happen. Perhaps we don't have
>> the level of resources needed to make this possible.
>>
> Which means that beta releases will be low-quality, half-translated
> products for the first part of the cycle.
>

Of course, and that would presumably be intentional. By combining Aurora
and Beta into one channel we would have the cost of lower quality at the
start of the Beta cycle, but the benefit of more eyes finding bugs with the
intention of shortening the stabilisation period. That's not necessarily
the right thing for Mozilla, but it is what one of our competitors does. As
Adam points out, if there's a reason we can't move as fast as they do, we
should identify the reasons for that.

The argument I believe is being put forward by Release Management is that
we don't have enough users using Aurora builds to see any benefit from
them. We can't simulatenously have the goal of putting lower quality
software in front of more users and putting lower quality software in front
of less users. Beta users have opted in to getting new features sooner in
return for a less polished product, we'd just be tweaking those dials a
little. That's not necessarily a bad thing if we communicate it well.

That's far from optimal: low quality in localization means perceived low
> quality of the product itself (poor Firefox in Italian = poor Firefox,
> period), and as a localizer I will be less than happy to have something
> like this available to the public (it would make my work look sloppy).


Aurora is available to the public too, it's just that less people use it.
That's been identified as a problem that needs solving, if that is a
problem then this is one potential solution.

Ben
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
123