Proposed changes to RelEng's OSX build and test infrastructure

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposed changes to RelEng's OSX build and test infrastructure

John O'Duinn
tl;dr: In order to improve our osx10.6 test capacity and to quickly
start osx10.9 testing, we're planning to make the following changes to
our OSX-build-and-test-infrastructure.

1) convert all 10.7 test machines as 10.6 test machines in order to
increase our 10.6 capacity.
2) convert all 10.8 test machines as 10.9 test machines.
3) do most 10.7 builds as osx-cross-compiling-on-linux-on-AWS, repurpose
10.7 builder machines to be additional 10.9 test machines. This
cross-compiler work is ongoing, it will take time to complete, and it
will take time to transition into production, hence, it is listed last
in this list. The curious can follow
https://bugzilla.mozilla.org/show_bug.cgi?id=921040.

Each of these items are large stand-alone projects involving the same
people across multiple groups, so we'll roll each out in the
aforementioned sequence.

Additional details:
1) Removing specific versions of an OS from our continuous integration
systems based on vendor support and/or usage data is not a new policy.
We have done this several times in the past. For example, we have
dropped WinXPsp0/sp1/sp2 for WinXPsp3; dropped WinVista for Win7;
dropped Win7 x64 for Win8 x64; and soon we will drop Win8.0 for Win8.1; ...
** Note for the record that this does *NOT* mean that Mozilla is
dropping support for osx10.7 or 10.8; it just means we think *automated*
testing on 10.6,10.9 is more beneficial.

2) To see Firefox's minimum OS requirements see:
https://www.mozilla.org/en-US/firefox/25.0.1/system-requirements

3) Apple is offering osx10.9 as a free upgrade to all users of osx10.7
and osx10.8. Also, note that 10.9 runs on any machine that can run 10.7
or 10.8. Because the osx10.9 release is a free upgrade, users are
quickly upgrading. We are seeing a drop in both 10.7 and 10.8 users and
in just a month since the 10.9 release, we already have more 10.9 users
than 10.8 users.

4) Distribution of Firefox users from the most to the least (data from
15-nov-2013):
    10.6 - 34%
    10.7 - 23% - slightly decreasing
    10.8 - 21% - notably decreasing
    10.9 - 21% - notably increasing
    more info:
http://armenzg.blogspot.ca/2013/11/re-thinking-our-mac-os-x-continuous.html

5) Apple is no longer providing security updates for 10.7; any user
looking for OS security updates will need to upgrade to 10.9. Because
OSX10.9 is a free upgrade for 10.8 users, we expect 10.8 to be in
similar situation soon.

6) If a developer lands a patch that works on 10.9, but it fails somehow
on 10.7 or 10.8, it is unlikely that we would back out the fix, and we
would instead tell users to upgrade to 10.9 anyways, for the security fixes.

7) It is no longer possible to buy any more of the 10.6 machines (known
as revision 4 minis), as they are long desupported. Recycling 10.7 test
machines means that we can continue to support osx10.6 at scale without
needing to buy/rack/recalibrate test and performance results.

8) Like all other large OS changes, this change would ride the trains.
Most 10.7 and 10.8 test machines would be reimaged when we make these
changes live on mozilla-central and try, while we'd leave a few behind.
The few remaining would be reimaged at each 6-week train migration.

If we move quickly, this reimaging work can be done by IT before they
all get busy with the 650-Castro -> Evelyn move.

For further details, see Armen's blog
http://armenzg.blogspot.ca/2013/11/re-thinking-our-mac-os-x-continuous.html.
To make sure this is not missed, I've cross-posted this to dev.planning,
dev.platform and also blogged about it on
http://oduinn.com/blog/2013/11/21/proposed-changes-to-relengs-osx-build-and-test-infrastructure/.
If you know of anything we have missed, please reply in the dev.planning
thread.


John and Armen.

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

signature.asc (965 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposed changes to RelEng's OSX build and test infrastructure

Nicholas Nethercote
Last I heard, the official recommendation from IT was for Mozilla
folks (esp. devs) to not upgrade to 10.9 yet, because some things
might not work.  Is that no longer the case?

Nick
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed changes to RelEng's OSX build and test infrastructure

Mike Hommey
In reply to this post by John O'Duinn
On Thu, Nov 21, 2013 at 04:56:50PM -0500, John O'Duinn wrote:
> 6) If a developer lands a patch that works on 10.9, but it fails somehow
> on 10.7 or 10.8, it is unlikely that we would back out the fix, and we
> would instead tell users to upgrade to 10.9 anyways, for the security fixes.

It's not because we tell users to do so that they will. Chances are they
will choose another browser that works on their OS than upgrade it. I do
know a few people that don't like the new things in 10.8 and are keeping
10.7 on purpose.

How much effort would it be to image the test machines such that they
can boot multiple versions of OSX, and have them reboot under 10.7 and
10.8 instead of 10.6 and 10.9 when demand is low, and run tests on those
platforms?

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed changes to RelEng's OSX build and test infrastructure

John O'Duinn
In reply to this post by Nicholas Nethercote
hi Nick;

Yes, there was/is a bug about osx10.9 compiler issues, see bug#936977
for details. However, that bug specifically happens with builds-only,
and is unrelated to this tests-only proposal. I'm not aware of anything
preventing us from running 10.9 *testing*.

Hope that helps.
John.
=====
On 11/22/13 12:10 AM, Nicholas Nethercote wrote:
> Last I heard, the official recommendation from IT was for Mozilla
> folks (esp. devs) to not upgrade to 10.9 yet, because some things
> might not work.  Is that no longer the case?
>
> Nick
>

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

signature.asc (965 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposed changes to RelEng's OSX build and test infrastructure

Andrew  McCreight
In reply to this post by John O'Duinn
Note that we currently have a large set of intermittent 10.8 OSX opt-only leaks-until-shutdown with the debugger:
  https://bugzilla.mozilla.org/show_bug.cgi?id=942102

I don't know if these will show up on 10.9 or not, or if they really matter that much.

Andrew

----- Original Message -----

> tl;dr: In order to improve our osx10.6 test capacity and to quickly
> start osx10.9 testing, we're planning to make the following changes to
> our OSX-build-and-test-infrastructure.
>
> 1) convert all 10.7 test machines as 10.6 test machines in order to
> increase our 10.6 capacity.
> 2) convert all 10.8 test machines as 10.9 test machines.
> 3) do most 10.7 builds as osx-cross-compiling-on-linux-on-AWS, repurpose
> 10.7 builder machines to be additional 10.9 test machines. This
> cross-compiler work is ongoing, it will take time to complete, and it
> will take time to transition into production, hence, it is listed last
> in this list. The curious can follow
> https://bugzilla.mozilla.org/show_bug.cgi?id=921040.
>
> Each of these items are large stand-alone projects involving the same
> people across multiple groups, so we'll roll each out in the
> aforementioned sequence.
>
> Additional details:
> 1) Removing specific versions of an OS from our continuous integration
> systems based on vendor support and/or usage data is not a new policy.
> We have done this several times in the past. For example, we have
> dropped WinXPsp0/sp1/sp2 for WinXPsp3; dropped WinVista for Win7;
> dropped Win7 x64 for Win8 x64; and soon we will drop Win8.0 for Win8.1; ...
> ** Note for the record that this does *NOT* mean that Mozilla is
> dropping support for osx10.7 or 10.8; it just means we think *automated*
> testing on 10.6,10.9 is more beneficial.
>
> 2) To see Firefox's minimum OS requirements see:
> https://www.mozilla.org/en-US/firefox/25.0.1/system-requirements
>
> 3) Apple is offering osx10.9 as a free upgrade to all users of osx10.7
> and osx10.8. Also, note that 10.9 runs on any machine that can run 10.7
> or 10.8. Because the osx10.9 release is a free upgrade, users are
> quickly upgrading. We are seeing a drop in both 10.7 and 10.8 users and
> in just a month since the 10.9 release, we already have more 10.9 users
> than 10.8 users.
>
> 4) Distribution of Firefox users from the most to the least (data from
> 15-nov-2013):
>     10.6 - 34%
>     10.7 - 23% - slightly decreasing
>     10.8 - 21% - notably decreasing
>     10.9 - 21% - notably increasing
>     more info:
> http://armenzg.blogspot.ca/2013/11/re-thinking-our-mac-os-x-continuous.html
>
> 5) Apple is no longer providing security updates for 10.7; any user
> looking for OS security updates will need to upgrade to 10.9. Because
> OSX10.9 is a free upgrade for 10.8 users, we expect 10.8 to be in
> similar situation soon.
>
> 6) If a developer lands a patch that works on 10.9, but it fails somehow
> on 10.7 or 10.8, it is unlikely that we would back out the fix, and we
> would instead tell users to upgrade to 10.9 anyways, for the security fixes.
>
> 7) It is no longer possible to buy any more of the 10.6 machines (known
> as revision 4 minis), as they are long desupported. Recycling 10.7 test
> machines means that we can continue to support osx10.6 at scale without
> needing to buy/rack/recalibrate test and performance results.
>
> 8) Like all other large OS changes, this change would ride the trains.
> Most 10.7 and 10.8 test machines would be reimaged when we make these
> changes live on mozilla-central and try, while we'd leave a few behind.
> The few remaining would be reimaged at each 6-week train migration.
>
> If we move quickly, this reimaging work can be done by IT before they
> all get busy with the 650-Castro -> Evelyn move.
>
> For further details, see Armen's blog
> http://armenzg.blogspot.ca/2013/11/re-thinking-our-mac-os-x-continuous.html.
> To make sure this is not missed, I've cross-posted this to dev.planning,
> dev.platform and also blogged about it on
> http://oduinn.com/blog/2013/11/21/proposed-changes-to-relengs-osx-build-and-test-infrastructure/.
> If you know of anything we have missed, please reply in the dev.planning
> thread.
>
>
> John and Armen.
>
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed changes to RelEng's OSX build and test infrastructure

Armen Zambrano G.
In reply to this post by Mike Hommey


----- Original Message -----

> From: "Mike Hommey" <[hidden email]>
> To: "John O'Duinn" <[hidden email]>
> Cc: "dev. planning" <[hidden email]>, [hidden email], "release" <[hidden email]>
> Sent: Friday, November 22, 2013 1:34:46 AM
> Subject: Re: Proposed changes to RelEng's OSX build and test infrastructure
>
> On Thu, Nov 21, 2013 at 04:56:50PM -0500, John O'Duinn wrote:
> > 6) If a developer lands a patch that works on 10.9, but it fails somehow
> > on 10.7 or 10.8, it is unlikely that we would back out the fix, and we
> > would instead tell users to upgrade to 10.9 anyways, for the security
> > fixes.
>
> It's not because we tell users to do so that they will. Chances are they
> will choose another browser that works on their OS than upgrade it. I do
> know a few people that don't like the new things in 10.8 and are keeping
> 10.7 on purpose.
>
> How much effort would it be to image the test machines such that they
> can boot multiple versions of OSX, and have them reboot under 10.7 and
> 10.8 instead of 10.6 and 10.9 when demand is low, and run tests on those
> platforms?
>
We have thought of this in the past, however, we have always been getting higher up goals.
We would have to consult with RelOps (IT's supporting branch for Releng) and it would require some re-work on the imaging systems.
In fact, we're currently working on switching from our current system, DeployStudio, to casper.

On the scheduling side it will require a decent amount of re-factoring. This would have to be added to our planned re-writing of our scheduling system.

This would still have to deal with not getting immediate feedback for commits on 10.7 and 10.8 and how to deal with backouts and regression hunting.

> Mike
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

RE: Re: Proposed changes to RelEng's OSX build and test infrastructure

Armen Zambrano G.
In reply to this post by John O'Duinn
Hi Andrew,
That's good to know. Thanks!




Sent from Samsung tabletAndrew McCreight <[hidden email]> wrote:Note that we currently have a large set of intermittent 10.8 OSX opt-only leaks-until-shutdown with the debugger:
  https://bugzilla.mozilla.org/show_bug.cgi?id=942102

I don't know if these will show up on 10.9 or not, or if they really matter that much.

Andrew

----- Original Message -----

> tl;dr: In order to improve our osx10.6 test capacity and to quickly
> start osx10.9 testing, we're planning to make the following changes to
> our OSX-build-and-test-infrastructure.
>
> 1) convert all 10.7 test machines as 10.6 test machines in order to
> increase our 10.6 capacity.
> 2) convert all 10.8 test machines as 10.9 test machines.
> 3) do most 10.7 builds as osx-cross-compiling-on-linux-on-AWS, repurpose
> 10.7 builder machines to be additional 10.9 test machines. This
> cross-compiler work is ongoing, it will take time to complete, and it
> will take time to transition into production, hence, it is listed last
> in this list. The curious can follow
> https://bugzilla.mozilla.org/show_bug.cgi?id=921040.
>
> Each of these items are large stand-alone projects involving the same
> people across multiple groups, so we'll roll each out in the
> aforementioned sequence.
>
> Additional details:
> 1) Removing specific versions of an OS from our continuous integration
> systems based on vendor support and/or usage data is not a new policy.
> We have done this several times in the past. For example, we have
> dropped WinXPsp0/sp1/sp2 for WinXPsp3; dropped WinVista for Win7;
> dropped Win7 x64 for Win8 x64; and soon we will drop Win8.0 for Win8.1; ...
> ** Note for the record that this does *NOT* mean that Mozilla is
> dropping support for osx10.7 or 10.8; it just means we think *automated*
> testing on 10.6,10.9 is more beneficial.
>
> 2) To see Firefox's minimum OS requirements see:
> https://www.mozilla.org/en-US/firefox/25.0.1/system-requirements
>
> 3) Apple is offering osx10.9 as a free upgrade to all users of osx10.7
> and osx10.8. Also, note that 10.9 runs on any machine that can run 10.7
> or 10.8. Because the osx10.9 release is a free upgrade, users are
> quickly upgrading. We are seeing a drop in both 10.7 and 10.8 users and
> in just a month since the 10.9 release, we already have more 10.9 users
> than 10.8 users.
>
> 4) Distribution of Firefox users from the most to the least (data from
> 15-nov-2013):
>     10.6 - 34%
>     10.7 - 23% - slightly decreasing
>     10.8 - 21% - notably decreasing
>     10.9 - 21% - notably increasing
>     more info:
> http://armenzg.blogspot.ca/2013/11/re-thinking-our-mac-os-x-continuous.html
>
> 5) Apple is no longer providing security updates for 10.7; any user
> looking for OS security updates will need to upgrade to 10.9. Because
> OSX10.9 is a free upgrade for 10.8 users, we expect 10.8 to be in
> similar situation soon.
>
> 6) If a developer lands a patch that works on 10.9, but it fails somehow
> on 10.7 or 10.8, it is unlikely that we would back out the fix, and we
> would instead tell users to upgrade to 10.9 anyways, for the security fixes.
>
> 7) It is no longer possible to buy any more of the 10.6 machines (known
> as revision 4 minis), as they are long desupported. Recycling 10.7 test
> machines means that we can continue to support osx10.6 at scale without
> needing to buy/rack/recalibrate test and performance results.
>
> 8) Like all other large OS changes, this change would ride the trains.
> Most 10.7 and 10.8 test machines would be reimaged when we make these
> changes live on mozilla-central and try, while we'd leave a few behind.
> The few remaining would be reimaged at each 6-week train migration.
>
> If we move quickly, this reimaging work can be done by IT before they
> all get busy with the 650-Castro -> Evelyn move.
>
> For further details, see Armen's blog
> http://armenzg.blogspot.ca/2013/11/re-thinking-our-mac-os-x-continuous.html.
> To make sure this is not missed, I've cross-posted this to dev.planning,
> dev.platform and also blogged about it on
> http://oduinn.com/blog/2013/11/21/proposed-changes-to-relengs-osx-build-and-test-infrastructure/.
> If you know of anything we have missed, please reply in the dev.planning
> thread.
>
>
> John and Armen.
>
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

RE: Re: Proposed changes to RelEng's OSX build and test infrastructure

Armen Zambrano G.
In reply to this post by John O'Duinn

I want to explicitely disable *all* 10.7 testing (unlike mentioned on point 8) at once by the end of Tuesday as it gains us no clear value (please help me if I'm missing anything wrt to its value on release branches).

I would like to have this extra 10.6 machines verified in preparation for the merge day; this can only be accomplished by taking the whole 10.7 pool.
Taking a partial amount of machines will not accomplish fixing the wait times.Andrew McCreight <[hidden email]> wrote:
> 8) Like all other large OS changes, this change would ride the trains.
> Most 10.7 and 10.8 test machines would be reimaged when we make these
> changes live on mozilla-central and try, while we'd leave a few behind.
> The few remaining would be reimaged at each 6-week train migration.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed changes to RelEng's OSX build and test infrastructure

Mike Hommey
In reply to this post by Armen Zambrano G.
On Fri, Nov 22, 2013 at 08:33:31AM -0800, Armen Zambrano Gasparnian
> This would still have to deal with not getting immediate feedback for
> commits on 10.7 and 10.8 and how to deal with backouts and regression
> hunting.

Note that part is already a problem caused by coalescing.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

RE: Re: Proposed changes to RelEng's OSX build and test infrastructure

Armen Zambrano G.
In reply to this post by John O'Duinn
Yes, that's right.
Moving it to back fill in low demand would make it more noticeable.
We have to have backfilling coverage tools to fix this in the coming year.

Thanks Mike.




Sent from Samsung tabletMike Hommey <[hidden email]> wrote:On Fri, Nov 22, 2013 at 08:33:31AM -0800, Armen Zambrano Gasparnian
> This would still have to deal with not getting immediate feedback for
> commits on 10.7 and 10.8 and how to deal with backouts and regression
> hunting.

Note that part is already a problem caused by coalescing.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning