Intent to uplift to 60: Taskgraph and Release automation changes

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Intent to uplift to 60: Taskgraph and Release automation changes

mozilla

tl;dr To avoid having our release automation diverging we should default to uplifting changes to code involved in release automation that doesn't affect the bits being shipped.


Over the lifetime of ESR52, the release process and surrounding automation for non-ESR releases changed significantly (due to the taskcluster migration). This required releaseduty to maintain very divergent instructions for ESR releases ([1] vs [2]). Given the large amount of changes involved in the final migration of buildbot, this was probably unavoidable. While we don't anticipate major changes there are a number of improvements in the pipeline[3][4][5] that will cause divergence.


We've also ran into issues (for example [1]) due to the build and release automation not being kept up-to-date. We also want to keep our automation tooling up-to-date with security fixes like [7].


We could just up-lift patches that have direct impact on the release process, but as the ESR cycles progresses, it will become more difficult to uplift changes. Instead I propose that we adopt a policy of uplifting relevant changes to ESR60:

- taskgraph code (`taskcluster/taskgraph`)

- release automation tasks (`taskcluster/ci/release-*` mostly)

- testing/mozharness cleanups

As none of the changes impact the built artifacts, they should be low risk, and make any changes that are critical to uplift less risky to uplift. I'm excluding from this proposal changes to

- toolchain version updates

- test configuration

- new build configurations


My initial plan is as follows. It open to suggestions and to change as we get experience with the process.

1. I will prepare a branch with changes to bring ESR60 up-to-date and post that review.

2. I will run a staging release with the changes to verify that the changes won’t cause issues when we go to do a release.

3. After each release cycle, I will repeat the above steps to pick up any changes from the most recent release cycle. (After the initial uplift, this should be fairly smooth)

There may be patches that require uplifting in a more timely manner (security fixes or things that require coordination with out-of-tree changes); they can continue to be handled in an ad-hoc manner. After a few cycles, once this process has worked smoothly, we’ll look at migrating the uplifts to be handled by releaseduty.


I don't have an exhaustive list of changes, but the following is a sampling of changes (some of which have been uplifted, or there are already plans to uplift).


- Mercurial Updates (Bug 1448438)

- Release Notification fixes (Bugs 1459185 and 1459116) [2]

- Converting actions to use hooks (Bug 1415868)

- Pretty release platforms ( https://bugzilla.mozilla.org/show_bug.cgi?id=1456234)

- Fetch tasks (https://bugzilla.mozilla.org/show_bug.cgi?id=1460777)

- Publishing buildhub data (https://bugzilla.mozilla.org/show_bug.cgi?id=1442306)

- Declarative artifacts (https://bugzilla.mozilla.org/show_bug.cgi?id=1462393)


Additionally, I plan to forward port some fixes that were made directly to esr60 in release automation to make this process easier.


-- Tom



[1] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto_58cycle_and_52esr_only.md

[2] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto.md

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1415868

[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1462393

[5] https://bugzilla.mozilla.org/show_bug.cgi?id=shipit-v2

[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1449350

    https://bugzilla.mozilla.org/show_bug.cgi?id=1459391

    https://bugzilla.mozilla.org/show_bug.cgi?id=1446296

[7] https://bugzilla.mozilla.org/show_bug.cgi?id=1385978


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Intent to uplift to 60: Taskgraph and Release automation changes

mozilla
I have prepared a branch with the changes I intend to uplift.  It is available at https://hg.mozilla.org/users/mozilla_hocat.ca/esr60-stage [1]. I intend to push these changes to mozilla-esr60 Thursday morning, unless I head objections before then. In the meantime, I am going to do staging releases with these changes, to verify they work.

-- Tom

[1] This repository has changeset evolution enabled. I intened to keep the state of the repo in-sync with what I am going to uplift.

On Fri, Jun 29, 2018 at 2:31 PM Tom Prince <[hidden email]> wrote:

tl;dr To avoid having our release automation diverging we should default to uplifting changes to code involved in release automation that doesn't affect the bits being shipped.


Over the lifetime of ESR52, the release process and surrounding automation for non-ESR releases changed significantly (due to the taskcluster migration). This required releaseduty to maintain very divergent instructions for ESR releases ([1] vs [2]). Given the large amount of changes involved in the final migration of buildbot, this was probably unavoidable. While we don't anticipate major changes there are a number of improvements in the pipeline[3][4][5] that will cause divergence.


We've also ran into issues (for example [1]) due to the build and release automation not being kept up-to-date. We also want to keep our automation tooling up-to-date with security fixes like [7].


We could just up-lift patches that have direct impact on the release process, but as the ESR cycles progresses, it will become more difficult to uplift changes. Instead I propose that we adopt a policy of uplifting relevant changes to ESR60:

- taskgraph code (`taskcluster/taskgraph`)

- release automation tasks (`taskcluster/ci/release-*` mostly)

- testing/mozharness cleanups

As none of the changes impact the built artifacts, they should be low risk, and make any changes that are critical to uplift less risky to uplift. I'm excluding from this proposal changes to

- toolchain version updates

- test configuration

- new build configurations


My initial plan is as follows. It open to suggestions and to change as we get experience with the process.

1. I will prepare a branch with changes to bring ESR60 up-to-date and post that review.

2. I will run a staging release with the changes to verify that the changes won’t cause issues when we go to do a release.

3. After each release cycle, I will repeat the above steps to pick up any changes from the most recent release cycle. (After the initial uplift, this should be fairly smooth)

There may be patches that require uplifting in a more timely manner (security fixes or things that require coordination with out-of-tree changes); they can continue to be handled in an ad-hoc manner. After a few cycles, once this process has worked smoothly, we’ll look at migrating the uplifts to be handled by releaseduty.


I don't have an exhaustive list of changes, but the following is a sampling of changes (some of which have been uplifted, or there are already plans to uplift).


- Mercurial Updates (Bug 1448438)

- Release Notification fixes (Bugs 1459185 and 1459116) [2]

- Converting actions to use hooks (Bug 1415868)

- Pretty release platforms ( https://bugzilla.mozilla.org/show_bug.cgi?id=1456234)

- Fetch tasks (https://bugzilla.mozilla.org/show_bug.cgi?id=1460777)

- Publishing buildhub data (https://bugzilla.mozilla.org/show_bug.cgi?id=1442306)

- Declarative artifacts (https://bugzilla.mozilla.org/show_bug.cgi?id=1462393)


Additionally, I plan to forward port some fixes that were made directly to esr60 in release automation to make this process easier.


-- Tom



[1] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto_58cycle_and_52esr_only.md

[2] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto.md

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1415868

[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1462393

[5] https://bugzilla.mozilla.org/show_bug.cgi?id=shipit-v2

[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1449350

    https://bugzilla.mozilla.org/show_bug.cgi?id=1459391

    https://bugzilla.mozilla.org/show_bug.cgi?id=1446296

[7] https://bugzilla.mozilla.org/show_bug.cgi?id=1385978


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Intent to uplift to 60: Taskgraph and Release automation changes

mozilla
I've prepared a second round of uplifts that include changes between FIREFOX_NIGHTLY_62_END and FIREFOX_NIGHTLY_64_END, available at https://hg.mozilla.org/users/mozilla_hocat.ca/esr60-stage [1]. I intend to push these changes to mozilla-esr60 Friday, unless I head objections before then.

On Mon, Jul 9, 2018 at 4:19 PM Tom Prince <[hidden email]> wrote:
I have prepared a branch with the changes I intend to uplift.  It is available at https://hg.mozilla.org/users/mozilla_hocat.ca/esr60-stage [1]. I intend to push these changes to mozilla-esr60 Thursday morning, unless I head objections before then. In the meantime, I am going to do staging releases with these changes, to verify they work.

-- Tom

[1] This repository has changeset evolution enabled. I intened to keep the state of the repo in-sync with what I am going to uplift.

On Fri, Jun 29, 2018 at 2:31 PM Tom Prince <[hidden email]> wrote:

tl;dr To avoid having our release automation diverging we should default to uplifting changes to code involved in release automation that doesn't affect the bits being shipped.


Over the lifetime of ESR52, the release process and surrounding automation for non-ESR releases changed significantly (due to the taskcluster migration). This required releaseduty to maintain very divergent instructions for ESR releases ([1] vs [2]). Given the large amount of changes involved in the final migration of buildbot, this was probably unavoidable. While we don't anticipate major changes there are a number of improvements in the pipeline[3][4][5] that will cause divergence.


We've also ran into issues (for example [1]) due to the build and release automation not being kept up-to-date. We also want to keep our automation tooling up-to-date with security fixes like [7].


We could just up-lift patches that have direct impact on the release process, but as the ESR cycles progresses, it will become more difficult to uplift changes. Instead I propose that we adopt a policy of uplifting relevant changes to ESR60:

- taskgraph code (`taskcluster/taskgraph`)

- release automation tasks (`taskcluster/ci/release-*` mostly)

- testing/mozharness cleanups

As none of the changes impact the built artifacts, they should be low risk, and make any changes that are critical to uplift less risky to uplift. I'm excluding from this proposal changes to

- toolchain version updates

- test configuration

- new build configurations


My initial plan is as follows. It open to suggestions and to change as we get experience with the process.

1. I will prepare a branch with changes to bring ESR60 up-to-date and post that review.

2. I will run a staging release with the changes to verify that the changes won’t cause issues when we go to do a release.

3. After each release cycle, I will repeat the above steps to pick up any changes from the most recent release cycle. (After the initial uplift, this should be fairly smooth)

There may be patches that require uplifting in a more timely manner (security fixes or things that require coordination with out-of-tree changes); they can continue to be handled in an ad-hoc manner. After a few cycles, once this process has worked smoothly, we’ll look at migrating the uplifts to be handled by releaseduty.


I don't have an exhaustive list of changes, but the following is a sampling of changes (some of which have been uplifted, or there are already plans to uplift).


- Mercurial Updates (Bug 1448438)

- Release Notification fixes (Bugs 1459185 and 1459116) [2]

- Converting actions to use hooks (Bug 1415868)

- Pretty release platforms ( https://bugzilla.mozilla.org/show_bug.cgi?id=1456234)

- Fetch tasks (https://bugzilla.mozilla.org/show_bug.cgi?id=1460777)

- Publishing buildhub data (https://bugzilla.mozilla.org/show_bug.cgi?id=1442306)

- Declarative artifacts (https://bugzilla.mozilla.org/show_bug.cgi?id=1462393)


Additionally, I plan to forward port some fixes that were made directly to esr60 in release automation to make this process easier.


-- Tom



[1] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto_58cycle_and_52esr_only.md

[2] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto.md

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1415868

[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1462393

[5] https://bugzilla.mozilla.org/show_bug.cgi?id=shipit-v2

[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1449350

    https://bugzilla.mozilla.org/show_bug.cgi?id=1459391

    https://bugzilla.mozilla.org/show_bug.cgi?id=1446296

[7] https://bugzilla.mozilla.org/show_bug.cgi?id=1385978


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Intent to uplift to 60: Taskgraph and Release automation changes

Ryan VanderMeulen-2
Thanks so much for doing this, Tom. Having consistent CI scheduling across our supported branches will make our lives much easier as ESR60 ages over the next year.

On Thu, Nov 8, 2018 at 11:39 AM Tom Prince <[hidden email]> wrote:
I've prepared a second round of uplifts that include changes between FIREFOX_NIGHTLY_62_END and FIREFOX_NIGHTLY_64_END, available at https://hg.mozilla.org/users/mozilla_hocat.ca/esr60-stage [1]. I intend to push these changes to mozilla-esr60 Friday, unless I head objections before then.

On Mon, Jul 9, 2018 at 4:19 PM Tom Prince <[hidden email]> wrote:
I have prepared a branch with the changes I intend to uplift.  It is available at https://hg.mozilla.org/users/mozilla_hocat.ca/esr60-stage [1]. I intend to push these changes to mozilla-esr60 Thursday morning, unless I head objections before then. In the meantime, I am going to do staging releases with these changes, to verify they work.

-- Tom

[1] This repository has changeset evolution enabled. I intened to keep the state of the repo in-sync with what I am going to uplift.

On Fri, Jun 29, 2018 at 2:31 PM Tom Prince <[hidden email]> wrote:

tl;dr To avoid having our release automation diverging we should default to uplifting changes to code involved in release automation that doesn't affect the bits being shipped.


Over the lifetime of ESR52, the release process and surrounding automation for non-ESR releases changed significantly (due to the taskcluster migration). This required releaseduty to maintain very divergent instructions for ESR releases ([1] vs [2]). Given the large amount of changes involved in the final migration of buildbot, this was probably unavoidable. While we don't anticipate major changes there are a number of improvements in the pipeline[3][4][5] that will cause divergence.


We've also ran into issues (for example [1]) due to the build and release automation not being kept up-to-date. We also want to keep our automation tooling up-to-date with security fixes like [7].


We could just up-lift patches that have direct impact on the release process, but as the ESR cycles progresses, it will become more difficult to uplift changes. Instead I propose that we adopt a policy of uplifting relevant changes to ESR60:

- taskgraph code (`taskcluster/taskgraph`)

- release automation tasks (`taskcluster/ci/release-*` mostly)

- testing/mozharness cleanups

As none of the changes impact the built artifacts, they should be low risk, and make any changes that are critical to uplift less risky to uplift. I'm excluding from this proposal changes to

- toolchain version updates

- test configuration

- new build configurations


My initial plan is as follows. It open to suggestions and to change as we get experience with the process.

1. I will prepare a branch with changes to bring ESR60 up-to-date and post that review.

2. I will run a staging release with the changes to verify that the changes won’t cause issues when we go to do a release.

3. After each release cycle, I will repeat the above steps to pick up any changes from the most recent release cycle. (After the initial uplift, this should be fairly smooth)

There may be patches that require uplifting in a more timely manner (security fixes or things that require coordination with out-of-tree changes); they can continue to be handled in an ad-hoc manner. After a few cycles, once this process has worked smoothly, we’ll look at migrating the uplifts to be handled by releaseduty.


I don't have an exhaustive list of changes, but the following is a sampling of changes (some of which have been uplifted, or there are already plans to uplift).


- Mercurial Updates (Bug 1448438)

- Release Notification fixes (Bugs 1459185 and 1459116) [2]

- Converting actions to use hooks (Bug 1415868)

- Pretty release platforms ( https://bugzilla.mozilla.org/show_bug.cgi?id=1456234)

- Fetch tasks (https://bugzilla.mozilla.org/show_bug.cgi?id=1460777)

- Publishing buildhub data (https://bugzilla.mozilla.org/show_bug.cgi?id=1442306)

- Declarative artifacts (https://bugzilla.mozilla.org/show_bug.cgi?id=1462393)


Additionally, I plan to forward port some fixes that were made directly to esr60 in release automation to make this process easier.


-- Tom



[1] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto_58cycle_and_52esr_only.md

[2] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto.md

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1415868

[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1462393

[5] https://bugzilla.mozilla.org/show_bug.cgi?id=shipit-v2

[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1449350

    https://bugzilla.mozilla.org/show_bug.cgi?id=1459391

    https://bugzilla.mozilla.org/show_bug.cgi?id=1446296

[7] https://bugzilla.mozilla.org/show_bug.cgi?id=1385978


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Intent to uplift to 60: Taskgraph and Release automation changes

Jordan Lund
Thanks so much for doing this, Tom. Having consistent CI scheduling across our supported branches will make our lives much easier as ESR60 ages over the next year.

Second this. Really helpful. I've asked Tom to document a rough runbook of how he is doing this so we can reduce the burden of responsibility on him and have someone else contribute next time.

On Thu, Nov 8, 2018 at 8:41 AM Ryan VanderMeulen <[hidden email]> wrote:
Thanks so much for doing this, Tom. Having consistent CI scheduling across our supported branches will make our lives much easier as ESR60 ages over the next year.

On Thu, Nov 8, 2018 at 11:39 AM Tom Prince <[hidden email]> wrote:
I've prepared a second round of uplifts that include changes between FIREFOX_NIGHTLY_62_END and FIREFOX_NIGHTLY_64_END, available at https://hg.mozilla.org/users/mozilla_hocat.ca/esr60-stage [1]. I intend to push these changes to mozilla-esr60 Friday, unless I head objections before then.

On Mon, Jul 9, 2018 at 4:19 PM Tom Prince <[hidden email]> wrote:
I have prepared a branch with the changes I intend to uplift.  It is available at https://hg.mozilla.org/users/mozilla_hocat.ca/esr60-stage [1]. I intend to push these changes to mozilla-esr60 Thursday morning, unless I head objections before then. In the meantime, I am going to do staging releases with these changes, to verify they work.

-- Tom

[1] This repository has changeset evolution enabled. I intened to keep the state of the repo in-sync with what I am going to uplift.

On Fri, Jun 29, 2018 at 2:31 PM Tom Prince <[hidden email]> wrote:

tl;dr To avoid having our release automation diverging we should default to uplifting changes to code involved in release automation that doesn't affect the bits being shipped.


Over the lifetime of ESR52, the release process and surrounding automation for non-ESR releases changed significantly (due to the taskcluster migration). This required releaseduty to maintain very divergent instructions for ESR releases ([1] vs [2]). Given the large amount of changes involved in the final migration of buildbot, this was probably unavoidable. While we don't anticipate major changes there are a number of improvements in the pipeline[3][4][5] that will cause divergence.


We've also ran into issues (for example [1]) due to the build and release automation not being kept up-to-date. We also want to keep our automation tooling up-to-date with security fixes like [7].


We could just up-lift patches that have direct impact on the release process, but as the ESR cycles progresses, it will become more difficult to uplift changes. Instead I propose that we adopt a policy of uplifting relevant changes to ESR60:

- taskgraph code (`taskcluster/taskgraph`)

- release automation tasks (`taskcluster/ci/release-*` mostly)

- testing/mozharness cleanups

As none of the changes impact the built artifacts, they should be low risk, and make any changes that are critical to uplift less risky to uplift. I'm excluding from this proposal changes to

- toolchain version updates

- test configuration

- new build configurations


My initial plan is as follows. It open to suggestions and to change as we get experience with the process.

1. I will prepare a branch with changes to bring ESR60 up-to-date and post that review.

2. I will run a staging release with the changes to verify that the changes won’t cause issues when we go to do a release.

3. After each release cycle, I will repeat the above steps to pick up any changes from the most recent release cycle. (After the initial uplift, this should be fairly smooth)

There may be patches that require uplifting in a more timely manner (security fixes or things that require coordination with out-of-tree changes); they can continue to be handled in an ad-hoc manner. After a few cycles, once this process has worked smoothly, we’ll look at migrating the uplifts to be handled by releaseduty.


I don't have an exhaustive list of changes, but the following is a sampling of changes (some of which have been uplifted, or there are already plans to uplift).


- Mercurial Updates (Bug 1448438)

- Release Notification fixes (Bugs 1459185 and 1459116) [2]

- Converting actions to use hooks (Bug 1415868)

- Pretty release platforms ( https://bugzilla.mozilla.org/show_bug.cgi?id=1456234)

- Fetch tasks (https://bugzilla.mozilla.org/show_bug.cgi?id=1460777)

- Publishing buildhub data (https://bugzilla.mozilla.org/show_bug.cgi?id=1442306)

- Declarative artifacts (https://bugzilla.mozilla.org/show_bug.cgi?id=1462393)


Additionally, I plan to forward port some fixes that were made directly to esr60 in release automation to make this process easier.


-- Tom



[1] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto_58cycle_and_52esr_only.md

[2] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto.md

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1415868

[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1462393

[5] https://bugzilla.mozilla.org/show_bug.cgi?id=shipit-v2

[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1449350

    https://bugzilla.mozilla.org/show_bug.cgi?id=1459391

    https://bugzilla.mozilla.org/show_bug.cgi?id=1446296

[7] https://bugzilla.mozilla.org/show_bug.cgi?id=1385978


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds