reducing high try macosx pending counts

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

reducing high try macosx pending counts

Kim Moir-2
You may have noticed that the time to wait for macosx test results on try
has been very long (>1day) this week.

[tracking] macosx test load is unsustainable
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625

There are several factors that contributed to this situation.
1) Additional macosx stylo tests were enabled on trunk which added
significant test load to a system already running at capacity
2) These tests were enabled before some optimizations were in place to
reduce the number of tests run per push
3) Since jobs on try have a lower priority than jobs on trunk, the trunk
jobs continue to consume the the mac test machine capacity before it can be
used by try jobs

We have taken the following steps to address this problem
1) I have disabled the mac stylo tests on m-i to reduce load (
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625)
2)  The optimizations have been enabled for autoland and m-i branches so
all the tests are not run on every push for the new stylo tests (
https://bugzilla.mozilla.org/show_bug.cgi?id=1386405)
3) Patches are in progress to disable some non-e10s tests to further reduce
load (https://bugzilla.mozilla.org/show_bug.cgi?id=1386689)

We will continue to monitor the situation and implement additional measures
to allow macosx test runs on try to complete in a more reasonable interval.

Please ping us in #releng or follow the tracking bug if you have questions
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625

Kim
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: reducing high try macosx pending counts

Bobby Holley-2
Thanks to everyone involved for accommodating the extra load from the stylo
side - it's a necessary part of our ramp-up to shipping. Luckily, we should
only be running the two configurations side-by-side for a month or two,
after which point load should drop back down to normal levels.

Thanks Kim for staying on top of this and coming up with ways to keep it
manageable!
bholley

On Wed, Aug 2, 2017 at 2:30 PM, Kim Moir <[hidden email]> wrote:

> You may have noticed that the time to wait for macosx test results on try
> has been very long (>1day) this week.
>
> [tracking] macosx test load is unsustainable
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386625
>
> There are several factors that contributed to this situation.
> 1) Additional macosx stylo tests were enabled on trunk which added
> significant test load to a system already running at capacity
> 2) These tests were enabled before some optimizations were in place to
> reduce the number of tests run per push
> 3) Since jobs on try have a lower priority than jobs on trunk, the trunk
> jobs continue to consume the the mac test machine capacity before it can be
> used by try jobs
>
> We have taken the following steps to address this problem
> 1) I have disabled the mac stylo tests on m-i to reduce load (
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386625)
> 2)  The optimizations have been enabled for autoland and m-i branches so
> all the tests are not run on every push for the new stylo tests (
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386405)
> 3) Patches are in progress to disable some non-e10s tests to further reduce
> load (https://bugzilla.mozilla.org/show_bug.cgi?id=1386689)
>
> We will continue to monitor the situation and implement additional measures
> to allow macosx test runs on try to complete in a more reasonable interval.
>
> Please ping us in #releng or follow the tracking bug if you have questions
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386625
>
> Kim
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning