Optimizing what runs on which push

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Optimizing what runs on which push

Dustin Mitchell
I wrote up
  https://docs.google.com/document/d/1kEEUC3ETCIerOxYDoRRZSEFspunJgU331CX2wwkFZ_8/edit
It's not a particularly ambitious text, aiming to capture a consensus
rather than propose something new. It's also worth repeating that this
is not a proposal for a big new project, but a design strategy to
guide work that is occurring anyway!

I omitted discussion of AFFECTS, since it's more detailed than a
"design strategy". I did include some of the important factors that
we've discussed here, and focused the goals a little more.

Please have a look and add comments!

Dustin
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Optimizing what runs on which push

Ehsan Akhgari
I haven't really read the thread in too much detail so I apologize if
what I'll say below is silly or covered elsewhere, but since there seems
to be a proposal around replacing the try syntax with machines figuring
out what jobs are needed for a push (presumably based on what changed in
the push), here is a use case that may not be satisfied if we do down
that route:

Developer pushes to try, runs N jobs, M << N jobs fail, developer
realizes her mistakes, fixes, wants to push to try but because of the
nature of the fix only a run on those M jobs is now needed, but if the
machines want to figure things out, they may pick a number K where N > K
 >= M jobs to run, so developer needs to politely ask the machines to
get out of her way and let her do her job.  :-)

Has this been taken into account?

Thanks,

Ehsan


On 06/02/2017 04:30 PM, Dustin Mitchell wrote:

> I wrote up
>    https://docs.google.com/document/d/1kEEUC3ETCIerOxYDoRRZSEFspunJgU331CX2wwkFZ_8/edit
> It's not a particularly ambitious text, aiming to capture a consensus
> rather than propose something new. It's also worth repeating that this
> is not a proposal for a big new project, but a design strategy to
> guide work that is occurring anyway!
>
> I omitted discussion of AFFECTS, since it's more detailed than a
> "design strategy". I did include some of the important factors that
> we've discussed here, and focused the goals a little more.
>
> Please have a look and add comments!
>
> Dustin
> _______________________________________________
> dev-builds mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-builds

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Optimizing what runs on which push

Dustin Mitchell
It could go two ways:

1. "Just try it" again and see what breaks. This runs extra jobs, but
has the benefit of finding any accidental bustage added in the fix; or
2. "try this" listing the M jobs that failed (or some simpler-to-write
superset like "all chunks of OS X debug mochitest-chrome").

It's a common scenario, so maybe we could add a way to generate a list
of all failed jobs in a push.  Either a button in Treeherder, or
something like `./mach try --all-failures-from <revision>`.  In any
case, lots of opportunity there for small incremental hacks to improve
usability for everyone within this design.

Dustin

2017-06-02 18:19 GMT-04:00 Ehsan Akhgari <[hidden email]>:

> I haven't really read the thread in too much detail so I apologize if what
> I'll say below is silly or covered elsewhere, but since there seems to be a
> proposal around replacing the try syntax with machines figuring out what
> jobs are needed for a push (presumably based on what changed in the push),
> here is a use case that may not be satisfied if we do down that route:
>
> Developer pushes to try, runs N jobs, M << N jobs fail, developer realizes
> her mistakes, fixes, wants to push to try but because of the nature of the
> fix only a run on those M jobs is now needed, but if the machines want to
> figure things out, they may pick a number K where N > K >= M jobs to run, so
> developer needs to politely ask the machines to get out of her way and let
> her do her job.  :-)
>
> Has this been taken into account?
>
> Thanks,
>
> Ehsan
>
>
>
> On 06/02/2017 04:30 PM, Dustin Mitchell wrote:
>>
>> I wrote up
>>
>> https://docs.google.com/document/d/1kEEUC3ETCIerOxYDoRRZSEFspunJgU331CX2wwkFZ_8/edit
>> It's not a particularly ambitious text, aiming to capture a consensus
>> rather than propose something new. It's also worth repeating that this
>> is not a proposal for a big new project, but a design strategy to
>> guide work that is occurring anyway!
>>
>> I omitted discussion of AFFECTS, since it's more detailed than a
>> "design strategy". I did include some of the important factors that
>> we've discussed here, and focused the goals a little more.
>>
>> Please have a look and add comments!
>>
>> Dustin
>> _______________________________________________
>> dev-builds mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-builds
>
>
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Optimizing what runs on which push

Ehsan Akhgari
On 06/02/2017 06:27 PM, Dustin Mitchell wrote:
> It could go two ways:
>
> 1. "Just try it" again and see what breaks. This runs extra jobs, but
> has the benefit of finding any accidental bustage added in the fix; or
At the cost of forcing you filter through which oranges are your fault
and which ones are intermittents...  (FWIW I sometimes spend a few
minutes to contstruct a long trychooser syntax using the HTML chooser UI
right now to avoid tens of minutes of time later sorting through oranges!)

> 2. "try this" listing the M jobs that failed (or some simpler-to-write
> superset like "all chunks of OS X debug mochitest-chrome").
>
> It's a common scenario, so maybe we could add a way to generate a list
> of all failed jobs in a push.  Either a button in Treeherder, or
> something like `./mach try --all-failures-from <revision>`.  In any
> case, lots of opportunity there for small incremental hacks to improve
> usability for everyone within this design.
Yeah, agreed.  (I'm drooling over the --all-failures-from idea!) Thanks
for thinking about this use case!

> Dustin
>
> 2017-06-02 18:19 GMT-04:00 Ehsan Akhgari <[hidden email]>:
>> I haven't really read the thread in too much detail so I apologize if what
>> I'll say below is silly or covered elsewhere, but since there seems to be a
>> proposal around replacing the try syntax with machines figuring out what
>> jobs are needed for a push (presumably based on what changed in the push),
>> here is a use case that may not be satisfied if we do down that route:
>>
>> Developer pushes to try, runs N jobs, M << N jobs fail, developer realizes
>> her mistakes, fixes, wants to push to try but because of the nature of the
>> fix only a run on those M jobs is now needed, but if the machines want to
>> figure things out, they may pick a number K where N > K >= M jobs to run, so
>> developer needs to politely ask the machines to get out of her way and let
>> her do her job.  :-)
>>
>> Has this been taken into account?
>>
>> Thanks,
>>
>> Ehsan
>>
>>
>>
>> On 06/02/2017 04:30 PM, Dustin Mitchell wrote:
>>> I wrote up
>>>
>>> https://docs.google.com/document/d/1kEEUC3ETCIerOxYDoRRZSEFspunJgU331CX2wwkFZ_8/edit
>>> It's not a particularly ambitious text, aiming to capture a consensus
>>> rather than propose something new. It's also worth repeating that this
>>> is not a proposal for a big new project, but a design strategy to
>>> guide work that is occurring anyway!
>>>
>>> I omitted discussion of AFFECTS, since it's more detailed than a
>>> "design strategy". I did include some of the important factors that
>>> we've discussed here, and focused the goals a little more.
>>>
>>> Please have a look and add comments!
>>>
>>> Dustin
>>> _______________________________________________
>>> dev-builds mailing list
>>> [hidden email]
>>> https://lists.mozilla.org/listinfo/dev-builds
>>

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Optimizing what runs on which push

Dustin Mitchell
2017-06-02 18:40 GMT-04:00 Ehsan Akhgari <[hidden email]>:

> On 06/02/2017 06:27 PM, Dustin Mitchell wrote:
>>
>> It could go two ways:
>>
>> 1. "Just try it" again and see what breaks. This runs extra jobs, but
>> has the benefit of finding any accidental bustage added in the fix; or
>
> At the cost of forcing you filter through which oranges are your fault and
> which ones are intermittents...  (FWIW I sometimes spend a few minutes to
> contstruct a long trychooser syntax using the HTML chooser UI right now to
> avoid tens of minutes of time later sorting through oranges!)

Yeah, intermittents suck. I'll leave the plum project of fixing that
to others :)

Dustin
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
12