Tool for generation of regression-range links

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Tool for generation of regression-range links

Benjamin Smedberg
tl;dr: I have a tool for generating regression-range links from
buildids. http://bsmedberg.github.io/firefox-regression-range-finder/

Often times when we're investigating regressions (crashes, etc), we have
the build ID of the nightly where the regression started. But it's not
the easiest thing in the world to turn that into an actual pushlog link
which lists the commits in the regression range.

To help fix this, I've created a tool that lets you paste in a buildid
or two and automatically figures out the proper regression range link,
and also links you to the nightly build directories.

This tool is pretty hacky: it loads http://ftp.mozilla.org directory
listings to find builds, and then loads the .txt files describing the
revisions involved. If we ever changed webserver for ftp.mozilla.org the
tool would likely break. But since it's something I do at least once a
week and I know other QA people do this even more regularly, I figured
it was worth automating.

Right now the tool uses ES6 features `let` and `for...of` and so it only
works in Firefox and not in Chrome. I figured for this audience that
wouldn't be a problem.

--BDS

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Tool for generation of regression-range links

Mike Hommey
On Mon, Oct 13, 2014 at 03:17:44PM -0400, Benjamin Smedberg wrote:
> tl;dr: I have a tool for generating regression-range links from buildids.
> http://bsmedberg.github.io/firefox-regression-range-finder/
>
> Often times when we're investigating regressions (crashes, etc), we have the
> build ID of the nightly where the regression started.

Changesets are in about:buildconfig aren't they? Why not (have people) take
them from there instead of going through hoops on the ftp server?

Mike
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Tool for generation of regression-range links

Benjamin Smedberg

On 10/13/2014 6:03 PM, Mike Hommey wrote:
> On Mon, Oct 13, 2014 at 03:17:44PM -0400, Benjamin Smedberg wrote:
>> tl;dr: I have a tool for generating regression-range links from buildids.
>> http://bsmedberg.github.io/firefox-regression-range-finder/
>>
>> Often times when we're investigating regressions (crashes, etc), we have the
>> build ID of the nightly where the regression started.
> Changesets are in about:buildconfig aren't they? Why not (have people) take
> them from there instead of going through hoops on the ftp server?

I'm almost always generating these from data (crash-stats or telemetry)
and not from the builds themselves. Or from what people told me in bug
comments. Having to install the builds would be unfortunate in my case
at least.

--BDS

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Tool for generation of regression-range links

L. David Baron
In reply to this post by Benjamin Smedberg
On Monday 2014-10-13 15:17 -0400, Benjamin Smedberg wrote:
> it's not the easiest thing in the world to turn that into an actual
> pushlog link which lists the commits in the regression range.

Just a reminder:  I expect many of you know this already, but I
think it's worth repeating:  if you're using pushlog links for
regression ranges, it makes a difference which repository's pushlog
you link to, since changesets can reach different repositories in
different orders, and the pushlog shows you the order the changesets
arrived in that repository.  You need to use the repository that the
builds you are using were built from.

For more details, see:
https://groups.google.com/d/msg/mozilla.dev.planning/vwZtxrQ1tUI/PfuvXwMj19sJ
and perhaps the rest of the thread:
https://groups.google.com/d/topic/mozilla.dev.planning/vwZtxrQ1tUI/discussion

-David

--
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                          https://www.mozilla.org/   𝄂
             Before I built a wall I'd ask to know
             What I was walling in or walling out,
             And to whom I was like to give offense.
               - Robert Frost, Mending Wall (1914)

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Tool for generation of regression-range links

Gregory Szorc-3
In reply to this post by Benjamin Smedberg
On 10/13/14 12:17 PM, Benjamin Smedberg wrote:

> tl;dr: I have a tool for generating regression-range links from
> buildids. http://bsmedberg.github.io/firefox-regression-range-finder/
>
> Often times when we're investigating regressions (crashes, etc), we have
> the build ID of the nightly where the regression started. But it's not
> the easiest thing in the world to turn that into an actual pushlog link
> which lists the commits in the regression range.
>
> To help fix this, I've created a tool that lets you paste in a buildid
> or two and automatically figures out the proper regression range link,
> and also links you to the nightly build directories.
>
> This tool is pretty hacky: it loads http://ftp.mozilla.org directory
> listings to find builds, and then loads the .txt files describing the
> revisions involved. If we ever changed webserver for ftp.mozilla.org the
> tool would likely break. But since it's something I do at least once a
> week and I know other QA people do this even more regularly, I figured
> it was worth automating.
>
> Right now the tool uses ES6 features `let` and `for...of` and so it only
> works in Firefox and not in Chrome. I figured for this audience that
> wouldn't be a problem.

FWIW, Robert Helmer wrote a tool for scraping the FTP server, storing
the relevant bits in a database, and exposing the result via HTTP+JSON.
It used to run on Mozilla's Stackato deployment, but he turned off his
instance for reasons I can't recall. The service was really useful:
someone should look into deploying this again.

https://github.com/rhelmer/releases-api/
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Tool for generation of regression-range links

Robert Helmer-4
On Tue, Oct 14, 2014 at 8:48 AM, Gregory Szorc <[hidden email]> wrote:

> On 10/13/14 12:17 PM, Benjamin Smedberg wrote:
>>
>> tl;dr: I have a tool for generating regression-range links from
>> buildids. http://bsmedberg.github.io/firefox-regression-range-finder/
>>
>> Often times when we're investigating regressions (crashes, etc), we have
>> the build ID of the nightly where the regression started. But it's not
>> the easiest thing in the world to turn that into an actual pushlog link
>> which lists the commits in the regression range.
>>
>> To help fix this, I've created a tool that lets you paste in a buildid
>> or two and automatically figures out the proper regression range link,
>> and also links you to the nightly build directories.
>>
>> This tool is pretty hacky: it loads http://ftp.mozilla.org directory
>> listings to find builds, and then loads the .txt files describing the
>> revisions involved. If we ever changed webserver for ftp.mozilla.org the
>> tool would likely break. But since it's something I do at least once a
>> week and I know other QA people do this even more regularly, I figured
>> it was worth automating.
>>
>> Right now the tool uses ES6 features `let` and `for...of` and so it only
>> works in Firefox and not in Chrome. I figured for this audience that
>> wouldn't be a problem.
>
>
> FWIW, Robert Helmer wrote a tool for scraping the FTP server, storing the
> relevant bits in a database, and exposing the result via HTTP+JSON. It used
> to run on Mozilla's Stackato deployment, but he turned off his instance for
> reasons I can't recall. The service was really useful: someone should look
> into deploying this again.
>
> https://github.com/rhelmer/releases-api/


The code for this was pulled out of Socorro and modified to run as a
standalone service, the reasons I decided not to go forward at that
time were:

1) the code in Socorro was undergoing big changes and it was hard to
keep in sync
2) I'd rather work with releng to expose this in a more direct way

My plan at the time was to move Socorro over to using this service too
(so point 1 would be moot) but I didn't have time to pull it off right
then.

I am ambivalent about the second point, we could probably put up a
REST API that scrapes FTP now and change the way it works internally
later without breaking the API.
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality