Quantcast

Using Firefox to test the Visual C++ compiler

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Using Firefox to test the Visual C++ compiler

lgratian
Hello,

*I posted this on multiple mailing lists, I'm not sure which is the right place for these questions.

I'm a developer on the Microsoft Visual C++ compiler (code optimizer) and I'm looking into extending our test suite with popular open-source projects. This helps  us to find bugs earlier, ensuring that these projects are not broken by frontend or backend changes. It also helps the projects themselves, by making upgrades to the never compiler easier - ideally without any problems.

I have several questions about building Firefox on Windows and especially about running the correctness tests:

1. What is the latest version of Visual Studio that should be used? I tried first VS2015 Update 3 and there seems to be a C++ error about "constexpr". Next I tried Update 2 and that one finishes the build.

2. What are the tests that should be run to test correctness? The idea here is to ensure that new optimizations don't introduce new bugs, which should be exposed by new test failures in Firefox. I looked over the QA Automated testing page and there seem to be a lot of different test suites - running all is probably not feasible. Is there a subset that is run when a checkin is done, for example?

3. What Windows OS do you use for testing? Window 10, Windows 8, etc? Any other configuration work needed, such as disabling the antivirus/firewall?

After the build completed, I tried to run a few of the tests and there seem to be many failures even with a debug build. I assume that the checked in tests are supposed to pass, so it's likely a Windows config problem or failure to access extra files from the Web.

For example, ./mach reftests had about 700 failures. ./mach gtest, jsapi-tests, jsbrowsertest had none. The tests were run under Windows 10 Anniversary update with the AV disabled.

Thanks,
Gratian Lup
Microsoft Visual C++ team
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Firefox to test the Visual C++ compiler

Ted Mielczarek-2
On Wed, Nov 16, 2016, at 03:27 AM, [hidden email] wrote:
> Hello,
>
> *I posted this on multiple mailing lists, I'm not sure which is the right
> place for these questions.

Hi Gratian,

Finding the right communication channel always feels like a problem at
Mozilla. :) dev-builds is a fine venue for this discussion.

> I'm a developer on the Microsoft Visual C++ compiler (code optimizer) and
> I'm looking into extending our test suite with popular open-source
> projects. This helps  us to find bugs earlier, ensuring that these
> projects are not broken by frontend or backend changes. It also helps the
> projects themselves, by making upgrades to the never compiler easier -
> ideally without any problems.

This sounds great! We always struggle with updating to a new toolchain,
whether with compiler bugs or issues in our codebase surfaced by
correctness fixes in the compiler.  For example, here's the dependency
tree of bugs we fixed that are marked as blocking our "Support Visual
C++ 2015" bug:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1119082&hide_resolved=0

> I have several questions about building Firefox on Windows and especially
> about running the correctness tests:
>
> 1. What is the latest version of Visual Studio that should be used? I
> tried first VS2015 Update 3 and there seems to be a C++ error about
> "constexpr". Next I tried Update 2 and that one finishes the build.

We are using VS2015u3 for our builds in automation currently, so it
should work:
https://dxr.mozilla.org/mozilla-central/rev/79feeed4293336089590320a9f30a813fade8e3c/browser/config/tooltool-manifests/win32/releng.manifest#32

Are you building from the mozilla-central repository (our development
repository) or something else?

> 2. What are the tests that should be run to test correctness? The idea
> here is to ensure that new optimizations don't introduce new bugs, which
> should be exposed by new test failures in Firefox. I looked over the QA
> Automated testing page and there seem to be a lot of different test
> suites - running all is probably not feasible. Is there a subset that is
> run when a checkin is done, for example?

We run a lot of tests on every Firefox checkin. Here are all the Windows
build and test jobs from an arbitrary recent mozilla-central revision:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=f8ba9c9b401f57b0047ddd6932cb830190865b38&filter-searchStr=Windows

The secret decoder key for all those letters is here (I'm not sure if
it's fully kept up-to-date):
https://treeherder.mozilla.org/userguide.html

We do have an elaborate system that tries not to run every single test
on every single checkin, but toolchain changes have a way of showing up
as test failures in very strange places, especially with things like our
PGO builds. (Ask me sometime about debugging an issue that showed up
when I turned on PGO in the Firefox 3 release cycle...) That being said,
if you were going to run any tests I would probably recommend some of
the Mochitest test suites, since they launch a browser and run a lot of
HTML+JS, which exercises a good amount of our code, and the JIT tests,
which exercise the JavaScript engine pretty heavily. The JS team tends
to write more complicated C++ than other groups within Mozilla, so they
also hit a lot of compiler issues.

> 3. What Windows OS do you use for testing? Window 10, Windows 8, etc? Any
> other configuration work needed, such as disabling the
> antivirus/firewall?

I will have to confirm specifics, but we currently run tests on Windows
7 (x86) and Windows 8 (x64), with just a small handful of tests running
on Windows 10 (x64). I'm fairly certain we do a lot of configuration on
our test machines, but I don't have the details handy. We generally
attempt to restrict network access from our test suites so that network
issues do not cause test failures, but we do run multiple servers on
localhost during testing, so at the very least that needs to be enabled.

Note also that we do our builds and tests on separate machines in
automation. Running the tests from a build on the same machine should
work fine, that's what developers do locally, it just doesn't scale well
in automation and we also want to do our builds on Server and run our
tests on a desktop OS.

> After the build completed, I tried to run a few of the tests and there
> seem to be many failures even with a debug build. I assume that the
> checked in tests are supposed to pass, so it's likely a Windows config
> problem or failure to access extra files from the Web.
>
> For example, ./mach reftests had about 700 failures. ./mach gtest,
> jsapi-tests, jsbrowsertest had none. The tests were run under Windows 10
> Anniversary update with the AV disabled.

Many tests are extremely sensitive to the OS environment, unfortunately.
I don't think we have reftests running in Windows 10, so there's likely
to be issues there due to test expectations. You might have better luck
with them on a Windows 7 machine, which we do run in automation. Some
test suites are not as sensitive. In your case maybe you'd get the best
mileage out of running the JIT tests for now? I don't think there's a
standalone mach command to run them, sadly, but I think `./mach build
js/src/check-jit-test` will run them. You will probably need to add
`ac_add_options --enable-js-shell` to your mozconfig to make that work.

I'd love to get you sorted out, if only for the selfish reasons that you
can fix issues before new versions of VC get released. :)

Regards,
-Ted
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Firefox to test the Visual C++ compiler

Ted Mielczarek-2
In reply to this post by lgratian
Gratian,

One of my coworkers reminded me of something that might be an option for
you--we have scripts that would allow you to provide a Firefox build
that you generated (at a publicly accessible URL) and trigger test jobs
on that build in our CI infrastructure. If that's something that sounds
useful to you we can definitely make that happen.

You'd have to produce a Firefox build, run the `mach package` and `mach
package-tests` targets, upload a few zip files from the $objdir/dist
directory to somewhere accessible via public HTTP and then run a Python
script to schedule test jobs against those files.

-Ted
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Firefox to test the Visual C++ compiler

lgratian
In reply to this post by lgratian
On Wednesday, November 16, 2016 at 5:23:58 AM UTC-8, Ted Mielczarek wrote:

> Gratian,
>
> One of my coworkers reminded me of something that might be an option for
> you--we have scripts that would allow you to provide a Firefox build
> that you generated (at a publicly accessible URL) and trigger test jobs
> on that build in our CI infrastructure. If that's something that sounds
> useful to you we can definitely make that happen.
>
> You'd have to produce a Firefox build, run the `mach package` and `mach
> package-tests` targets, upload a few zip files from the $objdir/dist
> directory to somewhere accessible via public HTTP and then run a Python
> script to schedule test jobs against those files.
>
> -Ted

Hi Ted,

Thanks a lot for your help!
Using Windows 7 or 8 to do the tests should be fine - I actually don't see any reason it shouldn't be OK to also do other builds and tests on them when not used for Firefox.

The idea of testing on your infrastructure is tempting, but probably would consume too many resources, since this new testing system is intended to be used both overnight against the latest good compiler build, but also by every developer on its own while working on new features - this would be quite a lot of people. Doing a test build now would still be a good idea, though, at least to see  if everything passes in the right environment.

I have a few more questions about running the tests:

1. How exactly should the build artifacts be copied to the test machine? Something like ./mach package? After copying over, running the tests with ./mach is going to pick the binaries after copying, or is some ./mach "unpack" needed? I assume the entire mozilla-central enlistment is also required.

2. Can I see on Treeherder the exact command line that was used to launch the test suite? I looked over the log files and didn't find anything like that.

I'm going to try again with Update 3, I might have used instead the most recent build, which can indeed show some new errors, the frontend team do a lot of changes.

Thanks,
Gratian
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Firefox to test the Visual C++ compiler

Gregory Szorc-3
On Wed, Nov 16, 2016 at 12:25 PM, Gratian Lup <[hidden email]> wrote:
On Wednesday, November 16, 2016 at 5:23:58 AM UTC-8, Ted Mielczarek wrote:
> Gratian,
>
> One of my coworkers reminded me of something that might be an option for
> you--we have scripts that would allow you to provide a Firefox build
> that you generated (at a publicly accessible URL) and trigger test jobs
> on that build in our CI infrastructure. If that's something that sounds
> useful to you we can definitely make that happen.
>
> You'd have to produce a Firefox build, run the `mach package` and `mach
> package-tests` targets, upload a few zip files from the $objdir/dist
> directory to somewhere accessible via public HTTP and then run a Python
> script to schedule test jobs against those files.
>
> -Ted

Hi Ted,

Thanks a lot for your help!
Using Windows 7 or 8 to do the tests should be fine - I actually don't see any reason it shouldn't be OK to also do other builds and tests on them when not used for Firefox.

The idea of testing on your infrastructure is tempting, but probably would consume too many resources, since this new testing system is intended to be used both overnight against the latest good compiler build, but also by every developer on its own while working on new features - this would be quite a lot of people. Doing a test build now would still be a good idea, though, at least to see  if everything passes in the right environment.

Something Mozilla has been wanting to do for a while is stand up "nightly" builds using the latest, unreleased toolchain from upstream. This would help us identify compiler compatibility issues early. And it may even help compiler toolchain developers find unwanted regressions.

In theory, Microsoft developers could gain access to our "Try" server, which allows you to push changes made to Firefox in version control (including its automation config) and run the results on our infrastructure. We have a file in the repository referencing a zip file of the Visual Studio toolchain to use. So, it would be possible for someone at Microsoft to create a zip file of an unreleased VC++ toolchain and have our automation run against that. Of course, there are scaling and privacy issues to work out. (I'm guessing Microsoft LCA will have an opinion on mostly-in-the-public-domain Mozilla infrastructure accessing pre-release Microsoft software.)

 

I have a few more questions about running the tests:

1. How exactly should the build artifacts be copied to the test machine? Something like ./mach package? After copying over, running the tests with ./mach is going to pick the binaries after copying, or is some ./mach "unpack" needed? I assume the entire mozilla-central enlistment is also required.

`mach package` will get you a standalone archive or installer suitable for deploying to test machines.

For historical reasons, not all `mach *test` commands integrate really well with pre-built archives/installers. There is a whole other layer that deals with running tests in automation. That being said, some `mach` commands do accept an argument with an alternate path to a binary to use. e.g. `mach reftest --appname <path>` and `mach mochitest --appname <path>`. Running tests from arbitrary binaries is definitely something we'd like to make a bit easier. So if there's a feature you'd like, don't hesitate to ask.


2. Can I see on Treeherder the exact command line that was used to launch the test suite? I looked over the log files and didn't find anything like that.

In some tests. e.g. in https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1479739510/mozilla-central_win7_vm_test-web-platform-tests-1-bm129-tests1-windows-build17.txt.gz the main command is:

  c:/mozilla-build/python27/python' '-u' 'scripts/scripts/web_platform_tests.py' '--cfg' 'web_platform_tests/prod_config_windows.py' '--test-type=testharness' '--total-chunks' '5' '--this-chunk' '1' '--blob-upload-branch' 'mozilla-central' '--download-symbols' 'ondemand'

This is invoking a tool called "mozharness" which is our "execute in automation" layer. This tool performs tons of setup work, such as obtaining a version control checkout, creating Python virtualenvs, etc. Once that's all in place, it executes a test runner. e.g.

08:22:59 INFO - Calling ['c:\\slave\\test\\build\\venv\\Scripts\\python', '-u', 'c:\\slave\\test\\build\\tests\\web-platform\\runtests.py', '--log-raw=-', '--log-raw=c:\\slave\\test\\build\\blobber_upload_dir\\wpt_raw.log', '--log-errorsummary=c:\\slave\\test\\build\\blobber_upload_dir\\wpt_errorsummary.log', '--binary=c:\\slave\\test\\build\\application\\firefox\\firefox.exe', '--symbols-path=https://queue.taskcluster.net/v1/task/G49XlpNvSWK9pkFLL2pT_A/artifacts/public/build/firefox-53.0a1.en-US.win32.crashreporter-symbols.zip', '--stackwalk-binary=c:\\slave\\test\\build\\win32-minidump_stackwalk.exe', '--stackfix-dir=c:\\slave\\test\\build\\tests\\bin', '--test-type=testharness', '--disable-e10s', '--total-chunks=5', '--this-chunk=1', '--prefs-root=c:\\slave\\test\\build\\tests\\web-platform/prefs', '--processes=1', '--config=c:\\slave\\test\\build\\tests\\web-platform/wptrunner.ini', '--ca-cert-path=c:\\slave\\test\\build\\tests\\web-platform/certs/cacert.pem', '--host-key-path=c:\\slave\\test\\build\\tests\\web-platform/certs/web-platform.test.key', '--host-cert-path=c:\\slave\\test\\build\\tests\\web-platform/certs/web-platform.test.pem', '--certutil-binary=c:\\slave\\test\\build\\tests/bin/certutil'] with output_timeout 1000
There is a lot of complexity here. Most of it unique to our automation infrastructure. I recommend you avoid mozharness completely and try to execute things with "mach." (This is the direction we're trying to take things.) Again, if you need features to make that easier, please ask and we'll see what we can do.


I'm going to try again with Update 3, I might have used instead the most recent build, which can indeed show some new errors, the frontend team do a lot of changes.

Thanks,
Gratian
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Firefox to test the Visual C++ compiler

Jonathan Wilson-7
In reply to this post by lgratian
> issues to work out. (I'm guessing Microsoft LCA will have an opinion on
> mostly-in-the-public-domain Mozilla infrastructure accessing pre-release
> Microsoft software.)
Microsoft does release "nightly" compiler builds these days
https://blogs.msdn.microsoft.com/vcblog/2016/04/26/stay-up-to-date-with-the-visual-c-tools-on-nuget/
so maybe it wouldn't be so much of a problem.

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Firefox to test the Visual C++ compiler

lgratian
In reply to this post by Gregory Szorc-3
Hi,

Thanks a lot for your help! I think doing the testing on our machines is a better approach overall, even from an engineering perspective - in case of a failure you can just grab the machine and start debugging, recompiling, etc. The legal work, if possible at all, would be quite complicated too. 

This new approach of testing benefits both us and the tested projects. What we are doing:
- If the failure is in the compiler frontend, it is either a bug or a problem in the code itself. There were a few cases when the code had problems and we informed and helped those project to solve it - this would also be the case with Firefox.
- If the failure is in the compiler backend/optimizer, fix the bug. This is pretty much always a real bug, unless the source code triggers some cases of undefined behavior - we didn't find such a case yet.

I tried running the tests on Windows 7 this time, with the AV and firewall disabled, just to be sure they don't interfere. When using the mach commands, there are still quite a lot of failures, so I assume that something is still not configured right. I was curious if it's different on Linux, but tests are also failing there (I used Xubuntu 16.10 with KDE). 

Most failures are in the mochitest, reftest and wpt suites. If it's not possible to make all the tests pass, a good enough approach is to establish a baseline of tests that are known to fail using a debug/non-optimized build and consider a real failure only a failed test not in that set. We already do this for a few other large projects and seems like a good compromise.

Is there a way to skip over the tests that are known to fail? With Gtest this is easy using the --gtest_filter flag. If it's not possible, then a script is needed to parse the results and ignore those known failures. 

Thanks,
Gratian

Here are some results from the tests running on Windows 7. You can find some log files here: https://1drv.ms/f/s!AmGHUgGfTN19hjuFKlOeZ7VtpUsd

o Tests without failures:
mach gtest
mach marionette-test
mach firefox-ui-functional
mach jsapi-tests
mach jsbrowsertest

o Spidermonkey tests
mach check-spidermonkey doesn't seem to work because js.exe is not built by default. To test it I built the js folder and followed the Spidermonkey test instructions to run the JS and JIT tests - both pass without failures.

o Tests with failures
- mach mochitest

Part 1

1 INFO Passed:  675002

2 INFO Failed:  23

3 INFO Todo:    1586

4 INFO Mode:    e10s

Part 2

1 INFO Passed:  676509

2 INFO Failed:  16

3 INFO Todo:    1592

4 INFO Mode:    e10s


Example of failed test: dom/base/test/test_noAudioNotificationOnMutedElement.html | Test timed out.


- mach reftest

REFTEST INFO | Successful: 13436 (13416 pass, 20 load only)

REFTEST INFO | Unexpected: 114 (113 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 1 failed load, 0 exception)

REFTEST INFO | Known problems: 700 (285 known fail, 0 known asserts, 69 random, 346 skipped, 0 slow)

Example of failed test: c:/firefox/mozilla-central/layout/reftests/bugs/321402-4.xul == file:///c:/firefox/mozilla-central/layout/reftests/bugs/321402-4-ref.xul | image comparison, max difference: 32, number of differing pixels: 1


The images look identical to me, but 1 pixel is supposedly different - I'm wondering why the test fails if the max. acceptable diff is 32. Some other cases I look into also had "identical" images.


- mach web-platform-test

- mach crash-test



On Mon, Nov 21, 2016 at 9:54 AM, Gregory Szorc <[hidden email]> wrote:
On Wed, Nov 16, 2016 at 12:25 PM, Gratian Lup <[hidden email]> wrote:
On Wednesday, November 16, 2016 at 5:23:58 AM UTC-8, Ted Mielczarek wrote:
> Gratian,
>
> One of my coworkers reminded me of something that might be an option for
> you--we have scripts that would allow you to provide a Firefox build
> that you generated (at a publicly accessible URL) and trigger test jobs
> on that build in our CI infrastructure. If that's something that sounds
> useful to you we can definitely make that happen.
>
> You'd have to produce a Firefox build, run the `mach package` and `mach
> package-tests` targets, upload a few zip files from the $objdir/dist
> directory to somewhere accessible via public HTTP and then run a Python
> script to schedule test jobs against those files.
>
> -Ted

Hi Ted,

Thanks a lot for your help!
Using Windows 7 or 8 to do the tests should be fine - I actually don't see any reason it shouldn't be OK to also do other builds and tests on them when not used for Firefox.

The idea of testing on your infrastructure is tempting, but probably would consume too many resources, since this new testing system is intended to be used both overnight against the latest good compiler build, but also by every developer on its own while working on new features - this would be quite a lot of people. Doing a test build now would still be a good idea, though, at least to see  if everything passes in the right environment.

Something Mozilla has been wanting to do for a while is stand up "nightly" builds using the latest, unreleased toolchain from upstream. This would help us identify compiler compatibility issues early. And it may even help compiler toolchain developers find unwanted regressions.

In theory, Microsoft developers could gain access to our "Try" server, which allows you to push changes made to Firefox in version control (including its automation config) and run the results on our infrastructure. We have a file in the repository referencing a zip file of the Visual Studio toolchain to use. So, it would be possible for someone at Microsoft to create a zip file of an unreleased VC++ toolchain and have our automation run against that. Of course, there are scaling and privacy issues to work out. (I'm guessing Microsoft LCA will have an opinion on mostly-in-the-public-domain Mozilla infrastructure accessing pre-release Microsoft software.)

 

I have a few more questions about running the tests:

1. How exactly should the build artifacts be copied to the test machine? Something like ./mach package? After copying over, running the tests with ./mach is going to pick the binaries after copying, or is some ./mach "unpack" needed? I assume the entire mozilla-central enlistment is also required.

`mach package` will get you a standalone archive or installer suitable for deploying to test machines.

For historical reasons, not all `mach *test` commands integrate really well with pre-built archives/installers. There is a whole other layer that deals with running tests in automation. That being said, some `mach` commands do accept an argument with an alternate path to a binary to use. e.g. `mach reftest --appname <path>` and `mach mochitest --appname <path>`. Running tests from arbitrary binaries is definitely something we'd like to make a bit easier. So if there's a feature you'd like, don't hesitate to ask.


2. Can I see on Treeherder the exact command line that was used to launch the test suite? I looked over the log files and didn't find anything like that.

In some tests. e.g. in https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1479739510/mozilla-central_win7_vm_test-web-platform-tests-1-bm129-tests1-windows-build17.txt.gz the main command is:

  c:/mozilla-build/python27/python' '-u' 'scripts/scripts/web_platform_tests.py' '--cfg' 'web_platform_tests/prod_config_windows.py' '--test-type=testharness' '--total-chunks' '5' '--this-chunk' '1' '--blob-upload-branch' 'mozilla-central' '--download-symbols' 'ondemand'

This is invoking a tool called "mozharness" which is our "execute in automation" layer. This tool performs tons of setup work, such as obtaining a version control checkout, creating Python virtualenvs, etc. Once that's all in place, it executes a test runner. e.g.

08:22:59 INFO - Calling ['c:\\slave\\test\\build\\venv\\Scripts\\python', '-u', 'c:\\slave\\test\\build\\tests\\web-platform\\runtests.py', '--log-raw=-', '--log-raw=c:\\slave\\test\\build\\blobber_upload_dir\\wpt_raw.log', '--log-errorsummary=c:\\slave\\test\\build\\blobber_upload_dir\\wpt_errorsummary.log', '--binary=c:\\slave\\test\\build\\application\\firefox\\firefox.exe', '--symbols-path=https://queue.taskcluster.net/v1/task/G49XlpNvSWK9pkFLL2pT_A/artifacts/public/build/firefox-53.0a1.en-US.win32.crashreporter-symbols.zip', '--stackwalk-binary=c:\\slave\\test\\build\\win32-minidump_stackwalk.exe', '--stackfix-dir=c:\\slave\\test\\build\\tests\\bin', '--test-type=testharness', '--disable-e10s', '--total-chunks=5', '--this-chunk=1', '--prefs-root=c:\\slave\\test\\build\\tests\\web-platform/prefs', '--processes=1', '--config=c:\\slave\\test\\build\\tests\\web-platform/wptrunner.ini', '--ca-cert-path=c:\\slave\\test\\build\\tests\\web-platform/certs/cacert.pem', '--host-key-path=c:\\slave\\test\\build\\tests\\web-platform/certs/web-platform.test.key', '--host-cert-path=c:\\slave\\test\\build\\tests\\web-platform/certs/web-platform.test.pem', '--certutil-binary=c:\\slave\\test\\build\\tests/bin/certutil'] with output_timeout 1000
There is a lot of complexity here. Most of it unique to our automation infrastructure. I recommend you avoid mozharness completely and try to execute things with "mach." (This is the direction we're trying to take things.) Again, if you need features to make that easier, please ask and we'll see what we can do.


I'm going to try again with Update 3, I might have used instead the most recent build, which can indeed show some new errors, the frontend team do a lot of changes.

Thanks,
Gratian
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds



_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Firefox to test the Visual C++ compiler

Ryan VanderMeulen
In reply to this post by lgratian
On Wednesday, November 16, 2016 at 3:27:45 AM UTC-5, Gratian Lup wrote:

> Hello,
>
> *I posted this on multiple mailing lists, I'm not sure which is the right place for these questions.
>
> I'm a developer on the Microsoft Visual C++ compiler (code optimizer) and I'm looking into extending our test suite with popular open-source projects. This helps  us to find bugs earlier, ensuring that these projects are not broken by frontend or backend changes. It also helps the projects themselves, by making upgrades to the never compiler easier - ideally without any problems.
>
> I have several questions about building Firefox on Windows and especially about running the correctness tests:
>
> 1. What is the latest version of Visual Studio that should be used? I tried first VS2015 Update 3 and there seems to be a C++ error about "constexpr". Next I tried Update 2 and that one finishes the build.
>
> 2. What are the tests that should be run to test correctness? The idea here is to ensure that new optimizations don't introduce new bugs, which should be exposed by new test failures in Firefox. I looked over the QA Automated testing page and there seem to be a lot of different test suites - running all is probably not feasible. Is there a subset that is run when a checkin is done, for example?
>
> 3. What Windows OS do you use for testing? Window 10, Windows 8, etc? Any other configuration work needed, such as disabling the antivirus/firewall?
>
> After the build completed, I tried to run a few of the tests and there seem to be many failures even with a debug build. I assume that the checked in tests are supposed to pass, so it's likely a Windows config problem or failure to access extra files from the Web.
>
> For example, ./mach reftests had about 700 failures. ./mach gtest, jsapi-tests, jsbrowsertest had none. The tests were run under Windows 10 Anniversary update with the AV disabled.
>
> Thanks,
> Gratian Lup
> Microsoft Visual C++ team

FYI, setting --enable-js-shell will get you js.exe as part of your Firefox build.
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Firefox to test the Visual C++ compiler

Gregory Szorc-3
In reply to this post by lgratian
Many of our tests have inconsistent results.

Generally speaking, developers only run a narrow subset of tests locally and defer running the full suite of tests to our automation infrastructure. Within that infrastructure, we have a database of known "intermittent" failures and tools that attempt to auto classify known intermittent failures. People find intermittent failures that fall through the cracks. In some cases, we outright disable tests that aren't reliable enough. In other cases we set thresholds as to what their expected failure rate should be. Unfortunately, we don't have a good answer for reproducing this classification infrastructure outside of Mozilla :/

Furthermore, there are some tests that are so wonky that reproducing behavior outside of our automation environment is difficult or impossible. It is not uncommon to have to "check out" a machine from our automation environment in order to debug a failure.

If your goal is to qualify pre-release compiler changes against Firefox, I'd start by focusing on tests that run reliably. Generally speaking, "headless" tests (tests not rendering a Firefox window) are more reliable. This includes gtest, xpcshell, and many JS engine tests. Reftests, crashtests, and web platform tests (WPT) are also pretty isolated and are generally more consistent. Mochitests tend to have the most inconsistency from my experience.

This general problem of inconsistent test execution is a complex topic and has consumed thousands of people hours at Mozilla. If you're intent on running large parts of Firefox automation, it's probably worth a meeting or video conference with some of us to go over in more detail.
 
On Fri, Nov 25, 2016 at 3:36 AM, Gratian Lup <[hidden email]> wrote:
Hi,

Thanks a lot for your help! I think doing the testing on our machines is a better approach overall, even from an engineering perspective - in case of a failure you can just grab the machine and start debugging, recompiling, etc. The legal work, if possible at all, would be quite complicated too. 

This new approach of testing benefits both us and the tested projects. What we are doing:
- If the failure is in the compiler frontend, it is either a bug or a problem in the code itself. There were a few cases when the code had problems and we informed and helped those project to solve it - this would also be the case with Firefox.
- If the failure is in the compiler backend/optimizer, fix the bug. This is pretty much always a real bug, unless the source code triggers some cases of undefined behavior - we didn't find such a case yet.

I tried running the tests on Windows 7 this time, with the AV and firewall disabled, just to be sure they don't interfere. When using the mach commands, there are still quite a lot of failures, so I assume that something is still not configured right. I was curious if it's different on Linux, but tests are also failing there (I used Xubuntu 16.10 with KDE). 

Most failures are in the mochitest, reftest and wpt suites. If it's not possible to make all the tests pass, a good enough approach is to establish a baseline of tests that are known to fail using a debug/non-optimized build and consider a real failure only a failed test not in that set. We already do this for a few other large projects and seems like a good compromise.

Is there a way to skip over the tests that are known to fail? With Gtest this is easy using the --gtest_filter flag. If it's not possible, then a script is needed to parse the results and ignore those known failures. 

Thanks,
Gratian

Here are some results from the tests running on Windows 7. You can find some log files here: https://1drv.ms/f/s!AmGHUgGfTN19hjuFKlOeZ7VtpUsd

o Tests without failures:
mach gtest
mach marionette-test
mach firefox-ui-functional
mach jsapi-tests
mach jsbrowsertest

o Spidermonkey tests
mach check-spidermonkey doesn't seem to work because js.exe is not built by default. To test it I built the js folder and followed the Spidermonkey test instructions to run the JS and JIT tests - both pass without failures.

o Tests with failures
- mach mochitest

Part 1

1 INFO Passed:  675002

2 INFO Failed:  23

3 INFO Todo:    1586

4 INFO Mode:    e10s

Part 2

1 INFO Passed:  676509

2 INFO Failed:  16

3 INFO Todo:    1592

4 INFO Mode:    e10s


Example of failed test: dom/base/test/test_noAudioNotificationOnMutedElement.html | Test timed out.


- mach reftest

REFTEST INFO | Successful: 13436 (13416 pass, 20 load only)

REFTEST INFO | Unexpected: 114 (113 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 1 failed load, 0 exception)

REFTEST INFO | Known problems: 700 (285 known fail, 0 known asserts, 69 random, 346 skipped, 0 slow)

Example of failed test: c:/firefox/mozilla-central/layout/reftests/bugs/321402-4.xul == file:///c:/firefox/mozilla-central/layout/reftests/bugs/321402-4-ref.xul | image comparison, max difference: 32, number of differing pixels: 1


The images look identical to me, but 1 pixel is supposedly different - I'm wondering why the test fails if the max. acceptable diff is 32. Some other cases I look into also had "identical" images.


- mach web-platform-test

- mach crash-test



On Mon, Nov 21, 2016 at 9:54 AM, Gregory Szorc <[hidden email]> wrote:
On Wed, Nov 16, 2016 at 12:25 PM, Gratian Lup <[hidden email]> wrote:
On Wednesday, November 16, 2016 at 5:23:58 AM UTC-8, Ted Mielczarek wrote:
> Gratian,
>
> One of my coworkers reminded me of something that might be an option for
> you--we have scripts that would allow you to provide a Firefox build
> that you generated (at a publicly accessible URL) and trigger test jobs
> on that build in our CI infrastructure. If that's something that sounds
> useful to you we can definitely make that happen.
>
> You'd have to produce a Firefox build, run the `mach package` and `mach
> package-tests` targets, upload a few zip files from the $objdir/dist
> directory to somewhere accessible via public HTTP and then run a Python
> script to schedule test jobs against those files.
>
> -Ted

Hi Ted,

Thanks a lot for your help!
Using Windows 7 or 8 to do the tests should be fine - I actually don't see any reason it shouldn't be OK to also do other builds and tests on them when not used for Firefox.

The idea of testing on your infrastructure is tempting, but probably would consume too many resources, since this new testing system is intended to be used both overnight against the latest good compiler build, but also by every developer on its own while working on new features - this would be quite a lot of people. Doing a test build now would still be a good idea, though, at least to see  if everything passes in the right environment.

Something Mozilla has been wanting to do for a while is stand up "nightly" builds using the latest, unreleased toolchain from upstream. This would help us identify compiler compatibility issues early. And it may even help compiler toolchain developers find unwanted regressions.

In theory, Microsoft developers could gain access to our "Try" server, which allows you to push changes made to Firefox in version control (including its automation config) and run the results on our infrastructure. We have a file in the repository referencing a zip file of the Visual Studio toolchain to use. So, it would be possible for someone at Microsoft to create a zip file of an unreleased VC++ toolchain and have our automation run against that. Of course, there are scaling and privacy issues to work out. (I'm guessing Microsoft LCA will have an opinion on mostly-in-the-public-domain Mozilla infrastructure accessing pre-release Microsoft software.)

 

I have a few more questions about running the tests:

1. How exactly should the build artifacts be copied to the test machine? Something like ./mach package? After copying over, running the tests with ./mach is going to pick the binaries after copying, or is some ./mach "unpack" needed? I assume the entire mozilla-central enlistment is also required.

`mach package` will get you a standalone archive or installer suitable for deploying to test machines.

For historical reasons, not all `mach *test` commands integrate really well with pre-built archives/installers. There is a whole other layer that deals with running tests in automation. That being said, some `mach` commands do accept an argument with an alternate path to a binary to use. e.g. `mach reftest --appname <path>` and `mach mochitest --appname <path>`. Running tests from arbitrary binaries is definitely something we'd like to make a bit easier. So if there's a feature you'd like, don't hesitate to ask.


2. Can I see on Treeherder the exact command line that was used to launch the test suite? I looked over the log files and didn't find anything like that.

In some tests. e.g. in https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1479739510/mozilla-central_win7_vm_test-web-platform-tests-1-bm129-tests1-windows-build17.txt.gz the main command is:

  c:/mozilla-build/python27/python' '-u' 'scripts/scripts/web_platform_tests.py' '--cfg' 'web_platform_tests/prod_config_windows.py' '--test-type=testharness' '--total-chunks' '5' '--this-chunk' '1' '--blob-upload-branch' 'mozilla-central' '--download-symbols' 'ondemand'

This is invoking a tool called "mozharness" which is our "execute in automation" layer. This tool performs tons of setup work, such as obtaining a version control checkout, creating Python virtualenvs, etc. Once that's all in place, it executes a test runner. e.g.

08:22:59 INFO - Calling ['c:\\slave\\test\\build\\venv\\Scripts\\python', '-u', 'c:\\slave\\test\\build\\tests\\web-platform\\runtests.py', '--log-raw=-', '--log-raw=c:\\slave\\test\\build\\blobber_upload_dir\\wpt_raw.log', '--log-errorsummary=c:\\slave\\test\\build\\blobber_upload_dir\\wpt_errorsummary.log', '--binary=c:\\slave\\test\\build\\application\\firefox\\firefox.exe', '--symbols-path=https://queue.taskcluster.net/v1/task/G49XlpNvSWK9pkFLL2pT_A/artifacts/public/build/firefox-53.0a1.en-US.win32.crashreporter-symbols.zip', '--stackwalk-binary=c:\\slave\\test\\build\\win32-minidump_stackwalk.exe', '--stackfix-dir=c:\\slave\\test\\build\\tests\\bin', '--test-type=testharness', '--disable-e10s', '--total-chunks=5', '--this-chunk=1', '--prefs-root=c:\\slave\\test\\build\\tests\\web-platform/prefs', '--processes=1', '--config=c:\\slave\\test\\build\\tests\\web-platform/wptrunner.ini', '--ca-cert-path=c:\\slave\\test\\build\\tests\\web-platform/certs/cacert.pem', '--host-key-path=c:\\slave\\test\\build\\tests\\web-platform/certs/web-platform.test.key', '--host-cert-path=c:\\slave\\test\\build\\tests\\web-platform/certs/web-platform.test.pem', '--certutil-binary=c:\\slave\\test\\build\\tests/bin/certutil'] with output_timeout 1000
There is a lot of complexity here. Most of it unique to our automation infrastructure. I recommend you avoid mozharness completely and try to execute things with "mach." (This is the direction we're trying to take things.) Again, if you need features to make that easier, please ask and we'll see what we can do.


I'm going to try again with Update 3, I might have used instead the most recent build, which can indeed show some new errors, the frontend team do a lot of changes.

Thanks,
Gratian
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds




_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Loading...