Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

Jet Villegas-2
This proposed resolution affects Gecko and Servo. Please review and comment.

Thanks,

--Jet


---------- Forwarded message ----------
From: Geoffrey Sneddon <[hidden email]>
Date: Tue, Sep 27, 2016 at 9:23 AM
Subject: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests
To: www-style list <[hidden email]>


(Bcc'd public-css-testsuite and public-test-infra; this pertains to a
CSS WG proposed resolution and hence we should keep discussion on the
CSS WG's public mailing list.)

The proposed resolution is to merge csswg-test into
web-platform-tests, doing the following:

1. Land https://github.com/w3c/wpt-tools/pull/90 into wpt-tools so
that the web-platform-tests tools create an accurate manifest (i.e.,
list of tests) for the CSS testsuite.

2. Add a number of lints to wpt-tools, for the (currently
non-existent) css subdirectory, to ensure that the build system keeps
working (primarily we need lints to ensure that we have no duplicate
file-extensionless-basenames that aren't byte-for-byte identical and
that all files referenced by test files are in an adjacent support or
reference directory, with a couple of exceptions).

3. Ensure web-platform-tests's documentation is up-to-date and
cohesive, both for submitting tests and reviewing them. Especially
make sure it's easier to find documentation than it is currently!

4. Make https://hg.csswg.org/test/ and http://test.csswg.org/shepherd/
read-only. (Really this can be any step up until this point; exact
timing doesn't matter.)

5. Merge csswg-test into web-platform-tests, in a css/ subdirectory,
maintaining all csswg-test history. (Do we want to copy w3ctestlib and
apiclient into it as well, given they currently live in Mercurial, and
are needed to build?)

6. Move over, at the very least, all open issues and PRs from the
csswg-test repository.

7. (Sometime in the more distant future) drop the current build system
and the lints we had for its requirements.

I believe Alan's opinion was to give people a week to respond on the
mailing list and then potentially have a final call for objections (or
discussion!) on the next telecon (5 Oct). I'll let the chairs to say
what they want to do, though. :)

/Geoffrey.
_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

James Graham
On 27/09/16 18:36, Jet Villegas wrote:
> This proposed resolution affects Gecko and Servo. Please review and comment.

FWIW we (well Ms2ger and I) have been heavily pushing for this change
for some time now; it considerably simplifies many things for us. For
example with the change it will be possible for both Gecko and Servo
hackers to contribute to the CSS testsuite with the same process that
already works for web-platform-tests. In Gecko it will also make it easy
to have an upd-to-date copy of the CSS tests in tree (my understanding
is that the current import is some years out of date).

For gecko specifically there might be some concerns with running the
entire CSS testsuite through the wptrunner harness, as performance that
is good enough with 500 reftests might be deemed less acceptable with
10,000. However I think this is something we can benchmark and optimise
post-merge. Certainly I don't think it's a reason not to go ahead with
this change, which will have the extremely positive effect of making the
CSS testsuite something we can both use an contribute to without
significant manual burden.

_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

Lars Bergstrom
On Tue, Sep 27, 2016 at 12:54 PM, James Graham <[hidden email]> wrote:
> On 27/09/16 18:36, Jet Villegas wrote:
>>
>> This proposed resolution affects Gecko and Servo. Please review and
>> comment.
>
>
> FWIW we (well Ms2ger and I) have been heavily pushing for this change for
> some time now; it considerably simplifies many things for us.

Indeed! Fixing up the CSS test suite's build system and upstreaming
process is something we contracted gsnedders to work on in the latter
half of 2015 and have continued to sponsor his CSSWG attendance in
support of his continued efforts here. Speaking from the Servo side,
we've been reviewing this changes throughout and are really pleased
with this direction.

> For gecko specifically there might be some concerns with running the entire
> CSS testsuite through the wptrunner harness, as performance that is good
> enough with 500 reftests might be deemed less acceptable with 10,000.
> However I think this is something we can benchmark and optimise post-merge.

This is a concern that the Chromium folks also had, and I *think*
gsneeders is currently working with them to improve that situation
(though I'm not certain).
- Lars
_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

L. David Baron-2
In reply to this post by James Graham
On Tuesday 2016-09-27 18:54 +0100, James Graham wrote:

> On 27/09/16 18:36, Jet Villegas wrote:
> >This proposed resolution affects Gecko and Servo. Please review and comment.
>
> FWIW we (well Ms2ger and I) have been heavily pushing for this change for
> some time now; it considerably simplifies many things for us. For example
> with the change it will be possible for both Gecko and Servo hackers to
> contribute to the CSS testsuite with the same process that already works for
> web-platform-tests. In Gecko it will also make it easy to have an
> upd-to-date copy of the CSS tests in tree (my understanding is that the
> current import is some years out of date).
>
> For gecko specifically there might be some concerns with running the entire
> CSS testsuite through the wptrunner harness, as performance that is good
> enough with 500 reftests might be deemed less acceptable with 10,000.
> However I think this is something we can benchmark and optimise post-merge.
> Certainly I don't think it's a reason not to go ahead with this change,
> which will have the extremely positive effect of making the CSS testsuite
> something we can both use an contribute to without significant manual
> burden.
I'd like to see us run the reftests with our existing reftest
harness, which is optimized to do image comparison in compiled code,
and which also has a bunch of useful failure-annotation features [1]
and features that test for other sorts of failures such as
NS_ASSERTIONs.

(We should probably run them at the smaller screen resolution
mandated by the test suite rather than the 800x1000 that we really
ought to switch away from, though.)

-David

[1] although I'd really like to improve fuzzy annotations per
    https://bugzil.la/1252361

--
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                          https://www.mozilla.org/   𝄂
             Before I built a wall I'd ask to know
             What I was walling in or walling out,
             And to whom I was like to give offense.
               - Robert Frost, Mending Wall (1914)

_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

James Graham
On 27/09/16 19:20, L. David Baron wrote:

> I'd like to see us run the reftests with our existing reftest
> harness, which is optimized to do image comparison in compiled code,
> and which also has a bunch of useful failure-annotation features [1]
> and features that test for other sorts of failures such as
> NS_ASSERTIONs.

I think there are a number of ways that this is non-trivial; for example
I'm not aware that we have the infrastructure needed to generate reftest
manifests from the source tree, I don't think the existing reftest
harness can handle the possibility of a test causing a crash cleanly, I
don't think we have the ability to autogenerate expectation data to
account for the fact that many tests will fail, and so on. I would
rather start with wptrunner, which already solves these problems, and
improve the harness with better performance and error reporting since
these can be added incrementally whereas the other things are
prerequisites for any frequently updated testsuite from upstream.
_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

L. David Baron-2
On Tuesday 2016-09-27 19:36 +0100, James Graham wrote:

> On 27/09/16 19:20, L. David Baron wrote:
>
> >I'd like to see us run the reftests with our existing reftest
> >harness, which is optimized to do image comparison in compiled code,
> >and which also has a bunch of useful failure-annotation features [1]
> >and features that test for other sorts of failures such as
> >NS_ASSERTIONs.
>
> I think there are a number of ways that this is non-trivial; for example I'm
> not aware that we have the infrastructure needed to generate reftest
> manifests from the source tree, I don't think the existing reftest harness
> can handle the possibility of a test causing a crash cleanly,
I don't think this is important.  Crashes are rare enough that, in
the rare case that we do have a non-intermittent crash, we can
simply skip the test until it's fixed.

> I don't think
> we have the ability to autogenerate expectation data to account for the fact
> that many tests will fail, and so on. I would rather start with wptrunner,
> which already solves these problems, and improve the harness with better
> performance and error reporting since these can be added incrementally
> whereas the other things are prerequisites for any frequently updated
> testsuite from upstream.

I don't think it's acceptable to convert any existing reftest
coverage in wptrunner's current state, or to ask our engineers to
write tests for new features targeting wptrunner rather than the
existing reftest harness.  The existing reftest harness has had a
significant amount of investment in the ability for engineers to
debug local failures and get useful information out of failures in
continuous integration.  Since we haven't really had a lot of that
experience with wptrunner, and the experience we have had has been
ignored rather than issues being fixed (see, say,
https://bugzil.la/1265575 and https://bugzil.la/1265582 , which are
completely unacceptable), and as a result of such basic things being
unfixed and the general lack of use, I don't think you have even
remotely close to the full list of issues that need to be fixed in
order to bring wptrunner up to parity.

I think you'd be better off trying to solve the autogeneration and
manifest issues in the existing harness.

-David

--
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                          https://www.mozilla.org/   𝄂
             Before I built a wall I'd ask to know
             What I was walling in or walling out,
             And to whom I was like to give offense.
               - Robert Frost, Mending Wall (1914)

_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

James Graham
On 27/09/16 19:57, L. David Baron wrote:

> I don't think it's acceptable to convert any existing reftest
> coverage in wptrunner's current state, or to ask our engineers to
> write tests for new features targeting wptrunner rather than the
> existing reftest harness.  The existing reftest harness has had a
> significant amount of investment in the ability for engineers to
> debug local failures and get useful information out of failures in
> continuous integration.  Since we haven't really had a lot of that
> experience with wptrunner, and the experience we have had has been
> ignored rather than issues being fixed (see, say,
> https://bugzil.la/1265575 and https://bugzil.la/1265582 , which are
> completely unacceptable), and as a result of such basic things being
> unfixed and the general lack of use, I don't think you have even
> remotely close to the full list of issues that need to be fixed in
> order to bring wptrunner up to parity.
>
> I think you'd be better off trying to solve the autogeneration and
> manifest issues in the existing harness.

I think this argument is backwards. Today we don't run a recent copy of
the CSS tests because there are technical issues preventing it from
working with the reftest harness. It is not clear to me why this
reorganisation would suddenly cause us to fix those issues when we could
have done at any previous time with similar effort. On the other hand
the change will allow us to run a copy of the CSS tests in wptrunner
immediately for "free". Running more tests in wptrunner inevitably
increases the chance of fixing issues with that harness both because it
will be easier to prioritise the work and more people will be motivated
to fix the issues.

Apologies for missing the bugs you pointed to at the time; I believe one
of them has a solution and I will need to investigate the second issue
(with debugger-related bugs I find it very helpful to get a set of steps
to reproduce the problem).
_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

L. David Baron-2
On Tuesday 2016-09-27 20:18 +0100, James Graham wrote:
> I think this argument is backwards. Today we don't run a recent copy of the
> CSS tests because there are technical issues preventing it from working with
> the reftest harness. It is not clear to me why this reorganisation would
> suddenly cause us to fix those issues when we could have done at any
> previous time with similar effort.

Because you were doing the work on the approach that uses your test
harness, and we didn't have anybody assigned to do the other work?

> On the other hand the change will allow
> us to run a copy of the CSS tests in wptrunner immediately for "free".
> Running more tests in wptrunner inevitably increases the chance of fixing
> issues with that harness both because it will be easier to prioritise the
> work and more people will be motivated to fix the issues.

I think running a large number of reftests in wptrunner will be an
unacceptable load on our CI infrastructure due to
https://bugzil.la/1265586 .

> Apologies for missing the bugs you pointed to at the time; I believe one of
> them has a solution and I will need to investigate the second issue (with
> debugger-related bugs I find it very helpful to get a set of steps to
> reproduce the problem).

I think those bugs are the very tip of an iceberg that involves many
person-months of work.

-David

--
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                          https://www.mozilla.org/   𝄂
             Before I built a wall I'd ask to know
             What I was walling in or walling out,
             And to whom I was like to give offense.
               - Robert Frost, Mending Wall (1914)

_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

L. David Baron-2
On Tuesday 2016-09-27 12:24 -0700, L. David Baron wrote:
> On Tuesday 2016-09-27 20:18 +0100, James Graham wrote:
> > I think this argument is backwards. Today we don't run a recent copy of the
> > CSS tests because there are technical issues preventing it from working with
> > the reftest harness. It is not clear to me why this reorganisation would
> > suddenly cause us to fix those issues when we could have done at any
> > previous time with similar effort.
>
> Because you were doing the work on the approach that uses your test
> harness, and we didn't have anybody assigned to do the other work?

And to clarify:  I'm not objecting to merging csswg-test into wpt.
I'm objecting to merging the result *back* into mozilla-central
under the wptrunner harness.

-David

--
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                          https://www.mozilla.org/   𝄂
             Before I built a wall I'd ask to know
             What I was walling in or walling out,
             And to whom I was like to give offense.
               - Robert Frost, Mending Wall (1914)

_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

James Graham
In reply to this post by L. David Baron-2
On 27/09/16 20:24, L. David Baron wrote:

>> On the other hand the change will allow
>> us to run a copy of the CSS tests in wptrunner immediately for "free".
>> Running more tests in wptrunner inevitably increases the chance of fixing
>> issues with that harness both because it will be easier to prioritise the
>> work and more people will be motivated to fix the issues.
>
> I think running a large number of reftests in wptrunner will be an
> unacceptable load on our CI infrastructure due to
> https://bugzil.la/1265586 .

That, at least, seems like a testable hypothesis once the merge is made.
I don't think the performance concerns there are insurmountable even in
the wpt architecture; as I said in the bug we can add better primitives
to marionette to do this work.

>> Apologies for missing the bugs you pointed to at the time; I believe one of
>> them has a solution and I will need to investigate the second issue (with
>> debugger-related bugs I find it very helpful to get a set of steps to
>> reproduce the problem).
>
> I think those bugs are the very tip of an iceberg that involves many
> person-months of work.

Perhaps, but I think there is also significant work in making the
reftest harness suitable for running these tests in a way that allows
them to be updated as frequently and painlessly as possible. The
difference is that I see an incremental path to victory with wptrunner
(start with running just the highest value tests for features we are
working on today, optimise the performance, fix ergonomics issues,
switch on more tests), and the improvements we make along the way will
not just help with these tests but also with all other
web-platform-tests in both gecko and servo.

_______________________________________________
dev-tech-layout mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-layout
Loading...