Announcing TC39 ECMAScript Test Suite Project - Test262

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Announcing TC39 ECMAScript Test Suite Project - Test262

Allen Wirfs-Brock-2
At the last TC39 meeting we agreed to launch a project to develop an official ECMAScript specification test suite that developers and users can use to verify that implementations accurately implement the standard.  The hope is that we can integrate the various existing test suites along with new tests to create a single comprehensive test suite for the ECMAScript standard.

Because of the nature of Ecma's intellectual property policies, only Ecma members can directly contribute code to the project.  However, the project code will be BSD licensed and development will be carried out in a publically accessible manner.  The working name of the test suite is Test262 and I'll be serving as the initial project editor. The source code of the project will be hosted  in the TS39 repository at  http://hg.ecmascript.org  (coming soon) and issues will be tracked at http://bugs.ecmascript.org/report/15 .  In addition, I'm setting up a separate  test262-discuss mailing list for topics relating to ECMAScript testing and the Test262 test suite. Test262-discuss is open to the public. You can subscribe at t http://mail.mozilla.org/listinfo/test262-discuss

I'm now in the early stages of organizing  the project infrastructure and collecting initial code contributions from Ecma members. To kick things off, Microsoft is contributing the ES5Confrom test suite as well as a significant number of additional tests. Christian Plesner Hansen, leader of the Sputnik project , reports that Google will contribute their Sputnik code.  Other TC39 members have all been supportive of this effort and I anticipate that we will receive additional contributions from them in the near future.

So, if you're interested in ECMAScript testing subscribe to test262-discuss and join in.  After people have had a couple of days to subscribe, I'll be kicking things off with some of my ideas about how to structure and organize the Test262 suite. Please also share your thoughts.

Thanks,
Allen Wirfs-Brock

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Announcing TC39 ECMAScript Test Suite Project - Test262

Jonathan Chetwynd
Allen,

is the Test262 test suite more similar to W3C  suites or might there  
be an acidic element?

ie something the public might understand?

regards

Jonathan Chetwynd
On 21 Jun 2010, at 06:52, Allen Wirfs-Brock wrote:

> At the last TC39 meeting we agreed to launch a project to develop an  
> official ECMAScript specification test suite that developers and  
> users can use to verify that implementations accurately implement  
> the standard.  The hope is that we can integrate the various  
> existing test suites along with new tests to create a single  
> comprehensive test suite for the ECMAScript standard.
>
> Because of the nature of Ecma's intellectual property policies, only  
> Ecma members can directly contribute code to the project.  However,  
> the project code will be BSD licensed and development will be  
> carried out in a publically accessible manner.  The working name of  
> the test suite is Test262 and I'll be serving as the initial project  
> editor. The source code of the project will be hosted  in the TS39  
> repository at  http://hg.ecmascript.org  (coming soon) and issues  
> will be tracked at http://bugs.ecmascript.org/report/15 .  In  
> addition, I'm setting up a separate  test262-discuss mailing list  
> for topics relating to ECMAScript testing and the Test262 test  
> suite. Test262-discuss is open to the public. You can subscribe at t http://mail.mozilla.org/listinfo/test262-discuss
>
> I'm now in the early stages of organizing  the project  
> infrastructure and collecting initial code contributions from Ecma  
> members. To kick things off, Microsoft is contributing the  
> ES5Confrom test suite as well as a significant number of additional  
> tests. Christian Plesner Hansen, leader of the Sputnik project ,  
> reports that Google will contribute their Sputnik code.  Other TC39  
> members have all been supportive of this effort and I anticipate  
> that we will receive additional contributions from them in the near  
> future.
>
> So, if you're interested in ECMAScript testing subscribe to test262-
> discuss and join in.  After people have had a couple of days to  
> subscribe, I'll be kicking things off with some of my ideas about  
> how to structure and organize the Test262 suite. Please also share  
> your thoughts.
>
> Thanks,
> Allen Wirfs-Brock
>
> _______________________________________________
> es-discuss mailing list
> [hidden email]
> https://mail.mozilla.org/listinfo/es-discuss

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

RE: Announcing TC39 ECMAScript Test Suite Project - Test262

Allen Wirfs-Brock-2
The intent is a suite that tests an implementation against the actual requirement of the Ecma-262 standard. It will be based upon and similar to what you can already see in the Sputnik and ES5conform suites.  Those tests generally identify a specific requirement of a specific section of the spec. and then try to prove that the requirement is/isn't met by an implementation.

Give that the standard specification itself  is not particularly understandable to the general public, it probably isn't reasonable to expect that all the tests of the specification would be understandable to the same public.

However, reasonable understandability is a good goal.  What are some of the characteristics of the W3C suites that make them "acidic"?

Allen

> -----Original Message-----
> From: Jonathan Chetwynd [mailto:[hidden email]]
> Sent: Monday, June 21, 2010 1:40 AM
> To: Allen Wirfs-Brock
> Cc: [hidden email]
> Subject: Re: Announcing TC39 ECMAScript Test Suite Project - Test262
>
> Allen,
>
> is the Test262 test suite more similar to W3C  suites or might there be an acidic
> element?
>
> ie something the public might understand?
>
> regards
>
> Jonathan Chetwynd
> On 21 Jun 2010, at 06:52, Allen Wirfs-Brock wrote:
>
> > At the last TC39 meeting we agreed to launch a project to develop an
> > official ECMAScript specification test suite that developers and users
> > can use to verify that implementations accurately implement the
> > standard.  The hope is that we can integrate the various existing test
> > suites along with new tests to create a single comprehensive test
> > suite for the ECMAScript standard.
> >
> > Because of the nature of Ecma's intellectual property policies, only
> > Ecma members can directly contribute code to the project.  However,
> > the project code will be BSD licensed and development will be carried
> > out in a publically accessible manner.  The working name of the test
> > suite is Test262 and I'll be serving as the initial project editor.
> > The source code of the project will be hosted  in the TS39 repository
> > at  http://hg.ecmascript.org  (coming soon) and issues will be tracked
> > at http://bugs.ecmascript.org/report/15 .  In addition, I'm setting up
> > a separate  test262-discuss mailing list for topics relating to
> > ECMAScript testing and the Test262 test suite. Test262-discuss is open
> > to the public. You can subscribe at t
> > http://mail.mozilla.org/listinfo/test262-discuss
> >
> > I'm now in the early stages of organizing  the project infrastructure
> > and collecting initial code contributions from Ecma members. To kick
> > things off, Microsoft is contributing the ES5Confrom test suite as
> > well as a significant number of additional tests. Christian Plesner
> > Hansen, leader of the Sputnik project , reports that Google will
> > contribute their Sputnik code.  Other TC39 members have all been
> > supportive of this effort and I anticipate that we will receive
> > additional contributions from them in the near future.
> >
> > So, if you're interested in ECMAScript testing subscribe to test262-
> > discuss and join in.  After people have had a couple of days to
> > subscribe, I'll be kicking things off with some of my ideas about how
> > to structure and organize the Test262 suite. Please also share your
> > thoughts.
> >
> > Thanks,
> > Allen Wirfs-Brock
> >
> > _______________________________________________
> > es-discuss mailing list
> > [hidden email]
> > https://mail.mozilla.org/listinfo/es-discuss
>

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Announcing TC39 ECMAScript Test Suite Project - Test262

Sam Tobin-Hochstadt
On Mon, Jun 21, 2010 at 1:42 PM, Allen Wirfs-Brock
<[hidden email]> wrote:
> The intent is a suite that tests an implementation against the actual requirement of the Ecma-262 standard. It will be based upon and similar to what you can already see in the Sputnik and ES5conform suites.  Those tests generally identify a specific requirement of a specific section of the spec. and then try to prove that the requirement is/isn't met by an implementation.
>
> Give that the standard specification itself  is not particularly understandable to the general public, it probably isn't reasonable to expect that all the tests of the specification would be understandable to the same public.
>
> However, reasonable understandability is a good goal.  What are some of the characteristics of the W3C suites that make them "acidic"?

I believe that the various "Acid" tests for rendering are what's being
referred to here [1].  I think the relevant characteristc is that
anyone can run them in their browser, and they display a visual
rendering of how "good" a job the browser does, making them into
easily-distinguishable markers of conformance to whatever is tested.
It's not clear to me whether this would be good for this test suite.

[1] http://en.wikipedia.org/wiki/Acid3
--
sam th
[hidden email]
_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss
Reply | Threaded
Open this post in threaded view
|

RE: Announcing TC39 ECMAScript Test Suite Project - Test262

Allen Wirfs-Brock-2
(I'm going to try to shift this thread to the test262-discuss list, by bccing: es-discuss.)

I think we all agree that a web based test runner is part of what we want to have.  It seems inevitable that some sort of pass % or similar score has to display even though it's not clear whether such a score is very meaningful for this style of test suite.  You can get big fluctuation because a single bugs can cause many individual tests to fail while in other cases a really significant bugs  might only show up in a small number.

If anybody has any ideas for a meaningful metric that doesn't have those sorts of sensitivities, let's hear it.

Allen

> -----Original Message-----
> From: Christian Plesner Hansen [mailto:[hidden email]]
> Sent: Tuesday, June 22, 2010 4:00 PM
> To: Sam Tobin-Hochstadt
> Cc: Allen Wirfs-Brock; [hidden email]; Jonathan Chetwynd; es-
> [hidden email]
> Subject: Re: Announcing TC39 ECMAScript Test Suite Project - Test262
>
> Yes, there is an acidic element to this project.  There will eventually be a website
> where anyone can test their browser's conformance against the test262 suite
> like they can today against the acid tests.
>
>
> -- Christian
>
> On Tue, Jun 22, 2010 at 3:47 AM, Sam Tobin-Hochstadt <[hidden email]>
> wrote:
> > On Mon, Jun 21, 2010 at 1:42 PM, Allen Wirfs-Brock
> > <[hidden email]> wrote:
> >> The intent is a suite that tests an implementation against the actual
> requirement of the Ecma-262 standard. It will be based upon and similar to what
> you can already see in the Sputnik and ES5conform suites.  Those tests generally
> identify a specific requirement of a specific section of the spec. and then try to
> prove that the requirement is/isn't met by an implementation.
> >>
> >> Give that the standard specification itself  is not particularly understandable
> to the general public, it probably isn't reasonable to expect that all the tests of
> the specification would be understandable to the same public.
> >>
> >> However, reasonable understandability is a good goal.  What are some of the
> characteristics of the W3C suites that make them "acidic"?
> >
> > I believe that the various "Acid" tests for rendering are what's being
> > referred to here [1].  I think the relevant characteristc is that
> > anyone can run them in their browser, and they display a visual
> > rendering of how "good" a job the browser does, making them into
> > easily-distinguishable markers of conformance to whatever is tested.
> > It's not clear to me whether this would be good for this test suite.
> >
> > [1] http://en.wikipedia.org/wiki/Acid3
> > --
> > sam th
> > [hidden email]
> > _______________________________________________
> > test262-discuss mailing list
> > [hidden email]
> > https://mail.mozilla.org/listinfo/test262-discuss
> >

_______________________________________________
es-discuss mailing list
[hidden email]
https://mail.mozilla.org/listinfo/es-discuss