Personal thoughts on test automation

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Personal thoughts on test automation

Alex Vincent
Having recently patched a couple nasty regressions in DOM content code,
I'm somewhat interested in monitoring and slightly participating in this
project.  I looked at the Test Automation thread, and I didn't see a
spot for code contributions in general.

First, my company has just released a source tarball that includes
content-to-chrome messaging.  This means you could have sandboxed
webpages (HTML, XUL, whatever) running tests, and use my chrome
messenger to send results upstairs to the chrome.  I'm going to blog
about it shortly, to start a process of possibly checking the messenger
code into cvs.m.o.  (I'll want to get some people thinking about whether
they want it in the codebase, but that's a discussion for my blog, not
here.)

Second, I'd like to offer what testcases I can, but my time is more
limited than I'd like to admit.  I'd probably just write ordinary
Bugzilla-based testcases, not designed to any particular system.  In
this case, having a good written guide to converting ordinary
HTML/XUL/XML testcases for compatibility with your system would be nice.

My particular focus is on DOM issues; we had a really spectacular
bustage at https://bugzilla.mozilla.org/show_bug.cgi?id=338541.

I was wondering if XULRunner might be a better platform to base tests on
than Firefox.  I didn't see a chrome/ directory listed...

I thought briefly about the idea of firing assertions from test
failures, but bsmedberg talked me out of that one.  Still, it is
possible, if anyone's interested.  The sort of regressions I think about
are often serious enough that Mozilla products on trunk can become unusable.

Thoughts?



Alex Vincent
Weblog: http://weblogs.mozillazine.org/weirdal/
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Personal thoughts on test automation

Bob Clary
Alex Vincent wrote:
> Having recently patched a couple nasty regressions in DOM content code,
> I'm somewhat interested in monitoring and slightly participating in this
> project.  I looked at the Test Automation thread, and I didn't see a
> spot for code contributions in general.
>

I guess the contribution part is dependent on how we end up putting all
of this stuff in the tree. We have had a hiatus while people when to
XTech and now are on vacation. I hope to pick this thread up again in
earnest next week.

> First, my company has just released a source tarball that includes
> content-to-chrome messaging.  This means you could have sandboxed
> webpages (HTML, XUL, whatever) running tests, and use my chrome
> messenger to send results upstairs to the chrome.  I'm going to blog
> about it shortly, to start a process of possibly checking the messenger
> code into cvs.m.o.  (I'll want to get some people thinking about whether
> they want it in the codebase, but that's a discussion for my blog, not
> here.)
>

I am sure this could be used, but off hand don't see a major benefit.
That doesn't mean that there isn't one though. Sending messages to
chrome is interesting and probably has a number of use-cases, but we are
already able to inspect the dom of test pages using Spider
<http://bclary.com/projects/spider/> and I think with Dave's work on the
use of Alex Fritze's jsshh can also do something similar.

The idea of standardizing observer topics for testing is interesting too.

> Second, I'd like to offer what testcases I can, but my time is more
> limited than I'd like to admit.  I'd probably just write ordinary
> Bugzilla-based testcases, not designed to any particular system.  In
> this case, having a good written guide to converting ordinary
> HTML/XUL/XML testcases for compatibility with your system would be nice.
>

Thanks for the offer. The details of the format of testcases still has
to be worked out and will probably vary depending on the component. We
should know more hopefully soon.

> My particular focus is on DOM issues; we had a really spectacular
> bustage at https://bugzilla.mozilla.org/show_bug.cgi?id=338541.
>
> I was wondering if XULRunner might be a better platform to base tests on
> than Firefox.  I didn't see a chrome/ directory listed...
>

XULRunner is the future but whatever solution we pick will have to also
support Firefox 1.5.x and Firefox 2.x for their lifetimes. I am
interested in what XULRunner can provide us, but haven't got any
experience with it (yet).

Once we have more settled in the bookkeeping areas of infrastructure
location and test locations in cvs, we can think about the different
methods we might use on the 1.8.* branch vs. 1.9 going forward.

> I thought briefly about the idea of firing assertions from test
> failures, but bsmedberg talked me out of that one.  Still, it is
> possible, if anyone's interested.  The sort of regressions I think about
> are often serious enough that Mozilla products on trunk can become
> unusable.
>

Assertions for some limited classes of test I think would be good, but
for general tests they aren't appropriate. For example, nightly opt
builds and release builds should be testable as well which rules out
assertions for the normal run of the mill test.

> Thoughts?
>

Welcome and thanks for offering to help out. We need all of the help we
can get.

/bc
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Personal thoughts on test automation

Dave Liebreich
In reply to this post by Alex Vincent
Alex Vincent wrote:
> Thoughts?

I have found it much more productive to discuss automated testing issues
such as harnesses and platforms in the presence of a specific test case
we are looking to run.

Can you point us to one test case?  We'd need to know what components
are necessary in the testbed, as well as how to programmatically
determine if the test case passes or fails.

I'd prefer non-crashing test cases for now :-)

For example, load this web page (url) and make sure the innerHTML is
equal to ("serialized dom string").

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Personal thoughts on test automation

Rob Sayre-2
Dave Liebreich wrote:
 >
> Can you point us to one test case?  We'd need to know what components
> are necessary in the testbed, as well as how to programmatically
> determine if the test case passes or fails.

Ooh, I have one.

In the function attr_modified below, ev.prevValue should be
"width: 20em;" (broken right now, but I have a patch). How do I get this
in a test so it doesn't regress?

-Rob


<!DOCTYPE html>
<title>Testcase bug 338679</title>
<script>
function main() {
     with (document.getElementById("out")) {
         addEventListener("DOMAttrModified", attr_modified, false)
         style.width = "auto"
     }
}
function attr_modified(ev) {
     this.textContent = "Previous:\t" + ev.prevValue + "\nNew:\t\t" +
ev.newValue
}
window.onload = main
</script>

<dl>
<dt>Actual result:</dt>
<dd>
     <pre id="out" style="width: 20em">
     </pre>
</dd>

<dt>Expected result:</dt>

<dd>
     <pre>
Previous: width: 20em;
New: width: auto;
     </pre>
</dd>
</dl>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Personal thoughts on test automation

Alex Vincent
In reply to this post by Dave Liebreich
Dave Liebreich wrote:

> Alex Vincent wrote:
>> Thoughts?
>
> I have found it much more productive to discuss automated testing issues
> such as harnesses and platforms in the presence of a specific test case
> we are looking to run.
>
> Can you point us to one test case?  We'd need to know what components
> are necessary in the testbed, as well as how to programmatically
> determine if the test case passes or fails.
>
> I'd prefer non-crashing test cases for now :-)
>
> For example, load this web page (url) and make sure the innerHTML is
> equal to ("serialized dom string").
>

Modifying the testcase from
https://bugzilla.mozilla.org/show_bug.cgi?id=338541 shouldn't be too
hard.  :)
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Personal thoughts on test automation

Dave Liebreich
In reply to this post by Rob Sayre-2
I think jsUnit may be the way to go for both of these cases (bugs 338679
and 338541), since

1) the test logic does not require special privileges
2) jsUnit has support for "wait until the page loads and everything that
mucks with the DOM is done" support via setUpPage()
3) the other frameworks I'm playing with (foxunit and a homegrown
framework that uses jssh) do not have the support for #2

I'll write up the tests this weekend and post them when I'm done.

One interesting bit is that jsUnit has a server mode that can manage
runs across many machines.

jsUnit: http://www.jsunit.net/

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Personal thoughts on test automation

Dave Liebreich
I installed jsUnit 2.2alpha (March 23) on
http://people.mozilla.com/~davel/jsunit/

I converted the two test cases to jsUnit tests:

http://people.mozilla.com/~davel/jsunit/dom_338541.xhtml
http://people.mozilla.com/~davel/jsunit/dom_338679.html

and added a SVG test based on http://jwatt.org/svg/tests/x-dom.svg

http://people.mozilla.com/~davel/jsunit/x-dom-svg.xhtml

To run the tests, load the test runner
(http://people.mozilla.com/~davel/jsunit/testRunner.html) and run the
tests individually.  Or run a suite of all of them at
http://people.mozilla.com/~davel/jsunit/dom_suite.html

jsUnit works like most *Unit test harnesses.  Functions that start with
"test" are run after the page loads (and after an onload functions
defined in the page).  The test for 338679 demonstrates the setUpPage()
functionality of jsUnit - that function is called after the page loads,
and the test* functions are called only after the variable
setUpPageStatus is set to "complete".

338679 fails on Fx 1.5.0.4 (which will be released soon). I don't plan
on putting tests into the regression suite if we expect them to fail.

The conversion of these three test cases was very easy.  I will work on
installing jsUnit Server on a few machines.  I suspect that too will go
smoothly, since we already have an infrastructure for starting up builds
on our 3 core platforms (4 if you count mac ppc and mac intel separately).

Feel free to send me a list of bugs with testcases similar to these, and
I'll jsUnit-ify them too.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Personal thoughts on test automation

Rob Sayre-2
Dave Liebreich wrote:
>
> http://people.mozilla.com/~davel/jsunit/dom_338679.html
>
> 338679 fails on Fx 1.5.0.4 (which will be released soon). I don't plan
> on putting tests into the regression suite if we expect them to fail.
>

The patch will be for FF3/1.9a, since it requires an interface change.
Is there a way to add tests for the trunk?

-Rob
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Personal thoughts on test automation

Dave Liebreich
Robert Sayre wrote:

> The patch will be for FF3/1.9a, since it requires an interface change.
> Is there a way to add tests for the trunk?
>
> -Rob

When we (me? me+build?) set something up to run these tests
continuously, I'll make sure there is a way to run a particular test
only on certain branches.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Personal thoughts on test automation

beaufour@gmail.com
In reply to this post by Dave Liebreich
On 5/29/06, Dave Liebreich <[hidden email]> wrote:
> The conversion of these three test cases was very easy.  I will work on
> installing jsUnit Server on a few machines.  I suspect that too will go
> smoothly, since we already have an infrastructure for starting up builds
> on our 3 core platforms (4 if you count mac ppc and mac intel separately).

I use jsUnit testing for my xforms tests too, and I also find it very
useful. If I could "hook into" a general jsUnit framework, that would
be great. I'm pondering how to get the test out of beaufour.dk and
into mozilla.org somehow.

The only thing I have not "fixed" yet is how to handle async. stuff in
a nice way, ie. handle submissions (like XHR). But that's not realy
jsUnit's problem :)

--
... Allan
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality