Re: regression testing

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
Darin Fisher wrote:

> This came up at today's Gecko 1.9 meeting, and I wanted to raise it
> again here for consideration.
>
> I believe that the Mozilla project suffers from in-adequate regression
> and unit testing.  We have difficult making large, impactful changes
> with confidence.
>
> I think that we should try to reach a point where we can require
> regression/unit tests for every patch that fixes a bug or introduces a
> new API.
>
> The challenge (as I see it) is getting a testing framework in place
> that makes it very easy for developers to contribute new tests.  We'd
> also need something that runs these tests regularly.
>
> What else is blocking the adoption of mandated regression/unit tests,
> and how do we get to a point where these blockers no longer exist?
>
> -Darin

We have a number of the building blocks for this - I had been listing
them at
<http://wiki.mozilla.org/SoftwareTesting:Catalog_of_Automated_Tests>
and the parent page <http://wiki.mozilla.org/SoftwareTesting>.  Those
pages are due for an update.

See also some of the recent postings to mozilla.dev.quality.

I think we need a few harnesses, and a little bit more work to hook them
up.  But what we need most of all is a developer (or developers) willing
to write tests for their code, and, at the same time, help work on the
last bits of the harnesses to run their tests.

-dave


--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

dolphinling-2
Dave Liebreich wrote:

> Darin Fisher wrote:
>> This came up at today's Gecko 1.9 meeting, and I wanted to raise it
>> again here for consideration.
>>
>> I believe that the Mozilla project suffers from in-adequate regression
>> and unit testing.  We have difficult making large, impactful changes
>> with confidence.
>>
>> I think that we should try to reach a point where we can require
>> regression/unit tests for every patch that fixes a bug or introduces a
>> new API.
>>
>> The challenge (as I see it) is getting a testing framework in place
>> that makes it very easy for developers to contribute new tests.  We'd
>> also need something that runs these tests regularly.
>>
>> What else is blocking the adoption of mandated regression/unit tests,
>> and how do we get to a point where these blockers no longer exist?
>>
>> -Darin
>
> We have a number of the building blocks for this - I had been listing
> them at
> <http://wiki.mozilla.org/SoftwareTesting:Catalog_of_Automated_Tests> and
> the parent page <http://wiki.mozilla.org/SoftwareTesting>.  Those pages
> are due for an update.
>
> See also some of the recent postings to mozilla.dev.quality.
>
> I think we need a few harnesses, and a little bit more work to hook them
> up.  But what we need most of all is a developer (or developers) willing
> to write tests for their code, and, at the same time, help work on the
> last bits of the harnesses to run their tests.
>
> -dave

I'm not a developer, and I certainly won't be fixing many bugs any time
soon, but I wouldn't mind helping out here off and on if it'd ease the
load on developers. I think it should definitely be set up so that it
doesn't *have* to be the person who fixed the bug that writes the test.

As a side note, this would be like writing testcases, right? Except
within a framework, and easier than a testcase because the problem's
already been identified, and you don't have to go through removing bit
by bit to find it?

--
dolphinling
<http://dolphinling.net/>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
dolphinling wrote:

> I'm not a developer, and I certainly won't be fixing many bugs any time
> soon, but I wouldn't mind helping out here off and on if it'd ease the
> load on developers. I think it should definitely be set up so that it
> doesn't *have* to be the person who fixed the bug that writes the test.

If the folks who write the code do not write the tests, then this effort
won't scale.  Especially with unit-style tests, for which we need a lot
of harness work.

I encourage others to write additional tests, and I'm glad to have you
help out.

>
> As a side note, this would be like writing testcases, right? Except
> within a framework, and easier than a testcase because the problem's
> already been identified, and you don't have to go through removing bit
> by bit to find it?

The test case should return failure before the fix is applied, and
return pass afterwards.  The harnesses I'm working on do not handle
crashes very well at this point.

If you want to work on some content test cases, take a look at my post
to mozilla.dev.quality on 28-May
(news://news.mozilla.org:119/[hidden email])

and feel free to convert other existing content test cases to use jsunit.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Boris Zbarsky
Dave Liebreich wrote:
> and feel free to convert other existing content test cases to use jsunit.

So is the plan to convert the existing xpcshell-based unit tests to jsunit then?
  Or just have both?

I guess it doesn't matter much as long as one can run "make check" and have it
all Just Work.

-Boris
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
Boris Zbarsky wrote:

> So is the plan to convert the existing xpcshell-based unit tests to
> jsunit then?  Or just have both?

Both for now, since the jsunit harness I've been using is designed to
run as content inside a browser instance.  That's a very different
environment than running straight js code via xpcshell, so having
different harnesses makes sense.

As I commented in the bug you files, I would like to unify the test
function names, so that test code can be portable between harnesses.
But it's more important to write more tests.

See Robert Sayre's feed validator tests in toolkit/components/feeds/test
for an example of another way to write a set of tests using xpcshell.

> I guess it doesn't matter much as long as one can run "make check" and
> have it all Just Work.

Yep.  Whenever possible, tests should be runnable by developers in their
sandbox, runnable using any app build, and runnable on whatever
continuous integration (tinderbox) machines we have set up.
--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
In reply to this post by Dave Liebreich
Robert O'Callahan wrote:
> Darin Fisher wrote:
>> I think that we should try to reach a point where we can require
>> regression/unit tests for every patch that fixes a bug or introduces a
>> new API.
>
> Sounds good to me. Sounds even better if this leverages QA
> staff/community volunteers with a minimum burden on developers :-)

I've never seen that work.  The only thing I've seen work is when the
developers are responsible for writing the bulk of the automated tests.

Developers are not as likely to write testable code if they are not the
ones experiencing the pain of writing tests for that code.

QA staff and community volunteers should be writing additional tests,
and tests of a different type than those written by developers.  But
developers need to step up and write tests.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
In reply to this post by Dave Liebreich
Bernd wrote:

> Darin Fisher wrote:
>
>> I think that we should try to reach a point where we can require
>> regression/unit tests for every patch that fixes a bug or introduces a
>> new API.
>
> There is the cvs textbook example where development speed goes to 0 if
> this is taken too serious. (see Karl Fogel, Producing Open Source
> Software, p.136f - http://www.producingoss.com/)
>

If you are referring to
<http://www.producingoss.com/html-chunk/managing-volunteers.html#automated-testing>,
then I believe you are overzealous in your interpretation of that
anecdote :-)

I challenge you to cc me (:davel) on your next 5 patches, and I'll
attempt to write tests for those changes.  I'm not an experienced
mozilla developer, and we don't yet have wonderful support for
developer-style or unit-style tests, but I wonder how much success I'll
have . . .


--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Robert O'Callahan-3
In reply to this post by Dave Liebreich
Dave Liebreich wrote:

> Robert O'Callahan wrote:
>> Darin Fisher wrote:
>>> I think that we should try to reach a point where we can require
>>> regression/unit tests for every patch that fixes a bug or introduces a
>>> new API.
>>
>> Sounds good to me. Sounds even better if this leverages QA
>> staff/community volunteers with a minimum burden on developers :-)
>
> I've never seen that work.  The only thing I've seen work is when the
> developers are responsible for writing the bulk of the automated tests.
>
> Developers are not as likely to write testable code if they are not the
> ones experiencing the pain of writing tests for that code.

When you say "tests" I'm not sure if you mean test cases or test
automation. Developers must support test automation. But for layout and
rendering code, at least, a single framework can test all kinds of
content, including new kinds of content, so we don't necessarily need to
do anything to make it "testable".

If you mean specifically test cases, then I have to say that I've met
several Mozilla QA volunteers and staff who are far better at writing
test cases than I am. Not to mention the fact that most of the bugs that
I fix come have decent test cases attached before I look at them.

Rob
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Boris Zbarsky
In reply to this post by Dave Liebreich
So I'm looking for a way to add the regression tests for bug 332182 [1] (which
I'm writing now) to some useful regression testing framework.  In particular,
I'm iterating the patch and each time I change things I want to run it against
all the tests; just loading them all one at a time takes a while...

So ideally I'd have the thing run completely automagically, with the popup
blocker disabled because the tests rely on window.open.  Given that they do, I'd
also want to test all three of our window.open modes (new tab, new window, same
window).  I can adjust the tests to dump() out whatever information a framework
is looking for, etc.

Do we have anything at this point that I could drop these tests into?  Or should
I just go on with attaching them to the bug and then we'll try to get back to
them somehow?

-Boris
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

L. David Baron
In reply to this post by Robert O'Callahan-3
On Friday 2006-06-23 12:07 +1200, Robert O'Callahan wrote:

> When you say "tests" I'm not sure if you mean test cases or test
> automation. Developers must support test automation. But for layout and
> rendering code, at least, a single framework can test all kinds of
> content, including new kinds of content, so we don't necessarily need to
> do anything to make it "testable".
>
> If you mean specifically test cases, then I have to say that I've met
> several Mozilla QA volunteers and staff who are far better at writing
> test cases than I am. Not to mention the fact that most of the bugs that
> I fix come have decent test cases attached before I look at them.
FWIW (sort of related):  One thing that occurred to me today is that one
way to write layout and CSS regression tests that *don't* require a
baseline run without the developer's changes is to have tests where the
reference rendering is also in HTML.  Many of Hixie's tests could have a
simple reference rendering written in HTML that would be independent of
the operating system, the user's fonts, and even of many changes in our
default behavior (e.g., changing the ua.css margin on BODY or other such
trivial changes that would require a completely new baseline).

Tests of this type would be easier to hook up to something like
tinderbox that can go red if the tests fail, although they are harder to
write.

-David

--
L. David Baron                                <URL: http://dbaron.org/ >
           Technical Lead, Layout & CSS, Mozilla Corporation

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Christian Biesinger
In reply to this post by Robert O'Callahan-3
Robert O'Callahan wrote:
> If you mean specifically test cases, then I have to say that I've met
> several Mozilla QA volunteers and staff who are far better at writing
> test cases than I am. Not to mention the fact that most of the bugs that
> I fix come have decent test cases attached before I look at them.

It seems to me that there are really two different kinds of test cases
relevant here. There are the layout test cases on the one hand, and unit
tests for most other modules on the other hand.

For layout, all you really need is input, you don't actually write the
test. You just compare before and after (which is, incidentally, a
problem in this algorithm, because it doesn't tell you whether the
testcase is correct, only that it didn't change)

But for other modules, you can't just provide input -- you usually want
to test an API, that it does what it should, that it doesn't crash, etc.
So you need to write some code to do that.

I don't think these two kinds can be considered the same...
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
In reply to this post by Boris Zbarsky
Boris Zbarsky wrote:
> So I'm looking for a way to add the regression tests for bug 332182 [1]
> (which I'm writing now) to some useful regression testing framework.

Cool.

>  In
> particular, I'm iterating the patch and each time I change things I want
> to run it against all the tests; just loading them all one at a time
> takes a while...

Yes.  Clicking on the button also takes up time, I'm guessing.

>
> So ideally I'd have the thing run completely automagically, with the
> popup blocker disabled because the tests rely on window.open.  Given
> that they do, I'd also want to test all three of our window.open modes
> (new tab, new window, same window).  I can adjust the tests to dump()
> out whatever information a framework is looking for, etc.
>
> Do we have anything at this point that I could drop these tests into?  
> Or should I just go on with attaching them to the bug and then we'll try
> to get back to them somehow?

We don't have anything today, but we can have something next week.
Especially if you rewrite the test cases as follows:

Assume there is javascript code that will run with chrome privileges.
This code will

1. use the enumerator-related methods of window-mediator to get the
window object for the first open window (w)

2. invoke w.content.location = "url-of-your-testcase"

3. call w.content.run_test()

4. examine the variable w.content.testResult.  If it is true, then the
test passed.  If false, then the test failed.

the "harness" code can also read a list of urls from a file and treat
each one as a separate test case.

If the test case requires other code to run first (to set preferences,
etc), then that code should be included in a function named setupTest()
defined in the test page.  That code will be run with chrome privs
before run_test() is called.

We can also set up the harness to look for a teardownTest() function to
run after the test completes.

You will be able to invoke the harness from the command line when you
start the app (browser, suite, whatever).  You will also be able to
start the harness from a menu command.

Let's get this running next week at the moco onsite.  I suspect it will
take an hour or two, tops.  And we can incrementally add stuff to the
harness as needed (like helper routines, a standard set of assert*
methods, etc.)


--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
In reply to this post by Christian Biesinger
Christian Biesinger wrote:

> It seems to me that there are really two different kinds of test cases
> relevant here. There are the layout test cases on the one hand, and unit
> tests for most other modules on the other hand.

[...]

> I don't think these two kinds can be considered the same...

That's a really good point.  There may even be more than two types of
test cases that are relevant.  But we should be clear what kind of test
cases we are talking about.  I'll try to do that more explicitly in my
future posts.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Boris Zbarsky
In reply to this post by Dave Liebreich
Dave Liebreich wrote:
> 2. invoke w.content.location = "url-of-your-testcase"
>
> 3. call w.content.run_test()
>
> 4. examine the variable w.content.testResult.  If it is true, then the
> test passed.  If false, then the test failed.

How do I indicate when the test is done?  These tests have async stuff (URL
loads, timeouts, etc).  Examining w.content.testResult right after
w.content.run_test() returns won't give the right answer...

> Let's get this running next week at the moco onsite.  I suspect it will
> take an hour or two, tops.  And we can incrementally add stuff to the
> harness as needed (like helper routines, a standard set of assert*
> methods, etc.)

Sounds good!

Thanks a ton,
-Boris
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
Boris Zbarsky wrote:

> How do I indicate when the test is done?  These tests have async stuff
> (URL loads, timeouts, etc).  Examining w.content.testResult right after
> w.content.run_test() returns won't give the right answer...

Yeah, that complicates things.  But not insurmountably. Your test should
define the variable testComplete (set to false) to indicate that it
contains async stuff, then set the variable to true when all the async
stuff is complete.  The harness will run something every 100ms or so to
check that variable, up to some max timeout value.

>
>> Let's get this running next week at the moco onsite.

[...]

> Sounds good!



--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Boris Zbarsky
Dave Liebreich wrote:
> Yeah, that complicates things.  But not insurmountably. Your test should
> define the variable testComplete (set to false) to indicate that it
> contains async stuff, then set the variable to true when all the async
> stuff is complete.

Sounds like a plan.  I'll try to get these tests converted this weekend so
they're all set to go when we're working on the harness.

Thanks again,
-Boris
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dietrich Ayala-2
In reply to this post by Dave Liebreich
Dave Liebreich wrote:

>
> We don't have anything today, but we can have something next week.
> Especially if you rewrite the test cases as follows:
>
> Assume there is javascript code that will run with chrome privileges.
> This code will
>
> 1. use the enumerator-related methods of window-mediator to get the
> window object for the first open window (w)
>
> 2. invoke w.content.location = "url-of-your-testcase"
>
> 3. call w.content.run_test()
>
> 4. examine the variable w.content.testResult.  If it is true, then the
> test passed.  If false, then the test failed.
>

Is there a bug that this plan correlates to? Bug 329721?

Also, would this framework work for xul files? Eg, if the URL of my testcase is a xul file which
opens new windows, tabs, etc?

-d
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
[I responded in person to Dietrich, and now I'm responding here for
visibility]

Dietrich Ayala wrote:

> Is there a bug that this plan correlates to? Bug 329721?

No, but there could be.  We could use that bug, or file a new one.  Do
you have a preference?

>
> Also, would this framework work for xul files? Eg, if the URL of my
> testcase is a xul file which opens new windows, tabs, etc?

That would be cool.  Let's work on that this week, or at least please
give me an example and I'll pester Neil to help me understand it :-)

>
> -d


--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality