regression testing

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

regression testing

Darin Fisher-2
This came up at today's Gecko 1.9 meeting, and I wanted to raise it
again here for consideration.

I believe that the Mozilla project suffers from in-adequate regression
and unit testing.  We have difficult making large, impactful changes
with confidence.

I think that we should try to reach a point where we can require
regression/unit tests for every patch that fixes a bug or introduces a
new API.

The challenge (as I see it) is getting a testing framework in place
that makes it very easy for developers to contribute new tests.  We'd
also need something that runs these tests regularly.

What else is blocking the adoption of mandated regression/unit tests,
and how do we get to a point where these blockers no longer exist?

-Darin
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
Darin Fisher wrote:

> This came up at today's Gecko 1.9 meeting, and I wanted to raise it
> again here for consideration.
>
> I believe that the Mozilla project suffers from in-adequate regression
> and unit testing.  We have difficult making large, impactful changes
> with confidence.
>
> I think that we should try to reach a point where we can require
> regression/unit tests for every patch that fixes a bug or introduces a
> new API.
>
> The challenge (as I see it) is getting a testing framework in place
> that makes it very easy for developers to contribute new tests.  We'd
> also need something that runs these tests regularly.
>
> What else is blocking the adoption of mandated regression/unit tests,
> and how do we get to a point where these blockers no longer exist?
>
> -Darin

We have a number of the building blocks for this - I had been listing
them at
<http://wiki.mozilla.org/SoftwareTesting:Catalog_of_Automated_Tests>
and the parent page <http://wiki.mozilla.org/SoftwareTesting>.  Those
pages are due for an update.

See also some of the recent postings to mozilla.dev.quality.

I think we need a few harnesses, and a little bit more work to hook them
up.  But what we need most of all is a developer (or developers) willing
to write tests for their code, and, at the same time, help work on the
last bits of the harnesses to run their tests.

-dave


--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

dolphinling-2
Dave Liebreich wrote:

> Darin Fisher wrote:
>> This came up at today's Gecko 1.9 meeting, and I wanted to raise it
>> again here for consideration.
>>
>> I believe that the Mozilla project suffers from in-adequate regression
>> and unit testing.  We have difficult making large, impactful changes
>> with confidence.
>>
>> I think that we should try to reach a point where we can require
>> regression/unit tests for every patch that fixes a bug or introduces a
>> new API.
>>
>> The challenge (as I see it) is getting a testing framework in place
>> that makes it very easy for developers to contribute new tests.  We'd
>> also need something that runs these tests regularly.
>>
>> What else is blocking the adoption of mandated regression/unit tests,
>> and how do we get to a point where these blockers no longer exist?
>>
>> -Darin
>
> We have a number of the building blocks for this - I had been listing
> them at
> <http://wiki.mozilla.org/SoftwareTesting:Catalog_of_Automated_Tests> and
> the parent page <http://wiki.mozilla.org/SoftwareTesting>.  Those pages
> are due for an update.
>
> See also some of the recent postings to mozilla.dev.quality.
>
> I think we need a few harnesses, and a little bit more work to hook them
> up.  But what we need most of all is a developer (or developers) willing
> to write tests for their code, and, at the same time, help work on the
> last bits of the harnesses to run their tests.
>
> -dave

I'm not a developer, and I certainly won't be fixing many bugs any time
soon, but I wouldn't mind helping out here off and on if it'd ease the
load on developers. I think it should definitely be set up so that it
doesn't *have* to be the person who fixed the bug that writes the test.

As a side note, this would be like writing testcases, right? Except
within a framework, and easier than a testcase because the problem's
already been identified, and you don't have to go through removing bit
by bit to find it?

--
dolphinling
<http://dolphinling.net/>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
dolphinling wrote:

> I'm not a developer, and I certainly won't be fixing many bugs any time
> soon, but I wouldn't mind helping out here off and on if it'd ease the
> load on developers. I think it should definitely be set up so that it
> doesn't *have* to be the person who fixed the bug that writes the test.

If the folks who write the code do not write the tests, then this effort
won't scale.  Especially with unit-style tests, for which we need a lot
of harness work.

I encourage others to write additional tests, and I'm glad to have you
help out.

>
> As a side note, this would be like writing testcases, right? Except
> within a framework, and easier than a testcase because the problem's
> already been identified, and you don't have to go through removing bit
> by bit to find it?

The test case should return failure before the fix is applied, and
return pass afterwards.  The harnesses I'm working on do not handle
crashes very well at this point.

If you want to work on some content test cases, take a look at my post
to mozilla.dev.quality on 28-May
(news://news.mozilla.org:119/[hidden email])

and feel free to convert other existing content test cases to use jsunit.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Boris Zbarsky
Dave Liebreich wrote:
> and feel free to convert other existing content test cases to use jsunit.

So is the plan to convert the existing xpcshell-based unit tests to jsunit then?
  Or just have both?

I guess it doesn't matter much as long as one can run "make check" and have it
all Just Work.

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
Boris Zbarsky wrote:

> So is the plan to convert the existing xpcshell-based unit tests to
> jsunit then?  Or just have both?

Both for now, since the jsunit harness I've been using is designed to
run as content inside a browser instance.  That's a very different
environment than running straight js code via xpcshell, so having
different harnesses makes sense.

As I commented in the bug you files, I would like to unify the test
function names, so that test code can be portable between harnesses.
But it's more important to write more tests.

See Robert Sayre's feed validator tests in toolkit/components/feeds/test
for an example of another way to write a set of tests using xpcshell.

> I guess it doesn't matter much as long as one can run "make check" and
> have it all Just Work.

Yep.  Whenever possible, tests should be runnable by developers in their
sandbox, runnable using any app build, and runnable on whatever
continuous integration (tinderbox) machines we have set up.
--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Mike Shaver
In reply to this post by Darin Fisher-2
On 6/21/06, Darin Fisher <[hidden email]> wrote:

> I think that we should try to reach a point where we can require
> regression/unit tests for every patch that fixes a bug or introduces a
> new API.
>
> The challenge (as I see it) is getting a testing framework in place
> that makes it very easy for developers to contribute new tests.  We'd
> also need something that runs these tests regularly.
>
> What else is blocking the adoption of mandated regression/unit tests,
> and how do we get to a point where these blockers no longer exist?

FWIW, there was a good thread in dev.quality about some of these
issues a while back, and I think there are people there who are
working on tools for these very issues.  People might find it
informative to dig through the archives there.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Robert O'Callahan-3
In reply to this post by Darin Fisher-2
Darin Fisher wrote:
> I believe that the Mozilla project suffers from in-adequate regression
> and unit testing.  We have difficult making large, impactful changes
> with confidence.

Actually, I think we've done surprisingly well at making a large number
of large, impactful changes.

> I think that we should try to reach a point where we can require
> regression/unit tests for every patch that fixes a bug or introduces a
> new API.

Sounds good to me. Sounds even better if this leverages QA
staff/community volunteers with a minimum burden on developers :-)

Rob
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Bernd-13
In reply to this post by Darin Fisher-2
Darin Fisher wrote:

> I think that we should try to reach a point where we can require
> regression/unit tests for every patch that fixes a bug or introduces a
> new API.

There is the cvs textbook example where development speed goes to 0 if
this is taken too serious. (see Karl Fogel, Producing Open Source
Software, p.136f - http://www.producingoss.com/)

Bernd




_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
In reply to this post by Robert O'Callahan-3
Robert O'Callahan wrote:
> Darin Fisher wrote:
>> I think that we should try to reach a point where we can require
>> regression/unit tests for every patch that fixes a bug or introduces a
>> new API.
>
> Sounds good to me. Sounds even better if this leverages QA
> staff/community volunteers with a minimum burden on developers :-)

I've never seen that work.  The only thing I've seen work is when the
developers are responsible for writing the bulk of the automated tests.

Developers are not as likely to write testable code if they are not the
ones experiencing the pain of writing tests for that code.

QA staff and community volunteers should be writing additional tests,
and tests of a different type than those written by developers.  But
developers need to step up and write tests.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
In reply to this post by Darin Fisher-2
Bernd wrote:

> Darin Fisher wrote:
>
>> I think that we should try to reach a point where we can require
>> regression/unit tests for every patch that fixes a bug or introduces a
>> new API.
>
> There is the cvs textbook example where development speed goes to 0 if
> this is taken too serious. (see Karl Fogel, Producing Open Source
> Software, p.136f - http://www.producingoss.com/)
>

If you are referring to
<http://www.producingoss.com/html-chunk/managing-volunteers.html#automated-testing>,
then I believe you are overzealous in your interpretation of that
anecdote :-)

I challenge you to cc me (:davel) on your next 5 patches, and I'll
attempt to write tests for those changes.  I'm not an experienced
mozilla developer, and we don't yet have wonderful support for
developer-style or unit-style tests, but I wonder how much success I'll
have . . .


--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Robert O'Callahan-3
In reply to this post by Dave Liebreich
Dave Liebreich wrote:

> Robert O'Callahan wrote:
>> Darin Fisher wrote:
>>> I think that we should try to reach a point where we can require
>>> regression/unit tests for every patch that fixes a bug or introduces a
>>> new API.
>>
>> Sounds good to me. Sounds even better if this leverages QA
>> staff/community volunteers with a minimum burden on developers :-)
>
> I've never seen that work.  The only thing I've seen work is when the
> developers are responsible for writing the bulk of the automated tests.
>
> Developers are not as likely to write testable code if they are not the
> ones experiencing the pain of writing tests for that code.

When you say "tests" I'm not sure if you mean test cases or test
automation. Developers must support test automation. But for layout and
rendering code, at least, a single framework can test all kinds of
content, including new kinds of content, so we don't necessarily need to
do anything to make it "testable".

If you mean specifically test cases, then I have to say that I've met
several Mozilla QA volunteers and staff who are far better at writing
test cases than I am. Not to mention the fact that most of the bugs that
I fix come have decent test cases attached before I look at them.

Rob
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

L. David Baron
On Friday 2006-06-23 12:07 +1200, Robert O'Callahan wrote:

> When you say "tests" I'm not sure if you mean test cases or test
> automation. Developers must support test automation. But for layout and
> rendering code, at least, a single framework can test all kinds of
> content, including new kinds of content, so we don't necessarily need to
> do anything to make it "testable".
>
> If you mean specifically test cases, then I have to say that I've met
> several Mozilla QA volunteers and staff who are far better at writing
> test cases than I am. Not to mention the fact that most of the bugs that
> I fix come have decent test cases attached before I look at them.
FWIW (sort of related):  One thing that occurred to me today is that one
way to write layout and CSS regression tests that *don't* require a
baseline run without the developer's changes is to have tests where the
reference rendering is also in HTML.  Many of Hixie's tests could have a
simple reference rendering written in HTML that would be independent of
the operating system, the user's fonts, and even of many changes in our
default behavior (e.g., changing the ua.css margin on BODY or other such
trivial changes that would require a completely new baseline).

Tests of this type would be easier to hook up to something like
tinderbox that can go red if the tests fail, although they are harder to
write.

-David

--
L. David Baron                                <URL: http://dbaron.org/ >
           Technical Lead, Layout & CSS, Mozilla Corporation

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Christian Biesinger
In reply to this post by Robert O'Callahan-3
Robert O'Callahan wrote:
> If you mean specifically test cases, then I have to say that I've met
> several Mozilla QA volunteers and staff who are far better at writing
> test cases than I am. Not to mention the fact that most of the bugs that
> I fix come have decent test cases attached before I look at them.

It seems to me that there are really two different kinds of test cases
relevant here. There are the layout test cases on the one hand, and unit
tests for most other modules on the other hand.

For layout, all you really need is input, you don't actually write the
test. You just compare before and after (which is, incidentally, a
problem in this algorithm, because it doesn't tell you whether the
testcase is correct, only that it didn't change)

But for other modules, you can't just provide input -- you usually want
to test an API, that it does what it should, that it doesn't crash, etc.
So you need to write some code to do that.

I don't think these two kinds can be considered the same...
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Dave Liebreich
Christian Biesinger wrote:

> It seems to me that there are really two different kinds of test cases
> relevant here. There are the layout test cases on the one hand, and unit
> tests for most other modules on the other hand.

[...]

> I don't think these two kinds can be considered the same...

That's a really good point.  There may even be more than two types of
test cases that are relevant.  But we should be clear what kind of test
cases we are talking about.  I'll try to do that more explicitly in my
future posts.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Bernd-13
In reply to this post by Dave Liebreich

> If you are referring to
> <http://www.producingoss.com/html-chunk/managing-volunteers.html#automated-testing>,
> then I believe you are overzealous in your interpretation of that
> anecdote :-)

I am just concerned that I see the next wave of bureocratic additional
payload where one is required to write a test case in a certain format
before one can checkin a line. Layout tests up to now could easily use
the minimal test cases from the bugs. However they are useful only for
comparative testing. What we have now, has major issues, either the
number of false positives is to large or only the visible part of the
window is considered. Writing test cases like David proposed (one
example would be the table dom  test cases -
http://lxr.mozilla.org/seamonkey/source/layout/html/tests/table/dom/ 
seems to be feasible but writing nice and nifty test case in the Hixie
style of obvious pass/fail is an art and will require too much resources.

Bernd
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: regression testing

Boris Zbarsky
In reply to this post by Dave Liebreich
Bernd wrote:
> Layout tests up to now could easily use the minimal test cases from the bugs.

This is something we want to maintain (though we should be checking in both the
minimal testcase and whatever non-minimal things got attached to the bug too, imo).

On the other hand, for DOM bugs having a way to write a testcase that can just
set some variables to indicate success/failure is often simpler than trying to
indicate those in a human-visible way.  If we settle on a known format for DOM
testcases, it'd be pretty easy to just create the minimal DOM testcases in that
format to start with, I would think.

> writing nice and nifty test case in the Hixie
> style of obvious pass/fail is an art and will require too much resources.

Yes, agreed.  But note again that that style is targeted at making it obvious to
a _human_ that the testcase passed or failed.  Again, for testcases where we're
testing what data structures some script produces (unit tests, most DOM tests,
etc) it's a lot easier to indicate success/fail to a framework than to a human.
  Then let the framework handle the human-centric presentation in a single
centralized place where the human can see, at a glance, exactly which tests
failed, etc.

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning