Where to put automated tests and testing tools

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Where to put automated tests and testing tools

Dave Liebreich
I think we have enough stuff (test cases, test tools, aux files) to
warrant placing it in source control.  Somewhere.

QA and dev folks need to put stuff in.  QA, dev, and tinderboxes need to
check stuff out.

I think we could use either the main cvs repository, or the private mofo
one.

Using the main cvs repository is good in that everyone can access it,
but bad in that not all folks we might want to grant check-in privs for
tests are folks we want to grant access to the entire repository.

Using the mofo repository is good in that the tinderbox configs and
release scripts live there, but bad in that it might be a pain to have
tests and code in separate repositories.

I suggest we use subdirectories of mozilla/tools/test-harness (already
exists) for test tools such as the visual regression test extension, put
test cases in a ./test subdirectory where the code under test lives, and
put suites (lists of test cases) in mozilla/tools/test-suites.

I also suggest creating a cvs module (or some other mechanism) for
checking out just the tests and packaging them up for "installation"
somewhere else.

Thoughts?

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Benjamin Smedberg
Dave Liebreich wrote:

> Using the main cvs repository is good in that everyone can access it,
> but bad in that not all folks we might want to grant check-in privs for
> tests are folks we want to grant access to the entire repository.

We absolutely should be using the main repository. We must allow everyone to
  pull the tests and run them. I cannot imagine a situation where we
wouldn't grant checkin privileges to appropriate QA developers.

Besides which, tests should be committed in appropriate source code
directories along with the module sources, which allows for branching the
tests along with the main source tree.

> I suggest we use subdirectories of mozilla/tools/test-harness (already
> exists) for test tools such as the visual regression test extension, put
> test cases in a ./test subdirectory where the code under test lives, and
> put suites (lists of test cases) in mozilla/tools/test-suites.

Testcases suites should probably live in the source tree as well.

> I also suggest creating a cvs module (or some other mechanism) for
> checking out just the tests and packaging them up for "installation"
> somewhere else.

client.mk rules the roost for this task.

--BDS
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

J. Paul Reed
In reply to this post by Dave Liebreich
Dave Liebreich wrote:

> Using the mofo repository is good in that the tinderbox configs and
> release scripts live there, but bad in that it might be a pain to have
> tests and code in separate repositories.

Tests and code should go in the same repo; the tinderbox configs are
only in the MoFo repo because it was easier to get them in there first;
they're slowly-but-surely moving into the main repo (bug 337362).

> I suggest we use subdirectories of mozilla/tools/test-harness (already
> exists) for test tools such as the visual regression test extension, put
> test cases in a ./test subdirectory where the code under test lives, and
> put suites (lists of test cases) in mozilla/tools/test-suites.

A couple of possibilities here:

Create a new mozilla/qa area for CVS, and move all QA/testing-related
things here. mozilla/tools has become a dumping ground for "whatever
goes," and it would be nice to put all testing-related things together
in their own area (mozilla/tools contains, for instance, the tinderbox
client code, some CVS cross commit stuff, the codesighs stuff, etc.,
etc., etc.)

Wherever you put it, you might also consider creating a X/Y structure,
where X is either mozilla/qa or mozilla/tools/test[s|ing] and Y is
harness, suites, tests, etc. (this, as opposed to
mozilla/tools/test-harness, mozilla/tools/test-suites,
mozilla/tools/test-foo).

> I also suggest creating a cvs module (or some other mechanism) for
> checking out just the tests and packaging them up for "installation"
> somewhere else.

Smedberg weighed in on this one; I'm inclined to agree, although I think
we under-utilize CVS's built-in method of managing that stuff (i.e. we
don't really use modules, when we easily could).

Later,
preed
--
J. Paul Reed
Build/Release Engineer - The Mozilla Corporation
smtp://[hidden email]
irc://irc.mozilla.org/preed
pots://650.903.0800/x256
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Benjamin Smedberg
J. Paul Reed wrote:

>> I also suggest creating a cvs module (or some other mechanism) for
>> checking out just the tests and packaging them up for "installation"
>> somewhere else.
>
> Smedberg weighed in on this one; I'm inclined to agree, although I think
> we under-utilize CVS's built-in method of managing that stuff (i.e. we
> don't really use modules, when we easily could).

The problem with CVS modules is that the module definitions themselves are
not versioned, which means that we can never remove anything from a module
definition because that would break historical checkouts. This has been an
acute problem in the past with the SeaMonkeyAll module, which is why we have
moved away (finally) from using CVS modules in the checkout process at all.

--BDS
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

J. Paul Reed
Benjamin Smedberg wrote:

> The problem with CVS modules is that the module definitions themselves
> are not versioned, which means that we can never remove anything from a
> module definition because that would break historical checkouts. This
> has been an acute problem in the past with the SeaMonkeyAll module,
> which is why we have moved away (finally) from using CVS modules in the
> checkout process at all.

It took me awhile to figure out what you meant by the module definitions
not being versioned, since CVSROOT/modules is versioned.

Yeah, I could see that... although, I think we limit our use of that
functionality in restrictive ways, due to thinking about the repository
based upon old restrictions.

For certain "mini-products" that aren't likely to (or won't, by policy)
ever change sub-directory definitions, I think modules are useful. A
new, top-level qa directory would be an example of this. It would nice
to be able to say "cvs co tests" and get all the relevant directories.

I've just been wrestling with trying to use client.mk's structure for
checking out things other than Firefox/Thunderbird/etc., and it hasn't
been a pleasant experience. Trying to tell client.mk I *don't*
(necessarily) want NSPR, the C LDAP SDK, and NSS, but do want the root
level files has been... annoying. (Yeah, I could hack it, and will
probably end up doing so, but... that's kinda messy. :-/

Later,
preed
--
J. Paul Reed
Build/Release Engineer - The Mozilla Corporation
smtp://[hidden email]
irc://irc.mozilla.org/preed
pots://650.903.0800/x256
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Dave Liebreich
In reply to this post by J. Paul Reed

> Create a new mozilla/qa area for CVS, and move all QA/testing-related
> things here.

I like mozilla/<something>.  I don't like "qa" for a variety of reasons,
but I can't think of anything better.  preed suggested "quality", and
I'm considering "tests", "teststuff", "Lt. Col. James Burton" :-)


> Wherever you put it, you might also consider creating a X/Y structure

Yeah.  Let's do that.  we can have ./tests for tests that are not
necessarily associated with specific areas of code, ./harnesses for
harness code, ./third-party for trees we import from other places (like
maybe jsunit), ./suites for test suite files, and others as needed.

Note: I think we should create test suite files (lists of tests to run)
in the code dirs, too.  mozilla/<foo>/suites can be for things like
"suite that runs after every build", "suite that runs once per day",
"suite that runs during the release process", etc.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Dave Liebreich
Dave Liebreich wrote:
>
>> Create a new mozilla/qa area for CVS, and move all QA/testing-related
>> things here.
>
> I like mozilla/<something>.  I don't like "qa" for a variety of reasons,
> but I can't think of anything better.  preed suggested "quality", and
> I'm considering "tests", "teststuff", "Lt. Col. James Burton" :-)

I thought of this on the drive home:  mozilla/q

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Dave Liebreich
In reply to this post by Dave Liebreich
Mike Shaver wrote:
> On 7/5/06, Dave Liebreich <[hidden email]> wrote:
>> Yeah.  Let's do that.  we can have ./tests for tests that are not
>> necessarily associated with specific areas of code, ./harnesses for
>> harness code, ./third-party for trees we import from other places (like
>> maybe jsunit), ./suites for test suite files, and others as needed.
>
> What tests would not be associated with specific code (or apps)?

I was thinking that the tests should go in the nearest common parent
directory of the code under test.  Sometimes that model does not fit
exactly, so those tests that do not fit the model should go in
mozilla/testing/tests.  The Tp and Ts tests probably should go in there,
for example.  Also maybe eggplant scripts.

As we make our codebase more loosely coupled, tests in this common
location may migrate to, or be replaced by tests in, the code directory
hierarchy.

>
> I suggest "mozilla/testing" for the infrastructure,

+1

> since this seems
> to be things that are all related to testing, and I share your
> objection to "mozilla/qa"!

Should mozilla/testing be its own module?  (I think so)

If so, who should be the module owner? (I volunteer, as long as the
commit/review policy is not restrictive)

Peers?  Members? (moco qa and build, for a start)

Bugzilla product/component? (I suggest a product named "testing", with
components)

>
>> Note: I think we should create test suite files (lists of tests to run)
>> in the code dirs, too.  mozilla/<foo>/suites can be for things like
>> "suite that runs after every build", "suite that runs once per day",
>> "suite that runs during the release process", etc.
>
> If they're going in the code directories, they shouldn't be called
> "suites".  "test-suites", or even just .test-suite files in the tests/
> directory will be much clearer for everyone involved.  Remember, even
> developers have to be able to understand this system. :)

By <foo>, I meant "testing".  Sorry for not being clear.

An example: mozilla/toolkit/component/feeds/test/all.test-suite would
contain a list of all the feed tests.  mozilla/testing/suites/release
would contain "mozilla/toolkit/component/feeds/test/all.test-suite" as
an entry.  Details will vary (as little as possible) by harness.

> "mozilla/q" is a nonstarter, but I'm pretty sure you were joking. :)

Yeah.  Though I do like the Ian Flemming reference.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Mike Shaver
On 7/6/06, Dave Liebreich <[hidden email]> wrote:
> I was thinking that the tests should go in the nearest common parent
> directory of the code under test.  Sometimes that model does not fit
> exactly, so those tests that do not fit the model should go in
> mozilla/testing/tests.  The Tp and Ts tests probably should go in there,
> for example.  Also maybe eggplant scripts.

I think Ts and Tp would go next to the app that they're testing, and
eggplant scripts next to either the app or the component that they're
testing, at the discretion of the module owner for the code under
test.  Ts and Tp are really pretty much functional tests at this point
(Tr less so), so they seem to fit well with the app whose function
we're evaluating in the large.

So I think maybe I'm saying "put tests next to the 'most-derived'
build product that is needed to run the tests", where "built later" is
probably a decent interpretation of "most-derived" for our purposes.
As we refactor, some things that depend on An App will come to depend
on some smaller/less-derived (earlier-built!) module instead, and can
be moved to be beside those instead.

This seems to fit better with the idea of the tests and code being
closely linked, to me, and therefore under the purview of the same
module owners.  I think that linkage has much virtue!

mozilla/testing is a great place for harnesses and other tools,
though, and I would personally be happy to have you act as module
owner.  (As owner you can set the review policies as you choose,
largely, so you would only have yourself to blame if they were too
onerous. :) )

Mike
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Axel Hecht-2
In reply to this post by Dave Liebreich
Dave Liebreich wrote:
> Mike Shaver wrote:

<...>
>
>> "mozilla/q" is a nonstarter, but I'm pretty sure you were joking. :)
>
> Yeah.  Though I do like the Ian Flemming reference.
>

I was thinking of an almighty something with a bad sense of humor,
though, being closer to star trek than 007.

Axel
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Rob Campbell-3
In reply to this post by Dave Liebreich
Hi. I've been thinking about this too as I've been looking at some
different unit testing frameworks and had been tentatively putting a
framework into a mozilla/xunit directory. I think ./testing would work
well though.

Most of the frameworks I've seen, with the exception of MinUnit have
subdirectories associated with them. E.g., mozilla/testing/cppunit/.
That's not really a bad thing, but does make the nesting a bit deeper.
Having two parent directories, like mozilla/test/unit-tests would just
compound that.

Dave Liebreich wrote:
> Mike Shaver wrote:
> > I suggest "mozilla/testing" for the infrastructure,
>
> +1

I'll put in a vote for this as well.

> > since this seems
> > to be things that are all related to testing, and I share your
> > objection to "mozilla/qa"!
>
> Should mozilla/testing be its own module?  (I think so)

A separate CVS module? Maybe, but if we're writing and releasing unit
tests alongside the application code, that could create more problems
than it would solve. Is there any reason not to include the testing
subdirectory with a full source tree?

> If so, who should be the module owner? (I volunteer, as long as the
> commit/review policy is not restrictive)

> Bugzilla product/component? (I suggest a product named "testing", with
> components)

That seems reasonable.

[...]

> > "mozilla/q" is a nonstarter, but I'm pretty sure you were joking. :)
>
> Yeah.  Though I do like the Ian Flemming reference.

moz/q would be less typing ;)

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Dave Liebreich

>> Should mozilla/testing be its own module?  (I think so)
>
> A separate CVS module?

No.  Not CVS module.  Mozilla module
(http://www.mozilla.org/hacking/module-ownership.html)

Sorry I wasn't more explicit.  "module" is one of those
heavily-overloaded terms.  Like "test" :-)

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Rob Campbell-3

Dave Liebreich wrote:
> >> Should mozilla/testing be its own module?  (I think so)
> >
> > A separate CVS module?
>
> No.  Not CVS module.  Mozilla module
> (http://www.mozilla.org/hacking/module-ownership.html)
>
> Sorry I wasn't more explicit.  "module" is one of those
> heavily-overloaded terms.  Like "test" :-)

Well in that case, I volunteer to own that module as well. Maybe we
could share the responsibility of it, since there will be sections that
you'll be maintaining, (test harness stuff) and sections I'll be
maintaining. (unit testing frameworks)

Does this make sense?

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Where to put automated tests and testing tools

Dave Liebreich
In reply to this post by Dave Liebreich
I filed Bug 343882 to track these tasks.

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality