Test Automation

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Test Automation

Dave Liebreich
Bob Clary wrote:

> I think you misunderstood me. I am not advocating the universal use of
> centralized web servers to serve tests. I do think there should be
> centralized results gathering and reporting servers though.

I agree.  I did not think we were discussing results gathering and
reporting.

>
> Currently each test machine hosts a copy of the tests and has a local
> virtual web server test.mozilla.com at 127.0.0.1

This means that a developer who wants to run these tests on his or her
own machine must first configure apache.  That might be too much to ask.

>
> There will be a need for special server environments which would
> possibly be better hosted at MoCo due to special or bizarre
> configurations, but those aren't the normal case.

I'd like to see reduced test cases that embody the defect and can be
reproduced with a captive web server (like one based on
HTTP::Server::Simple).  We can and should still use the full end-to-end
stuff in our hosted test environments so we can find other bugs.

>
> I don't know anything about HTTP::Server::Simple but if it can do
> everything that is required, I don't see why not but then I don't see
> the advantage over Apache either. I would prefer to have a single type
> of server that is easy to setup and maintain which is why I use Apache
> on all of the machines.

The advantage of a captive web server is that it requires zero setup and
maintenance by the person running the test.

Do you know of a particular test case or bug # that I could use as an
example and build a captive-web-server-based test script around?

--
Dave Liebreich
Test Architect, Mozilla Corporation
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Automation

Bob Clary
Dave Liebreich wrote:

>>
>> Currently each test machine hosts a copy of the tests and has a local
>> virtual web server test.mozilla.com at 127.0.0.1
>
> This means that a developer who wants to run these tests on his or her
> own machine must first configure apache.  That might be too much to ask.
>

Could be too much. Lets see what they say here ;-)

Another issue that I left out completely that might require centralized
servers is dealing with mail|news.

>>
>
> The advantage of a captive web server is that it requires zero setup and
> maintenance by the person running the test.
>
> Do you know of a particular test case or bug # that I could use as an
> example and build a captive-web-server-based test script around?
>

The current js browser and dom tests require a web server. Nothing
special about the web server requirements for the js browser, but the
dom tests require svg mime types.

For some other examples ping me on irc.

/bc
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Automation

beaufour@gmail.com
On 5/9/06, Bob Clary <[hidden email]> wrote:

> Dave Liebreich wrote:
>
> >>
> >> Currently each test machine hosts a copy of the tests and has a local
> >> virtual web server test.mozilla.com at 127.0.0.1
> >
> > This means that a developer who wants to run these tests on his or her
> > own machine must first configure apache.  That might be too much to ask.
> >
>
> Could be too much. Lets see what they say here ;-)

I would have no problem with it, but I'm not the right one to ask :)
But I think that is it too much. Do people in general have perl
installed? I'm more in favor of the simple perl server approach.

--
... Allan
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Automation

Mike Shaver
In reply to this post by Bob Clary
On 5/6/06, [hidden email] <[hidden email]> wrote:
> As some of you may already know, I have been working on automating test
> execution for Firefox for some time but have not made the project or
> work public before now.

Great, *great* to see this discussion and work happening.

I am going to be lazy and reply here rather than to specific messages,
because my intentions of doing the latter have led to me not replying
so far.  I hope it's still helpful.

- I think requiring Apache is probably a bad idea.  It's hard to get
it configured just the right way, especially if you already have an
Apache setup on your system (as with Linux and OS X boxes), and you
have to make sure the right things happen with system names for
redirects, and MIME types, and CGI paths, and error logging, and such.
 It picks up a lot of stuff from its environment, and that will tend
to make the tests less reliable, IMO.  A perl or python self-contained
server would work pretty well for almost everything, I think.  (I
looked hard at the Apache route when I was building web administration
tools in a recently-previous life, and decided instead to use the
Python http-server module instead, because of these issues -- and I
was targetting high-end sysadmin types, not mere developers! :) )

- I think these are the questions that developers will ask, and we
should work hard to make the answers simple and pleasant:

1) How do I run all/some tests against my Firefox build?

A good answer would be something like "from your objdir, run
`./test-firefox.sh dom js-suite xslt cache:bug-122233-regress` and
look for "NEW FAIL" messages".

2) How do I run a specific test?

(I used the "cache:bug-122233-regress" thing above.  A way to point to
a specific pathname would be very nice too.)

3) How do I add a test to the suite?

A few different types of tests that might need to be added, for which
we should have some pat answers:

- "I have an HTML page that the browser needs to load"
- "I have a script to run in xpcshell"
- "I have a shell script to run"

We should have standard and simple ways of indicating "PASSED",
"FAILED: details", "CAN'T RUN" (the latter for things like "running
Wibble tests, but build has it disabled") for each of those cases,
which are picked up by the "templates" or "subharnesses" or
whathaveyou.

A JS library (that works in content and chrome) to mechanize things
like "green or red?" or other such visual tests would help a lot too,
and I bet we have clever people following this thread who would love
to write that sort of thing.

Once we get to the point that we have good answers for those
questions, I think we will be able to add some additional automation,
centralized collection/republishing (I like the Atom idea!) and such.

I hope that's helpful in some way.  I suspect people have already
figured a lot of this stuff out, but I thought I'd share my thoughts
nonetheless.

Mike
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
12