Test coverage is 18.4%

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Test coverage is 18.4%

Gabor Szabo
I am not sure if people here are doing this but I was curious about
the test coverage of Bugzilla.
So I created a Makefile with the following content:

test:
        ./runtests.pl


and ran the following:

cover -delete
export DEVEL_COVER_OPTIONS="-ignore,perl5lib"
HARNESS_PERL_SWITCHES=-MDevel::Cover make test
cover -report html

According to this,

All tests successful.
Files=12, Tests=4066, 129 wallclock secs ( 0.64 usr  0.06 sys + 122.84
cusr  6.04 csys = 129.58 CPU)

and the total test coverage of Bugzilla is 18.4%

Are there tests other than those in t/*?

Gabor
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Gervase Markham
On 12/02/10 06:13, Gabor Szabo wrote:
> Are there tests other than those in t/*?

Nope.

I don't believe full code coverage has ever been an explicit goal for
our tests. Not that it's not a good thing - clearly it would be! - but
Bugzilla was not developed in a TDD manner. And no-one has yet seen it
as high priority to write the enormous number of tests it would require
to get very high coverage.

Gerv
_______________________________________________
dev-apps-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-bugzilla
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Byron Jones-4
On 12/02/10 06:13, Gabor Szabo wrote:
> Are there tests other than those in t/*?
>    
there's selenium test cases; https://wiki.mozilla.org/Bugzilla:QA



-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Gervase Markham
In reply to this post by Gervase Markham
On 12/02/10 11:01, Byron Jones wrote:
> On 12/02/10 06:13, Gabor Szabo wrote:
>> Are there tests other than those in t/*?
>>    
> there's selenium test cases; https://wiki.mozilla.org/Bugzilla:QA

My apologies; I had forgotten about these.

Gerv

_______________________________________________
dev-apps-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-bugzilla
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Max Kanat-Alexander
In reply to this post by Gabor Szabo
On 02/11/2010 10:13 PM, Gabor Szabo wrote:
> and the total test coverage of Bugzilla is 18.4%

        Yeah, just to say again what others have said, this is a problem, for sure.

> Are there tests other than those in t/*?

        There are the Selenium tests that Byron pointed out--those are our main
method of QA at the moment.

        The nice part about the Selenium tests is that they make sure that the
actual UI and the JavaScript in Bugzilla works. That is, we wouldn't
really have "full coverage" unless we tested those as well.

        Having unit tests for every single method and function in Bugzilla
would be great, though. The fact is, the lack of TDD in our development
process is one of the things that makes the review process so difficult.
But the number of tests that we'd have to write to truly have full
coverage is also something that's nearly overwhelming for us, because we
just don't have the current manpower to do it.

        If somebody wanted to work on getting Bugzilla fully unit-tested, I'd
be behind it 100% of the way. (Note that to get really full coverage on
the Perl code, we'd have to move all the .cgi code into Controller
modules, but that's something that I've want to do  anyway.)

        -Max
--
http://www.everythingsolved.com/
Competent, Friendly Bugzilla and Perl Services. Everything Else, too.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

David Marshall-5



On 2/12/10 1:54 PM, "Max Kanat-Alexander" <[hidden email]> wrote:

> On 02/11/2010 10:13 PM, Gabor Szabo wrote:
>> and the total test coverage of Bugzilla is 18.4%
>
> Yeah, just to say again what others have said, this is a problem, for sure.
>

I don't think it's a problem at all, because the t/* tests were never, as
far as I am aware, intended to be testing for coverage.  I'm rather
surprised that the number is as high as 18.4%!

At Yahoo!, where we've been putting a lot of effort into continuous
integration, I use t/* as a "commit test" - committing to the repository
triggers an automatic run of t/*.  The idea is to do a quick check that a
commit hasn't horked anything - the Perl compiles, etc.

We also have a home-grown test suite that we use as a "smoke test" - a
nightly test that interacts with Bugzilla through apache and a working
database.  It takes about twenty minutes to run at present.  That's what I
want to have 100% coverage.  Currently, we're at about 60% or so.

We have our own Selenium tests written as well, but I haven't included them
in our continuous integration implementation.

-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

ahdevans
In reply to this post by Max Kanat-Alexander
The biggest problem I've found trying to go back and fill in unit tests is you can either cover the obvious things that won't break unless there are API changes, but the type of stuff that more often breaks can't often be deduced from the code (not to mention you can't find existing bugs when the implementation is your spec.

On Fri, Feb 12, 2010 at 1:54 PM, Max Kanat-Alexander <[hidden email]> wrote:
On 02/11/2010 10:13 PM, Gabor Szabo wrote:
> and the total test coverage of Bugzilla is 18.4%

       Yeah, just to say again what others have said, this is a problem, for sure.

> Are there tests other than those in t/*?

       There are the Selenium tests that Byron pointed out--those are our main
method of QA at the moment.

       The nice part about the Selenium tests is that they make sure that the
actual UI and the JavaScript in Bugzilla works. That is, we wouldn't
really have "full coverage" unless we tested those as well.

       Having unit tests for every single method and function in Bugzilla
would be great, though. The fact is, the lack of TDD in our development
process is one of the things that makes the review process so difficult.
But the number of tests that we'd have to write to truly have full
coverage is also something that's nearly overwhelming for us, because we
just don't have the current manpower to do it.

       If somebody wanted to work on getting Bugzilla fully unit-tested, I'd
be behind it 100% of the way. (Note that to get really full coverage on
the Perl code, we'd have to move all the .cgi code into Controller
modules, but that's something that I've want to do  anyway.)

       -Max
--
http://www.everythingsolved.com/
Competent, Friendly Bugzilla and Perl Services. Everything Else, too.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=ahdevans@...>

Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Max Kanat-Alexander
In reply to this post by David Marshall-5
On 02/12/2010 02:04 PM, David Marshall wrote:
> I don't think it's a problem at all, because the t/* tests were never, as
> far as I am aware, intended to be testing for coverage.  I'm rather
> surprised that the number is as high as 18.4%!

        They probably weren't intended that way, but I'd like to see them be
more so. Test-Driven Development didn't exist as a popular philosophy
(though it should have been realized as simply a sensible thing to
do...) when the tests were originally written, so they probably just
weren't aimed in that direction.

> We also have a home-grown test suite that we use as a "smoke test" - a
> nightly test that interacts with Bugzilla through apache and a working
> database.  It takes about twenty minutes to run at present.  That's what I
> want to have 100% coverage.  Currently, we're at about 60% or so.

        Doesn't that seem like duplicated effort with our own Selenium tests? I
mean, we even had Selenium tests for 2.22, if you wanted to use those.

        -Max
--
http://www.everythingsolved.com/
Competent, Friendly Bugzilla and Perl Services. Everything Else, too.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Max Kanat-Alexander
In reply to this post by ahdevans
On 02/12/2010 02:05 PM, Aaron Evans wrote:
> The biggest problem I've found trying to go back and fill in unit tests
> is you can either cover the obvious things that won't break unless there
> are API changes, but the type of stuff that more often breaks can't
> often be deduced from the code (not to mention you can't find existing
> bugs when the implementation is your spec.

        Yeah, true, but I think that using a coverage tool helps a lot with
that, because then at least you can be sure that all code paths are
covered. Perl's testing tools are actually really top-of-the-line stuff,
for the most part, and Devel::Cover is pretty good.

        -Max
--
http://www.everythingsolved.com/
Competent, Friendly Bugzilla and Perl Services. Everything Else, too.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Gabor Szabo
Personally I don't think adding strictly speaking unit tests to already
existing code makes a lot of sense. At least not in the case of a full
blown application.

I'd rather have more Selenium tests or tests that execute the individual
.cgi scripts on the command line.
I never tried to measure coverage of Selenium tests but maybe that
can be done as well.

Then those tests can be included in the t/ directory as well and executed
on condition.

For new code I'd still write the unit tests.

Gabor
ps. In companies I would be happy with 30-40% coverage, it is usually
way lower but on CPAN modules I regularly see 70-80% see a few of
them here: http://szabgab.com/coverage/
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Max Kanat-Alexander
On 02/13/2010 11:14 AM, Gabor Szabo wrote:
> Personally I don't think adding strictly speaking unit tests to already
> existing code makes a lot of sense. At least not in the case of a full
> blown application.
>
> I'd rather have more Selenium tests or tests that execute the individual
> .cgi scripts on the command line.
> I never tried to measure coverage of Selenium tests but maybe that
> can be done as well.

        Yeah, I think it might make more sense to focus on the Selenium tests
for now, because we can guarantee that they can actually test the full
application.

        I do actually want unit tests, though. I think it's the lack of
existing unit tests that causes us to not write new ones for stuff that
we check in.

> Then those tests can be included in the t/ directory as well and executed
> on condition.

        Yeah, I was thinking that as well, although at the moment they require
basically an empty database in order to run.

> ps. In companies I would be happy with 30-40% coverage, it is usually
> way lower but on CPAN modules I regularly see 70-80% see a few of
> them here: http://szabgab.com/coverage/

        Yeah, I still like to try for 100% coverage, though. I think that my
current CPAN modules come pretty close, actually, although I just
checked one of them after this discussion and I noticed there are a few
extra code paths I need to test. :-)

        -Max
--
http://www.everythingsolved.com/
Competent, Friendly Bugzilla and Perl Services. Everything Else, too.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Gabor Szabo
In reply to this post by David Marshall-5
On Sat, Feb 13, 2010 at 12:04 AM, David Marshall <[hidden email]> wrote:

>
>
>
> On 2/12/10 1:54 PM, "Max Kanat-Alexander" <[hidden email]> wrote:
>
>> On 02/11/2010 10:13 PM, Gabor Szabo wrote:
>>> and the total test coverage of Bugzilla is 18.4%
>>
>> Yeah, just to say again what others have said, this is a problem, for sure.
>>
>
> I don't think it's a problem at all, because the t/* tests were never, as
> far as I am aware, intended to be testing for coverage.  I'm rather
> surprised that the number is as high as 18.4%!
>
> At Yahoo!, where we've been putting a lot of effort into continuous
> integration, I use t/* as a "commit test" - committing to the repository
> triggers an automatic run of t/*.  The idea is to do a quick check that a
> commit hasn't horked anything - the Perl compiles, etc.
>
> We also have a home-grown test suite that we use as a "smoke test" - a
> nightly test that interacts with Bugzilla through apache and a working
> database.  It takes about twenty minutes to run at present.  That's what I
> want to have 100% coverage.  Currently, we're at about 60% or so.
>
> We have our own Selenium tests written as well, but I haven't included them
> in our continuous integration implementation.
>

Do I understand correctly that you have your own test suite in Yahoo!
testing Bugzilla?
If so why is it not part of the standard set of test of Bugzilla?


Gabor
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

David Marshall-5



On 2/14/10 1:57 AM, "Gabor Szabo" <[hidden email]> wrote:


>
> Do I understand correctly that you have your own test suite in Yahoo!
> testing Bugzilla?
> If so why is it not part of the standard set of test of Bugzilla?
>

There are three reasons:

1) the non-Selenium tests use an in-house testing and reporting system
without which they're not very useful.

2) the Selenium tests are written against what is a very heavily customized
UI.

3) I don't have permission to send any changes upstream.  That's unlikely to
change anytime soon.

-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Gabor Szabo
On Sun, Feb 14, 2010 at 7:12 PM, David Marshall <[hidden email]> wrote:

> There are three reasons:

thanks for the clarification.

Gabor
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Gervase Markham
In reply to this post by Gabor Szabo
On 12/02/10 21:54, Max Kanat-Alexander wrote:
> Having unit tests for every single method and function in Bugzilla
> would be great, though. The fact is, the lack of TDD in our development
> process is one of the things that makes the review process so difficult.
> But the number of tests that we'd have to write to truly have full
> coverage is also something that's nearly overwhelming for us, because we
> just don't have the current manpower to do it.

Right. Mozilla has exactly the same problem. The way you deal with it is
to make sure the test frameworks are in place, and there are good docs
on writing and running the tests, and then require tests to accompany
all new patches. Tests accidentally/implicitly test more than they are
aimed at anyway, and this way the coverage percentage will rise
organically. When it gets nearer to 90% than 20%, we can look again at
what's necessary to get 100% coverage, if we still think it's worth it.

Gerv
_______________________________________________
dev-apps-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-bugzilla
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

Joel Peshkin

The 18.4% number seems to refer to the t/ tests.   I think, if we are
serious about this, we need to start by creating a mechanism in the t/
tests to have it use a test database.  So, you do the following...

perl Makefile.PL --db_server=localhost --db=bztest --db_user=foo
--db_passwd=bar
make test

and the tests create their own databases, preloading them with data, and
run.

Naturally, this also means that we need to figure out how to do the
equivalent of the selenium tests in a way that won't break all of the
tests every time a trivial change is made in a template.

-Joel

-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Test coverage is 18.4%

David Marshall-5



On 2/15/10 7:19 AM, "Joel Peshkin" <[hidden email]> wrote:

>
> The 18.4% number seems to refer to the t/ tests.   I think, if we are
> serious about this, we need to start by creating a mechanism in the t/
> tests to have it use a test database.  So, you do the following...

I don't agree that a low coverage number from what has always been static
tests means that interest in testing is not serious.
>
> perl Makefile.PL --db_server=localhost --db=bztest --db_user=foo
> --db_passwd=bar
> make test
>
> and the tests create their own databases, preloading them with data, and
> run.
>

I'd personally rather such tests existed in some separate directories from
the static tests.  But I agree that it would be useful to have dynamic
testing against a database.

I've wrestled with this at Yahoo!.  We do a lot of testing with a
LWP::UserAgent and a test database.  It wouldn't really be convenient to
start each test with the same database starting position.  I think sometimes
about creating a minimal database that I can initialize to a know state, run
the tests, and then look at how it has changed, but that may be overkill.
As it is now, for something like creating a new bug, we fill in the form,
submit it to get the new bug number, then go look around the database to
check that the attributes of the new bug are correct.


> Naturally, this also means that we need to figure out how to do the
> equivalent of the selenium tests in a way that won't break all of the
> tests every time a trivial change is made in a template.
>

If the a change to a template is trivial, shouldn't the corresponding change
to the Selenium tests be equally non-trivial?  If modifying the tests to
reflect the change is non-trivial, how can the change itself be trivial?

-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>