Branch release problems

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Branch release problems

Dave Miller
Tonight we released a 2.18.2 release with a non-functioning query page.

I had to post an announcement telling people not to download it and wait
for an update.  That's not a fun thing to do.  This has also happened
before (branch releases going out with major regressions in them) with
versions 2.16.9 and 2.14.5.  This is really unacceptable and it messes
badly with the public perception of our credibility.

We really need some kind of better QA process on our branch releases.  I
know it's hard because all of the developers run the cvs tip or the
developer releases for their own installations, so nobody's familiar
with the branches, but that's what we have landfill for, and we need to
make sure things get tested before they go out.

At a minimum we need to put together some kind of smoke test document
with lists of features to test for each branch, then we need to make
sure that testing *everything* on that list is made part of the release
process, before a tarball is rolled, no matter how few patches actually
got checked in, or what they touched.

I'd welcome any other ideas anyone has, too.

We've also got almost zero actual QA since MattyT got busy with real
life.  It'd probably be a cool thing to get some people to actually do
QA type things for us again.  Bugzilla is growing by leaps and bounds,
and we definitely have a big enough product to badly need a whole QA
team, let alone the one-man-show that MattyT used to run.

--
Dave Miller                                   http://www.justdave.net/
System Administrator, Mozilla Foundation       http://www.mozilla.org/
Project Leader, Bugzilla Bug Tracking System  http://www.bugzilla.org/
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Branch release problems

Frédéric Buclin
> versions 2.16.9 and 2.14.5.  This is really unacceptable and it messes
> badly with the public perception of our credibility.

> I'd welcome any other ideas anyone has, too.

Create a (small) QA team, if possible including some reviewers (who
should be the most familiar with bz code), and ask them to create this
smoke test document you are talking about. At least 24 hours before any
release, no more checkin is allowed. And security patches have to be
applied manually to avoid making them public (no CVS upload). During
this "really frozen" period, the QA team checks *all* points of the
smoke test document, such as:

- Queries, including query.cgi, buglist.cgi and Search.pm
- Creating/changing bugs and attachments, including flags
- UI
- Charts/reports
- E-Mails, including BugMail.pm and *whine*.*
- Admin pages (product, component, ... creation and deletion)
- Account creation/ prefs settings
- Some "security" tests
- Fresh installation/upgrade from older ones
- Improve and check sanitycheck.cgi (or create your own releasecheck.cgi)

I am a reviewer, I have access to all security bugs related to Webtools
and I already do some "QA testing" (kind of). So I agree to belong to
this QA team.

But doing checkins 15 minutes before releasing any new version is far
too short for me, especially when I'm away at this precise moment. :)

Frederic "LpSolit" Buclin
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Branch release problems

Ed-10
I'd suggest to use Test Runner to document smoke test and keep a track
record of their outcome.

Kind regards,

    Ed

On 7/8/05, Frédéric Buclin <[hidden email]> wrote:

> > versions 2.16.9 and 2.14.5.  This is really unacceptable and it messes
> > badly with the public perception of our credibility.
>
> > I'd welcome any other ideas anyone has, too.
>
> Create a (small) QA team, if possible including some reviewers (who
> should be the most familiar with bz code), and ask them to create this
> smoke test document you are talking about. At least 24 hours before any
> release, no more checkin is allowed. And security patches have to be
> applied manually to avoid making them public (no CVS upload). During
> this "really frozen" period, the QA team checks *all* points of the
> smoke test document, such as:
>
> - Queries, including query.cgi, buglist.cgi and Search.pm
> - Creating/changing bugs and attachments, including flags
> - UI
> - Charts/reports
> - E-Mails, including BugMail.pm and *whine*.*
> - Admin pages (product, component, ... creation and deletion)
> - Account creation/ prefs settings
> - Some "security" tests
> - Fresh installation/upgrade from older ones
> - Improve and check sanitycheck.cgi (or create your own releasecheck.cgi)
>
> I am a reviewer, I have access to all security bugs related to Webtools
> and I already do some "QA testing" (kind of). So I agree to belong to
> this QA team.
>
> But doing checkins 15 minutes before releasing any new version is far
> too short for me, especially when I'm away at this precise moment. :)
>
> Frederic "LpSolit" Buclin
> -
> To view or change your list settings, click here:
> <http://bugzilla.org/cgi-bin/mj_wwwusr?user=ed.fuentetaja@...>
>

-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Branch release problems

Jason Remillard
In reply to this post by Dave Miller
Hi,

David Miller wrote:

> Tonight we released a 2.18.2 release with a non-functioning query page.
>
> I had to post an announcement telling people not to download it and wait
> for an update.  That's not a fun thing to do.  This has also happened
> before (branch releases going out with major regressions in them) with
> versions 2.16.9 and 2.14.5.  This is really unacceptable and it messes
> badly with the public perception of our credibility.
>
> We really need some kind of better QA process on our branch releases.  I
> know it's hard because all of the developers run the cvs tip or the
> developer releases for their own installations, so nobody's familiar
> with the branches, but that's what we have landfill for, and we need to
> make sure things get tested before they go out.
>
> At a minimum we need to put together some kind of smoke test document
> with lists of features to test for each branch, then we need to make
> sure that testing *everything* on that list is made part of the release
> process, before a tarball is rolled, no matter how few patches actually
> got checked in, or what they touched.
>
> I'd welcome any other ideas anyone has, too.
>

I wrote a set or perl tests scripts using the LWP modules, HTML::Form,
HTML::LinkExtractor, HTML::Lint that are used to test codestriker out.
It is pretty easy to test a web base application with perl. You write a
proxy .pm for each page in bugzilla. The proxy uses LWP, and friends to
manipulate the actual service. Your .t files then use the proxies to do
the tests. We used to have a smoke test document. The smoke test is all
automated now + a bunch of other tests that we were not doing before.
Compared to the complexity of bugzilla, these tests would really not be
that big of a deal to write. It is really easy to at least make sure all
of the links are good, and the generated html is well formed.

The code is here.

http://cvs.sourceforge.net/viewcvs.py/codestriker/codestriker/test/

Once the infrastructure is in place, then you can insist that every new
features get checked in with test cases. I know that there are plans to
test the pm files inside of bugzilla directly, but you really can't
escape the fact that it needs to work property at the
   http/html level. I don't think doing both is unreasonable.

Not everything will be easy to automate, so a test document is also a
good idea.

Thanks
Jason.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Branch release problems

Benton, Kevin
In reply to this post by Dave Miller
> Tonight we released a 2.18.2 release with a non-functioning query
page.
>
> I had to post an announcement telling people not to download it and
wait
> for an update.  That's not a fun thing to do.  This has also happened
> before (branch releases going out with major regressions in them) with
> versions 2.16.9 and 2.14.5.  This is really unacceptable and it messes
> badly with the public perception of our credibility.
>
> We really need some kind of better QA process on our branch releases.
I
> know it's hard because all of the developers run the cvs tip or the
> developer releases for their own installations, so nobody's familiar
> with the branches, but that's what we have landfill for, and we need
to
> make sure things get tested before they go out.
>
> At a minimum we need to put together some kind of smoke test document
> with lists of features to test for each branch, then we need to make
> sure that testing *everything* on that list is made part of the
release
> process, before a tarball is rolled, no matter how few patches
actually
> got checked in, or what they touched.
>
> I'd welcome any other ideas anyone has, too.
>
> We've also got almost zero actual QA since MattyT got busy with real
> life.  It'd probably be a cool thing to get some people to actually do
> QA type things for us again.  Bugzilla is growing by leaps and bounds,
> and we definitely have a big enough product to badly need a whole QA
> team, let alone the one-man-show that MattyT used to run.

Dave - that's why I'm actively working on a pre-release testing guide
for Bugzilla use here.  Part of what I'm doing, however, is making it so
I can produce the necessary documentation for general Bugzilla testing
as well.  The process of creating this guide is so involved, however,
that I don't see it being done for at least a few months.

---
Kevin Benton
Perl/Bugzilla Developer
Advanced Micro Devices
 
The opinions stated in this communication do not necessarily reflect the
view of Advanced Micro Devices and have not been reviewed by management.
This communication may contain sensitive and/or confidential and/or
proprietary information.  Distribution of such information is strictly
prohibited without prior consent of Advanced Micro Devices.  This
communication is for the intended recipient(s) only.  If you have
received this communication in error, please notify the sender, then
destroy any remaining copies of this communication.


-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>