Firefox 3 PRD should include Quality requirements

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Firefox 3 PRD should include Quality requirements

Justin Dolske-2
I was looking at the PRD and meeting notes, and the thought struck me
that *quality* isn't really mentioned in the PRD. There's a couple of
vague references to it in the "Crosscutting Concerns" section, but I
feel it should really be called out as a peer to other major requirement
sections.

Some of the most common criticisms of FF2 are related to high memory
usage/leakage, and crashiness. That these two things are mentioned so
often ought to be enough to merit specific attention in FF3. We're
obviously aware of the issues, but these should be formally tracked as
FF3 requirements.

How should we track and these things? We could be lazy and simply add a
couple of NFRs to "be stable" and "use less memory". But I think we
should try to nail them down as FRs in a quantitative way.

As a brainstorming example:
- FR: reduce browser MTBF by X%, as compared to Firefox 2.0.0.x
- FR: MTBF > X hours for some particular use case or test.
- FR: total FF memory usage < X MB after completing some test
- FR: reduce total FF memory usage by X%, compared to FF2.

We would almost certainly need new (or modified) tests to generate
whatever measurements we decide are needed. Crash-related data, in
particular, unobtainable via Talkback might be obtainable with Airbag
(which has implications on the schedule for Airbag). And while getting
FF3 figures in absolute form would be good, also having figures for FF2
to enable relative comparisons would be great. [And as a blue-sky
thought, imagine being able to compare metrics against other browsers.]


In addition to having Quality requirements for plain-vanilla Firefox, I
think we should also have similar requirements Firefox with some set of
add-ons installed (extensions, themes, plugins). Achieving these
requirements would mean more than just fixing browser bugs -- we would
also need to make an active effort to engage the add-on community...
This might mean providing better documentation and tools for addressing
the problem, and even having Mozilla contribute or suggest code changes
to some add-ons wouldn't be entirely out of the question, IMO.

Justin
"Now wiht 47% less typos!"
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Gervase Markham
Justin Dolske wrote:
> I was looking at the PRD and meeting notes, and the thought struck me
> that *quality* isn't really mentioned in the PRD. There's a couple of
> vague references to it in the "Crosscutting Concerns" section, but I
> feel it should really be called out as a peer to other major requirement
> sections.

A fair point IMO.

> How should we track and these things? We could be lazy and simply add a
> couple of NFRs to "be stable" and "use less memory". But I think we
> should try to nail them down as FRs in a quantitative way.
>
> As a brainstorming example:
> - FR: reduce browser MTBF by X%, as compared to Firefox 2.0.0.x

Do we have MTBF numbers, from Talkback or elsewhere?

> - FR: total FF memory usage < X MB after completing some test
> - FR: reduce total FF memory usage by X%, compared to FF2.

That would require a usage scenario, of course.

> In addition to having Quality requirements for plain-vanilla Firefox, I
> think we should also have similar requirements Firefox with some set of
> add-ons installed (extensions, themes, plugins).

But which set? Pulling together a set of "common" addons is a difficult
thing, because in the grand scheme of things, people's addon preferences
are so varied that no addon is all that common.

I'd be concerned that this would be an enormous can of worms.

Gerv
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Chris Hofmann


Gervase Markham wrote:
> Justin Dolske wrote:
>> I
>>
>> As a brainstorming example:
>> - FR: reduce browser MTBF by X%, as compared to Firefox 2.0.0.x
>
> Do we have MTBF numbers, from Talkback or elsewhere?
>
http://talkback-public.mozilla.org/reports/firefox/FF2001/smart-analysis.all

The problem with trying set this as a criteria for release is that its a
circular dependency.

You need to have population that has similar usage patterns and amounts
of usage to compare 2.0.0.x to 3.0 so that means you have to have
shipped 3.0 to collect meaningful MTBF data, and it has to have shipped
for awhile to allow the MTBF number to "stabilize".     With talkback we
only get reports from users that have crashed so the MTBF number not
perfect, but shows some interesting release to release patters that are
helpful in understanding if we are getting better or worse.  The number
starts low on the first day, then grows as the possible number of uptime
hours grows.  After about 4-6 weeks the MTBF number usually stabilizes.  
We used to plot those graphs but I don't think that is turned on any
more.  There will also be some problems in trying to compare MTBF
numbers between 2.x and 3.0 if we switch over to airbag and a new
mechanism/quirks for computing MTBF.

The best kind of goal for stability and other quality criteria is to
specify that beta's and release candidates are shipped to a significant
number of users.  We should set goals of 2-3% market share ( from many
accounts IE 7 had about 2.3% market share when it shipped, and Firefox
1.0 had about 3%).

If you can get that many users happily using a pre-release it helps to
get the right amount of volume for all kinds of feedback and it
significantly reduces risk.

For more specific stability goals/actions you can specify that the top
crash lists for betas and release candidates show no significant
regressions from previous releases.

chris h.
>> -

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Mike Beltzner
In reply to this post by Justin Dolske-2
On 14-Feb-07, at 12:32 AM, Justin Dolske wrote:

> I was looking at the PRD and meeting notes, and the thought struck  
> me that *quality* isn't really mentioned in the PRD. There's a  
> couple of vague references to it in the "Crosscutting Concerns"  
> section, but I feel it should really be called out as a peer to  
> other major requirement sections.

First, the cross-cutting concerns are of equal or indeed higher  
importance than the FRs and NFRs, as I see things. They reflect  
concerns and requirements which *must* be taken into account across  
all work done on the product. They're extracted like that so that  
every "feature" doesn't have an additional P1 for things like "code  
is secure" or "feature doesn't crash." They represent the non-
negotiable basic set of expectations for any changes to the product.  
Perhaps that needs to be made clearer in the structure of the  
document or in the way they are introduced, or perhaps every  
functional area needs to have a P1 FR of "satisfied all cross-cutting  
concerns"?

Second, how does one measure "quality"? The issues you raise below  
imply "performance" (as measured by speed & memory usage) and  
"stability" (as measured by crashiness) which are captured in the  
cross-cutting concerns directly as "Performance" and "Reliability".

I do think, however, that there should be a separate part of the PRD  
devoted to scoping and assigning resource to the task of eliminating  
cruft within our codebase. Someone might Joel on Software me here,  
but I think it might be worth our time to try and figure out if  
there's pieces of code that can be refactored or made more efficient,  
or outright removed in the service of performance and download size.

> Some of the most common criticisms of FF2 are related to high  
> memory usage/leakage, and crashiness. That these two things are  
> mentioned so often ought to be enough to merit specific attention  
> in FF3. We're obviously aware of the issues, but these should be  
> formally tracked as FF3 requirements.
>
> How should we track and these things? We could be lazy and simply  
> add a couple of NFRs to "be stable" and "use less memory". But I  
> think we should try to nail them down as FRs in a quantitative way.

I agree entirely. We tried to get this nailed down for Fx2, but kept  
pushing off meetings scheduled for "figuring out what metrics to use  
and what reasonable targets are" until it became too late. We  
shouldn't repeat that mistake.

You'll note that the cross-cutting concern for performance states:

  * Performance metrics have been discussed, agreed upon, and recorded.
  * Does not have a negative impact on overall Firefox performance.
  * If performance impact is unavoidable, that impact is within a set  
and agreed upon range.
  * Optimized as much as possible to improve overall performance.
  * Performance tests have been run and verified by QA.

This applies to each feature area in the PRD, meaning that we'll have  
to come up with metrics for each. This should be done in conjunction  
with QA to ensure that the required test suites can be run frequently  
to checkpoint against those metrics, really.

> As a brainstorming example:
> - FR: reduce browser MTBF by X%, as compared to Firefox 2.0.0.x
> - FR: MTBF > X hours for some particular use case or test.
> - FR: total FF memory usage < X MB after completing some test
> - FR: reduce total FF memory usage by X%, compared to FF2.
>
> We would almost certainly need new (or modified) tests to generate  
> whatever measurements we decide are needed. Crash-related data, in  
> particular, unobtainable via Talkback might be obtainable with  
> Airbag (which has implications on the schedule for Airbag). And  
> while getting FF3 figures in absolute form would be good, also  
> having figures for FF2 to enable relative comparisons would be  
> great. [And as a blue-sky thought, imagine being able to compare  
> metrics against other browsers.]

Agreed; also, I know that QA is looking to figure out what sort of  
comparisons would be useful. A rough metric of crashiness could be  
assembled by taking the number of crash reports and dividing it by  
the number of AUS pings.

What other tests would be good for measuring performance? I know that  
Alice is working on some of these already, too ...

> In addition to having Quality requirements for plain-vanilla  
> Firefox, I think we should also have similar requirements Firefox  
> with some set of add-ons installed (extensions, themes, plugins).  
> Achieving these requirements would mean more than just fixing  
> browser bugs -- we would also need to make an active effort to  
> engage the add-on community... This might mean providing better  
> documentation and tools for addressing the problem, and even having  
> Mozilla contribute or suggest code changes to some add-ons wouldn't  
> be entirely out of the question, IMO.

The AMO team already has some Great Plans Afoot here, as I understand  
things, but definitely the idea of running our performance tests  
against a set of common plugins (ABP, FlashBlock, Forecast Fox, the  
ones that get mentioned all the time in articles, etc) seems like a  
great idea.

cheers,
mike

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Justin Dolske-2
In reply to this post by Gervase Markham
Gervase Markham wrote:

>> As a brainstorming example:
>> - FR: reduce browser MTBF by X%, as compared to Firefox 2.0.0.x
>
> Do we have MTBF numbers, from Talkback or elsewhere?

My understanding is that it's hard to get a sound number from Talkback,
mainly because (IIRC) a report says when the last crash was, but not how
much usage happened since then.

But I think the main point here is that a discussion about what metric
to use here will need to balance what we *should* measure with what we
*can* measure.

>> In addition to having Quality requirements for plain-vanilla Firefox,
>> I think we should also have similar requirements Firefox with some set
>> of add-ons installed (extensions, themes, plugins).
>
> But which set? Pulling together a set of "common" addons is a difficult
> thing, because in the grand scheme of things, people's addon preferences
> are so varied that no addon is all that common.

Good question. We obviously can't test all extensions, and even
determining a set of typical configurations might be impractical. But we
do have data from AMO to identify the most popular add-ons, and so a
simple starting point might be to test with the top-10 addons installed.

Another possibility might be to make test suites accessible to extension
developers, and allow them to do their own testing.

> I'd be concerned that this would be an enormous can of worms.

It could be, we just don't want to open the can too much. :-)

Justin
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Rafael Ebron-2
In reply to this post by Justin Dolske-2
Well, there was a lot of work done around quality in the past and there
were fairly clear metrics associated with them.  I hate to think that
we're starting from scratch here with definition and data/benchmarks.  
chofmann, dbaron, jay should know more.  Most of this info should be
somewhere in that black hole called mozilla.org.

MTBF, back and forward button performance, open new window (cold and
warm start), page load tests with the top x number of sites.  Should be
testing all these against IE 7 on Windows Vista, Firefox 2.0, Safari.

There's new ones to add for sure, e.g. Zimbra seems to have performance
metrics around their apps and I would bet Google does too.  Music and
video sites should be included in some of these measures too, watching a
video on YouTube, listening to music on someone's profile on MySpace.

So there's the set of objective quality measures around stability and
performance.  The more subjective ones around visual and UE experience
e.g. pixels are off or UI in preferences on Mac OS X shouldn't be
choppy, have to be defined separately.

But yes, quality (performance, stability, user experience) should be
defined in the prd, not in full detail but enough detail.  Performance
and stability should have it's own section in the PRD and each feature
won't necessarily have performance or stability requirements attached to
them, that's just overkill.  

Also, where it would be good to see more testing is with Flash, WMP,
QuickTime, some of the more original and more widely used "add-ons".

-Rafael



Justin Dolske wrote:

> I was looking at the PRD and meeting notes, and the thought struck me
> that *quality* isn't really mentioned in the PRD. There's a couple of
> vague references to it in the "Crosscutting Concerns" section, but I
> feel it should really be called out as a peer to other major
> requirement sections.
>
> Some of the most common criticisms of FF2 are related to high memory
> usage/leakage, and crashiness. That these two things are mentioned so
> often ought to be enough to merit specific attention in FF3. We're
> obviously aware of the issues, but these should be formally tracked as
> FF3 requirements.
>
> How should we track and these things? We could be lazy and simply add
> a couple of NFRs to "be stable" and "use less memory". But I think we
> should try to nail them down as FRs in a quantitative way.
>
> As a brainstorming example:
> - FR: reduce browser MTBF by X%, as compared to Firefox 2.0.0.x
> - FR: MTBF > X hours for some particular use case or test.
> - FR: total FF memory usage < X MB after completing some test
> - FR: reduce total FF memory usage by X%, compared to FF2.
>
> We would almost certainly need new (or modified) tests to generate
> whatever measurements we decide are needed. Crash-related data, in
> particular, unobtainable via Talkback might be obtainable with Airbag
> (which has implications on the schedule for Airbag). And while getting
> FF3 figures in absolute form would be good, also having figures for
> FF2 to enable relative comparisons would be great. [And as a blue-sky
> thought, imagine being able to compare metrics against other browsers.]
>
>
> In addition to having Quality requirements for plain-vanilla Firefox,
> I think we should also have similar requirements Firefox with some set
> of add-ons installed (extensions, themes, plugins). Achieving these
> requirements would mean more than just fixing browser bugs -- we would
> also need to make an active effort to engage the add-on community...
> This might mean providing better documentation and tools for
> addressing the problem, and even having Mozilla contribute or suggest
> code changes to some add-ons wouldn't be entirely out of the question,
> IMO.
>
> Justin
> "Now wiht 47% less typos!"
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

rkiddy@mozilla.com
In reply to this post by Justin Dolske-2
Rafael Ebron wrote:
> Well, there was a lot of work done around quality in the past and there
> were fairly clear metrics associated with them.  I hate to think that
> we're starting from scratch here with definition and data/benchmarks.  
> chofmann, dbaron, jay should know more.  Most of this info should be
> somewhere in that black hole called mozilla.org.

As someone who has recently been given the task of spelunking into that
black hole to find tests and test harnesses, I can tell you that those
people and many more have a lot of information about what has been done
and what is available. But a lot of knowledge at Mozilla is in the form
of "tribal knowledge" or in bits and pieces around the distributed
information space that is "Mozilla" and I think that most people do not
assume their view is comprehensive.

> MTBF, back and forward button performance, open new window (cold and
> warm start), page load tests with the top x number of sites.  Should be
> testing all these against IE 7 on Windows Vista, Firefox 2.0, Safari.
> There's new ones to add for sure, e.g. Zimbra seems to have performance
> metrics around their apps and I would bet Google does too.  Music and
> video sites should be included in some of these measures too, watching a
> video on YouTube, listening to music on someone's profile on MySpace.

People are testing many of these things and we are recording the results
of this testing. When some of the automation testing is launched, and it
is fairly near to being launched, the testing information will be much
more current and visible.

The expectations you are expressing are exactly what I have been hearing
from everybody at Mozilla that I have talked to. So do not let yourself
feel that you are alone. People know that all this needs to get done.

> But yes, quality (performance, stability, user experience) should be
> defined in the prd, not in full detail but enough detail.  Performance
> and stability should have it's own section in the PRD and each feature
> won't necessarily have performance or stability requirements attached to
> them, that's just overkill.

I have a feeling this kind of this will be happening. People know there
is work to be done for things like performance and stability. There was
work done that tightened up memory leakage, but I check for leaks and I
know there is still work to do.

http://xoatlicue.blogspot.com/2007/02/quick-and-easy-leaks-detection-on-mac.html

> Also, where it would be good to see more testing is with Flash, WMP,
> QuickTime, some of the more original and more widely used "add-ons".
> -Rafael

I think the inventory of testing that I have done brings me to say you
are completely right. Frankly, I want to try to find a way to
characterize the behavior of some of the add-ons, but I want to be sure
of what I am doing first. I am new to the Mozilla community and I do not
want to cause a back-and-forth of blame for quality issues by looking at
these issues in a simplistic way.

If you have suggestions for testing add-ons, please let us know what
they are.

I am often on the IRC channels (nick: "ray", lurker in: #qa, #devmo,
#litmus).

thanx - ray


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Gervase Markham
In reply to this post by Justin Dolske-2
Justin Dolske wrote:
> My understanding is that it's hard to get a sound number from Talkback,
> mainly because (IIRC) a report says when the last crash was, but not how
> much usage happened since then.

Yes; and I'm not sure we can ever get that figure without being too
intrusive. :-|

> Good question. We obviously can't test all extensions, and even
> determining a set of typical configurations might be impractical. But we
> do have data from AMO to identify the most popular add-ons, and so a
> simple starting point might be to test with the top-10 addons installed.

But if we isolated a problematic add-on, would we end up de facto
joining its development team? Surely part of the point of addons is that
they are addons, maintained and worried about by someone else? :-)

Gerv
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Gijs Kruitbosch ("Hannibal")
Gervase Markham wrote:

>> Good question. We obviously can't test all extensions, and even
>> determining a set of typical configurations might be impractical. But
>> we do have data from AMO to identify the most popular add-ons, and so
>> a simple starting point might be to test with the top-10 addons
>> installed.
>
> But if we isolated a problematic add-on, would we end up de facto
> joining its development team? Surely part of the point of addons is that
> they are addons, maintained and worried about by someone else? :-)
>
> Gerv

Yes and no, respectively, I believe.

Yes in that you're right as is: if we fix add-on problems ourselves then
we join their development team. Whether that's a really bad thing is
questionable, but I don't think Justin said we would have to develop the
fix themselves. Surely you wouldn't consider everyone who has ever filed
a bug in bugzilla "part of the Firefox development team" ?

More importantly, I think the answer to your second question is "no" in
the sense that (for instance) the Windows development team contacted
Mozilla some time before releasing Vista and offered them help in
adapting Firefox and their other software so it would run well on it. In
the same way, I could imagine the Firefox development team would want to
make sure that at least some of the well-known add-ons or applications
that use Firefox / the "Mozilla Platform" continue running well in new
versions. This is wise both from a developer relations point of view
(try not to have the extension developers find out about problems
themselves at the last minute) and a user experience point of view -
users will want their add-ons to continue to work, and if they don't
then they will first go to the add-on authors, who will point the finger
back at Mozilla if it's their fault.

~ Gijs
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Gervase Markham
Gijs Kruitbosch wrote:
> More importantly, I think the answer to your second question is "no" in
> the sense that (for instance) the Windows development team contacted
> Mozilla some time before releasing Vista and offered them help in
> adapting Firefox and their other software so it would run well on it.

True.

> In
> the same way, I could imagine the Firefox development team would want to
> make sure that at least some of the well-known add-ons or applications
> that use Firefox / the "Mozilla Platform" continue running well in new
> versions. This is wise both from a developer relations point of view
> (try not to have the extension developers find out about problems
> themselves at the last minute) and a user experience point of view -
> users will want their add-ons to continue to work, and if they don't
> then they will first go to the add-on authors, who will point the finger
> back at Mozilla if it's their fault.

Well, I would rather expect extension developers to be testing and
giving feedback throughout the development cycle anyway. The reason
Microsoft needs a Vista lab is because you can't download nightly builds
of Vista during the development and test against it, and you can't
propose patches to fix things you need fixed.

But maybe you are right. We should make some effort to ensure the X most
popular extensions work. However, what is the value of X?

It would be great if someone could take amo download data and produce a
graph of number of extensions vs. percentage of downloads, so we could
see if the top 50 made up 10% of downloads, or 90%. In other words, how
long is the long tail?

Gerv
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

James Ross
Gervase Markham wrote:
> It would be great if someone could take amo download data and produce a
> graph of number of extensions vs. percentage of downloads, so we could
> see if the top 50 made up 10% of downloads, or 90%. In other words, how
> long is the long tail?

I've got graphs of the downloads/week for the top 30 extensions (and
some others) at http://twpol.dyndns.org/mozilla/extensions/stats/, and
just for kicks I worked out the sum of downloads/week for them [1].

So, the total downloads/week for the current top N AMO extensions is:

N   D/week
5   630,000
10  980,000
20  1,425,000
30  1,675,000

Anyone know what the total downloads for all extensions is like?

--
James Ross <[hidden email]>
ChatZilla Developer

[1] I could have done this an easier way, actually, but doing graphs is
way more fun. See:
http://twpol.dyndns.org/mozilla/extensions/stats/?group=1;m=dlsum and
http://twpol.dyndns.org/mozilla/extensions/stats/?group=2;m=dlsum for
top 5/10 respectively (they take a while to load the image, need to fix
that).
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

rkiddy@mozilla.com
In reply to this post by Gervase Markham
<snip>
Gervase Markham wrote:

>> In the same way, I could imagine the Firefox development team would
>> want to make sure that at least some of the well-known add-ons or
>> applications that use Firefox / the "Mozilla Platform" continue
>> running well in new versions. This is wise both from a developer
>> relations point of view (try not to have the extension developers find
>> out about problems themselves at the last minute) and a user
>> experience point of view - users will want their add-ons to continue
>> to work, and if they don't then they will first go to the add-on
>> authors, who will point the finger back at Mozilla if it's their fault.
>
> Well, I would rather expect extension developers to be testing and
> giving feedback throughout the development cycle anyway. The reason
> Microsoft needs a Vista lab is because you can't download nightly builds
> of Vista during the development and test against it, and you can't
> propose patches to fix things you need fixed.
>

FYI, I proposed that, in addition to doing the nightly builds that the
build team does now, they also do builds that have tests enabled.

https://bugzilla.mozilla.org/show_bug.cgi?id=369809

This should make it easier for extension developers, and other members
of the community, to run the tests that can be made available.

<snip>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Justin Dolske-2
In reply to this post by Gervase Markham
Gervase Markham wrote:

> Well, I would rather expect extension developers to be testing and
> giving feedback throughout the development cycle anyway.

True. But that's placing the bar fairly high, as I don't think you can
necessarily rely on extension developers to test things beyond "does my
extension work" and "did it break anything obvious". I'd be surprised if
more than a handful of developers had any kind of test suite, or checks
for things like memory leaks and page load performance.

Justin
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Justin Dolske-2
Justin Dolske wrote:

> True. But that's placing the bar fairly high

Err, I mean "low". For some reason I had limbo stuck in my head. :-)

Justin
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Gervase Markham
In reply to this post by James Ross
James Ross wrote:
> Anyone know what the total downloads for all extensions is like?

Yes - we need to know this to know how long the tail is.

Gerv
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

Mike Shaver
On 2/20/07, Gervase Markham <[hidden email]> wrote:
> James Ross wrote:
> > Anyone know what the total downloads for all extensions is like?
>
> Yes - we need to know this to know how long the tail is.

Now is a _particularly_ bad time for me to ask people to dive into
mining that data, but -- to the extent that it's reliable -- I'll try
to extract that this week.

Update-pings would be a more interesting metric, perhaps, since they
reflect continuing use and not just new installs, but they have their
own problems!  (For one thing, the sheer size of the dataset makes
some of our tools sad-faces.)

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Firefox 3 PRD should include Quality requirements

alice nodelman
In reply to this post by Justin Dolske-2

> Agreed; also, I know that QA is looking to figure out what sort of
> comparisons would be useful. A rough metric of crashiness could be
> assembled by taking the number of crash reports and dividing it by the
> number of AUS pings.
>
> What other tests would be good for measuring performance? I know that
> Alice is working on some of these already, too ...

The discussion that QA recently had was concerning the ability for QA to
apply some sort of quality number to a given release.  The idea being
that we do a lot of testing on the front end, but don't really have a
measure to say if a release could be considered good or bad - other than
if we had to push out a rapid fix in short order.

We threw around a bunch of numbers (top crashes, bug filings, bug
priorities in filings, tp/ts numbers etc).  We mostly wanted to get to a
point where we could create some metric which would then be calculated
and posted after a reasonable time period post release.

Having such a metric would also help in the creation of future goals.
If we could say that a release was considered 80% stable (or somesuch)
we could then aim to create something with a higher number the next go
around.  It would also help to isolate areas of improvement (ie, the
perf component of the metric caused a release to be 'bad' so we need to
work on that number to improve the overall).

For me, it was an attempt to get all the different information that we
gather into a tidy package.  I realize that this may be a pipe dream,
but it could go a long way to providing us with the information we need
for properly directly our efforts.

alice.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning