Changing our Standard for Bug Verification

classic Classic list List threaded Threaded
92 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Changing our Standard for Bug Verification

Alexander Keybl
My timing for sending this out was inspired by Gareth's thread on a "Proposal for Infrastructure to help out RIL Builds" on other lists.

I've been considering ways to ensure that all bugs landing to mozilla-central get some sort of manual testing prior to release. I'm curious if others would be in favor of changing the "Assigned to You" query in BMO's "My Dashboard" from being

        [all open bugs assigned to you]

to

        [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]

Developers could then use their dashboard to ensure that all of their landed bugs have gotten the proper testing after the fact in an actual Nightly build, at which point it would be verified/fixed. The verifyme keyword would only be used when there needed to be complex/large-scale manual testing or the verification required hardware that a developer didn't have access to.

I suspect that this will lead to an increase in quality, and a more obvious delineation between bugs QA or developers validate post-landing.

-Alex
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Bobby Holley-2
On Tue, Jun 11, 2013 at 3:56 PM, Alex Keybl <[hidden email]> wrote:

> I've been considering ways to ensure that all bugs landing to
> mozilla-central get some sort of manual testing prior to release.


Why? There are many bugs for which manual testing is impossible or makes no
sense. Refactorings aside, Many bugs are better verified with automated
tests than by hand. The vast majority of issues I work on are too subtle to
be even exercised without a very carefully-constructed testcase. I would
posit that the need for manual testing is the uncommon case.


> I'm curious if others would be in favor of changing the "Assigned to You"
> query in BMO's "My Dashboard" from being
>
>         [all open bugs assigned to you]
>
> to
>
>         [all bugs assigned to you that are open, or resolved/fixed if it
> doesn't have the verifyme keyword]
>
> Developers could then use their dashboard to ensure that all of their
> landed bugs have gotten the proper testing after the fact in an actual
> Nightly build, at which point it would be verified/fixed.


I can't imagine that anyone would actually do this. Even ignoring the
issues above, "bugs assigned to me" is not a meaningful query for most busy
Gecko developers.

bholley
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Alexander Keybl
> Why?

For sake of quality. There have been many bugs, in our experience, that would have benefited from just being verified in an actual Nightly. As of right now, no one is directly responsible for the verification of every single issue. We many times leave that to our pre-release populations, which makes me uncomfortable.

> There are many bugs for which manual testing is impossible or makes no sense.

Agreed, and perhaps this was an oversight in my high level proposal. There could always be a Resolved/Unverifiable state that's set at the time of landing, if unverifiable through manual testing.

> The vast majority of issues I work on are too subtle to be even exercised without a very carefully-constructed testcase. I would posit that the need for manual testing is the uncommon case.

Why can't verified mean that it landed with a testcase in that case, and it's unverified until it does? That would then subsume testcase-wanted, and the responsibility would more obviously fall on the landing developer.

> I can't imagine that anyone would actually do this. Even ignoring the issues above, "bugs assigned to me" is not a meaningful query for most busy Gecko developers.

What if we split this out into its own query, with a resolution date floor of today?

-Alex

On Jun 11, 2013, at 4:08 PM, Bobby Holley <[hidden email]> wrote:

> On Tue, Jun 11, 2013 at 3:56 PM, Alex Keybl <[hidden email]> wrote:
> I've been considering ways to ensure that all bugs landing to mozilla-central get some sort of manual testing prior to release.
>
> Why? There are many bugs for which manual testing is impossible or makes no sense. Refactorings aside, Many bugs are better verified with automated tests than by hand. The vast majority of issues I work on are too subtle to be even exercised without a very carefully-constructed testcase. I would posit that the need for manual testing is the uncommon case.
>  
> I'm curious if others would be in favor of changing the "Assigned to You" query in BMO's "My Dashboard" from being
>
>         [all open bugs assigned to you]
>
> to
>
>         [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]
>
> Developers could then use their dashboard to ensure that all of their landed bugs have gotten the proper testing after the fact in an actual Nightly build, at which point it would be verified/fixed.
>
> I can't imagine that anyone would actually do this. Even ignoring the issues above, "bugs assigned to me" is not a meaningful query for most busy Gecko developers.
>
> bholley

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

L. David Baron
In reply to this post by Alexander Keybl
On Tuesday 2013-06-11 15:56 -0700, Alex Keybl wrote:

> I've been considering ways to ensure that all bugs landing to
> mozilla-central get some sort of manual testing prior to release.
> I'm curious if others would be in favor of changing the "Assigned
> to You" query in BMO's "My Dashboard" from being
>
> [all open bugs assigned to you]
>
> to
>
> [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]


What is the purpose of bug verification in our system?  If we want
to encourage it, I think it should be like what I described in
https://bugzilla.mozilla.org/show_bug.cgi?id=172191#c16 , but I
don't think that's the definition that we currently use.  I also
don't think that's a definition that we have the resources to apply
to all bugs, or ought to.  I think it's a definition that we ought
to apply with good judgment of what needs testing and what doesn't.

-David

--
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                           http://www.mozilla.org/   𝄂
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Justin Lebar-3
In reply to this post by Alexander Keybl
Please don't forget that adding more tasks for developers will slow us
down.  If 1% of bugfixes don't actually work, and making developers
responsible for verifying every one of their fixes on nightly can
reduce that down to 0.5%, that is not necessarily a win, depending on
how much developer time this costs.

Verifying every one of my bugfixes in a nightly would probably take
about 10% of my time.  Is that worth catching one or two mistakes a
year?

In my experience and developers are in general pretty motivated to be
productive, so if you're getting pushback on this proposal, it's
probably because developers feel like it would make them less
productive overall.  I certainly feel that way.

I expect that very few people land code without testing it in a local
build first.  It's not clear to me that requiring that each developer
attest that they also tested it in a nightly would result in a
significant change to quality.  (Consider the matrix of possibilities:
Is a programmer who's careless enough not to test locally going to be
careful enough when testing nightly?  Is a programer who's careful
enough when testing locally likely to find a bug in nightly?  And so
on.)

On Tue, Jun 11, 2013 at 7:45 PM, Alex Keybl <[hidden email]> wrote:

>> Why?
>
> For sake of quality. There have been many bugs, in our experience, that would have benefited from just being verified in an actual Nightly. As of right now, no one is directly responsible for the verification of every single issue. We many times leave that to our pre-release populations, which makes me uncomfortable.
>
>> There are many bugs for which manual testing is impossible or makes no sense.
>
> Agreed, and perhaps this was an oversight in my high level proposal. There could always be a Resolved/Unverifiable state that's set at the time of landing, if unverifiable through manual testing.
>
>> The vast majority of issues I work on are too subtle to be even exercised without a very carefully-constructed testcase. I would posit that the need for manual testing is the uncommon case.
>
> Why can't verified mean that it landed with a testcase in that case, and it's unverified until it does? That would then subsume testcase-wanted, and the responsibility would more obviously fall on the landing developer.
>
>> I can't imagine that anyone would actually do this. Even ignoring the issues above, "bugs assigned to me" is not a meaningful query for most busy Gecko developers.
>
> What if we split this out into its own query, with a resolution date floor of today?
>
> -Alex
>
> On Jun 11, 2013, at 4:08 PM, Bobby Holley <[hidden email]> wrote:
>
>> On Tue, Jun 11, 2013 at 3:56 PM, Alex Keybl <[hidden email]> wrote:
>> I've been considering ways to ensure that all bugs landing to mozilla-central get some sort of manual testing prior to release.
>>
>> Why? There are many bugs for which manual testing is impossible or makes no sense. Refactorings aside, Many bugs are better verified with automated tests than by hand. The vast majority of issues I work on are too subtle to be even exercised without a very carefully-constructed testcase. I would posit that the need for manual testing is the uncommon case.
>>
>> I'm curious if others would be in favor of changing the "Assigned to You" query in BMO's "My Dashboard" from being
>>
>>         [all open bugs assigned to you]
>>
>> to
>>
>>         [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]
>>
>> Developers could then use their dashboard to ensure that all of their landed bugs have gotten the proper testing after the fact in an actual Nightly build, at which point it would be verified/fixed.
>>
>> I can't imagine that anyone would actually do this. Even ignoring the issues above, "bugs assigned to me" is not a meaningful query for most busy Gecko developers.
>>
>> bholley
>
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Alexander Keybl
In reply to this post by L. David Baron
Aren't you in favor of explicitly bucketing bugs as "verified", "awaiting verification", or "unverifiable" though? I don't think that puts too much pressure on developers, once we do define our bar for verification.

I agree that we should come up with a common definition of verified, but I don't think we shouldn't just leave it at r+ and landed with or without tests.

-Alex

On Jun 11, 2013, at 4:53 PM, "L. David Baron" <[hidden email]> wrote:

> On Tuesday 2013-06-11 15:56 -0700, Alex Keybl wrote:
>> I've been considering ways to ensure that all bugs landing to
>> mozilla-central get some sort of manual testing prior to release.
>> I'm curious if others would be in favor of changing the "Assigned
>> to You" query in BMO's "My Dashboard" from being
>>
>> [all open bugs assigned to you]
>>
>> to
>>
>> [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]
>
>
> What is the purpose of bug verification in our system?  If we want
> to encourage it, I think it should be like what I described in
> https://bugzilla.mozilla.org/show_bug.cgi?id=172191#c16 , but I
> don't think that's the definition that we currently use.  I also
> don't think that's a definition that we have the resources to apply
> to all bugs, or ought to.  I think it's a definition that we ought
> to apply with good judgment of what needs testing and what doesn't.
>
> -David
>
> --
> 𝄞   L. David Baron                         http://dbaron.org/   𝄂
> 𝄢   Mozilla                           http://www.mozilla.org/   𝄂

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Alexander Keybl
In reply to this post by Justin Lebar-3
> In my experience and developers are in general pretty motivated to be
> productive, so if you're getting pushback on this proposal, it's
> probably because developers feel like it would make them less
> productive overall.  I certainly feel that way.

I have a strong suspicion that this is tied to a definition of productivity that may not take into consideration the organizational cost of dot releases, late partner blockers, etc. All of the related work also turns into technical debt, and may actually take up a significant amount of your time.

> Verifying every one of my bugfixes in a nightly would probably take
> about 10% of my time.  Is that worth catching one or two mistakes a
> year?

I think that depends on our definition of verification (manual testing, landing a test case, etc). Especially on projects like B2G where many of the regressions we find would be quickly identified if a high level verification pass was completed. Perhaps this policy does incur too much overhead for desktop/mobile, but we may still decide to discuss this for B2G specifically at least until full automated testing comes together.

-Alex

On Jun 11, 2013, at 4:57 PM, Justin Lebar <[hidden email]> wrote:

> Please don't forget that adding more tasks for developers will slow us
> down.  If 1% of bugfixes don't actually work, and making developers
> responsible for verifying every one of their fixes on nightly can
> reduce that down to 0.5%, that is not necessarily a win, depending on
> how much developer time this costs.
>
> Verifying every one of my bugfixes in a nightly would probably take
> about 10% of my time.  Is that worth catching one or two mistakes a
> year?
>
> In my experience and developers are in general pretty motivated to be
> productive, so if you're getting pushback on this proposal, it's
> probably because developers feel like it would make them less
> productive overall.  I certainly feel that way.
>
> I expect that very few people land code without testing it in a local
> build first.  It's not clear to me that requiring that each developer
> attest that they also tested it in a nightly would result in a
> significant change to quality.  (Consider the matrix of possibilities:
> Is a programmer who's careless enough not to test locally going to be
> careful enough when testing nightly?  Is a programer who's careful
> enough when testing locally likely to find a bug in nightly?  And so
> on.)
>
> On Tue, Jun 11, 2013 at 7:45 PM, Alex Keybl <[hidden email]> wrote:
>>> Why?
>>
>> For sake of quality. There have been many bugs, in our experience, that would have benefited from just being verified in an actual Nightly. As of right now, no one is directly responsible for the verification of every single issue. We many times leave that to our pre-release populations, which makes me uncomfortable.
>>
>>> There are many bugs for which manual testing is impossible or makes no sense.
>>
>> Agreed, and perhaps this was an oversight in my high level proposal. There could always be a Resolved/Unverifiable state that's set at the time of landing, if unverifiable through manual testing.
>>
>>> The vast majority of issues I work on are too subtle to be even exercised without a very carefully-constructed testcase. I would posit that the need for manual testing is the uncommon case.
>>
>> Why can't verified mean that it landed with a testcase in that case, and it's unverified until it does? That would then subsume testcase-wanted, and the responsibility would more obviously fall on the landing developer.
>>
>>> I can't imagine that anyone would actually do this. Even ignoring the issues above, "bugs assigned to me" is not a meaningful query for most busy Gecko developers.
>>
>> What if we split this out into its own query, with a resolution date floor of today?
>>
>> -Alex
>>
>> On Jun 11, 2013, at 4:08 PM, Bobby Holley <[hidden email]> wrote:
>>
>>> On Tue, Jun 11, 2013 at 3:56 PM, Alex Keybl <[hidden email]> wrote:
>>> I've been considering ways to ensure that all bugs landing to mozilla-central get some sort of manual testing prior to release.
>>>
>>> Why? There are many bugs for which manual testing is impossible or makes no sense. Refactorings aside, Many bugs are better verified with automated tests than by hand. The vast majority of issues I work on are too subtle to be even exercised without a very carefully-constructed testcase. I would posit that the need for manual testing is the uncommon case.
>>>
>>> I'm curious if others would be in favor of changing the "Assigned to You" query in BMO's "My Dashboard" from being
>>>
>>>        [all open bugs assigned to you]
>>>
>>> to
>>>
>>>        [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]
>>>
>>> Developers could then use their dashboard to ensure that all of their landed bugs have gotten the proper testing after the fact in an actual Nightly build, at which point it would be verified/fixed.
>>>
>>> I can't imagine that anyone would actually do this. Even ignoring the issues above, "bugs assigned to me" is not a meaningful query for most busy Gecko developers.
>>>
>>> bholley
>>
>> _______________________________________________
>> dev-planning mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Justin Lebar-3
> we may still decide to discuss this for B2G specifically at least until full automated testing comes together.

We don't have anything resembling full automated testing in desktop, either.

On Tue, Jun 11, 2013 at 8:24 PM, Alex Keybl <[hidden email]> wrote:

>> In my experience and developers are in general pretty motivated to be
>> productive, so if you're getting pushback on this proposal, it's
>> probably because developers feel like it would make them less
>> productive overall.  I certainly feel that way.
>
> I have a strong suspicion that this is tied to a definition of productivity that may not take into consideration the organizational cost of dot releases, late partner blockers, etc. All of the related work also turns into technical debt, and may actually take up a significant amount of your time.
>
>> Verifying every one of my bugfixes in a nightly would probably take
>> about 10% of my time.  Is that worth catching one or two mistakes a
>> year?
>
> I think that depends on our definition of verification (manual testing, landing a test case, etc). Especially on projects like B2G where many of the regressions we find would be quickly identified if a high level verification pass was completed. Perhaps this policy does incur too much overhead for desktop/mobile, but we may still decide to discuss this for B2G specifically at least until full automated testing comes together.
>
> -Alex
>
> On Jun 11, 2013, at 4:57 PM, Justin Lebar <[hidden email]> wrote:
>
>> Please don't forget that adding more tasks for developers will slow us
>> down.  If 1% of bugfixes don't actually work, and making developers
>> responsible for verifying every one of their fixes on nightly can
>> reduce that down to 0.5%, that is not necessarily a win, depending on
>> how much developer time this costs.
>>
>> Verifying every one of my bugfixes in a nightly would probably take
>> about 10% of my time.  Is that worth catching one or two mistakes a
>> year?
>>
>> In my experience and developers are in general pretty motivated to be
>> productive, so if you're getting pushback on this proposal, it's
>> probably because developers feel like it would make them less
>> productive overall.  I certainly feel that way.
>>
>> I expect that very few people land code without testing it in a local
>> build first.  It's not clear to me that requiring that each developer
>> attest that they also tested it in a nightly would result in a
>> significant change to quality.  (Consider the matrix of possibilities:
>> Is a programmer who's careless enough not to test locally going to be
>> careful enough when testing nightly?  Is a programer who's careful
>> enough when testing locally likely to find a bug in nightly?  And so
>> on.)
>>
>> On Tue, Jun 11, 2013 at 7:45 PM, Alex Keybl <[hidden email]> wrote:
>>>> Why?
>>>
>>> For sake of quality. There have been many bugs, in our experience, that would have benefited from just being verified in an actual Nightly. As of right now, no one is directly responsible for the verification of every single issue. We many times leave that to our pre-release populations, which makes me uncomfortable.
>>>
>>>> There are many bugs for which manual testing is impossible or makes no sense.
>>>
>>> Agreed, and perhaps this was an oversight in my high level proposal. There could always be a Resolved/Unverifiable state that's set at the time of landing, if unverifiable through manual testing.
>>>
>>>> The vast majority of issues I work on are too subtle to be even exercised without a very carefully-constructed testcase. I would posit that the need for manual testing is the uncommon case.
>>>
>>> Why can't verified mean that it landed with a testcase in that case, and it's unverified until it does? That would then subsume testcase-wanted, and the responsibility would more obviously fall on the landing developer.
>>>
>>>> I can't imagine that anyone would actually do this. Even ignoring the issues above, "bugs assigned to me" is not a meaningful query for most busy Gecko developers.
>>>
>>> What if we split this out into its own query, with a resolution date floor of today?
>>>
>>> -Alex
>>>
>>> On Jun 11, 2013, at 4:08 PM, Bobby Holley <[hidden email]> wrote:
>>>
>>>> On Tue, Jun 11, 2013 at 3:56 PM, Alex Keybl <[hidden email]> wrote:
>>>> I've been considering ways to ensure that all bugs landing to mozilla-central get some sort of manual testing prior to release.
>>>>
>>>> Why? There are many bugs for which manual testing is impossible or makes no sense. Refactorings aside, Many bugs are better verified with automated tests than by hand. The vast majority of issues I work on are too subtle to be even exercised without a very carefully-constructed testcase. I would posit that the need for manual testing is the uncommon case.
>>>>
>>>> I'm curious if others would be in favor of changing the "Assigned to You" query in BMO's "My Dashboard" from being
>>>>
>>>>        [all open bugs assigned to you]
>>>>
>>>> to
>>>>
>>>>        [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]
>>>>
>>>> Developers could then use their dashboard to ensure that all of their landed bugs have gotten the proper testing after the fact in an actual Nightly build, at which point it would be verified/fixed.
>>>>
>>>> I can't imagine that anyone would actually do this. Even ignoring the issues above, "bugs assigned to me" is not a meaningful query for most busy Gecko developers.
>>>>
>>>> bholley
>>>
>>> _______________________________________________
>>> dev-planning mailing list
>>> [hidden email]
>>> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

L. David Baron
In reply to this post by Alexander Keybl
On Tuesday 2013-06-11 17:15 -0700, Alex Keybl wrote:
> Aren't you in favor of explicitly bucketing bugs as "verified",
> "awaiting verification", or "unverifiable" though? I don't think
> that puts too much pressure on developers, once we do define our
> bar for verification.

No.  I don't think it makes any sense unless we have an idea of what
the purpose of verification is.

I'd also like to see QA team members empowered to use whatever
approach they conclude is most likely to improve quality, rather
than having a process that requires verification of bugs where
verification (whatever it is) is not likely to be worthwhile.

-David

--
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                           http://www.mozilla.org/   𝄂
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Bobby Holley-2
In reply to this post by Alexander Keybl
On Tue, Jun 11, 2013 at 4:45 PM, Alex Keybl <[hidden email]> wrote:

> Why can't verified mean that it landed with a testcase in that case, and
> it's unverified until it does? That would then subsume testcase-wanted, and
> the responsibility would more obviously fall on the landing developer.


I can't remember the last time I pushed a changeset without tests that
would have been possible to verify manually. So I might just be living in a
very different world that the developers you're targeting here. But if
someone is too busy to write tests, it seems unlikely that you're going to
get traction with them verifying their own bugs.

In the type of engineering I do, test coverage is orders of magnitude more
useful than manual testing. I can imagine things might be different in
other areas (like frontend), so I can't speak for everyone. But in general
I think energy is better spent getting people to write tests.


> I can't imagine that anyone would actually do this. Even ignoring the
> issues above, "bugs assigned to me" is not a meaningful query for most busy
> Gecko developers.
>
> What if we split this out into its own query, with a resolution date floor
> of today?
>

My point is that developers rarely look at these queries, whatever they
might be. I'm probably one of the only developers that actively tracks my
in-testsuite? bugs, mostly because I do a lot of work on security bugs and
want to make sure that my tests eventually get checked in. For better or
for worse, our culture relies on poking people (pings, email whines, etc)
to signal importance and get things done. And adding more whines for more
things just dilutes the importance of the whine. We already ping people
about security bugs, tracked bugs, blockers, and things that individual
people in the org want to push forward. I think we should be hesitant to
add more.

bholley
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Benjamin Smedberg
In reply to this post by Alexander Keybl
On 6/11/2013 6:56 PM, Alex Keybl wrote:

> I've been considering ways to ensure that all bugs landing to mozilla-central get some sort of manual testing prior to release. I'm curious if others would be in favor of changing the "Assigned to You" query in BMO's "My Dashboard" from being
>
> [all open bugs assigned to you]
>
> to
>
> [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]
>
> Developers could then use their dashboard to ensure that all of their landed bugs have gotten the proper testing after the fact in an actual Nightly build, at which point it would be verified/fixed. The verifyme keyword would only be used when there needed to be complex/large-scale manual testing or the verification required hardware that a developer didn't have access to.
>
> I suspect that this will lead to an increase in quality, and a more obvious delineation between bugs QA or developers validate post-landing.

I do *not* think that developers should be responsible for verifying
their own bugs, if we actually do need a verification step. Developers
are likely to just test the use cases that they already had in mind when
they were developing the patch in the first place. The whole point of
external verification is that you have somebody with fewer preconceived
notions about a change who can report on behaviors that the original
developer and reviewer were not considering and were not included in any
automated tests.

I also don't think that the proposed technical change is going to change
anyone's behavior. I may be mistaken, but I don't think that changing
something in the BMO dashboard is going to change behaviors, since I
haven't heard of anyone using those dashboards. I didn't even know that
dashboard existed until your message, and the existing tools I have
present better information.

--BDS

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Mike Connor-4
In reply to this post by Alexander Keybl
On 2013-06-11 6:56 PM, Alex Keybl wrote:
> My timing for sending this out was inspired by Gareth's thread on a "Proposal for Infrastructure to help out RIL Builds" on other lists.
>
> I've been considering ways to ensure that all bugs landing to mozilla-central get some sort of manual testing prior to release. I'm curious if others would be in favor of changing the "Assigned to You" query in BMO's "My Dashboard" from being

Is this necessary for all bugs, or just bugs not covered by appropriate
automated tests?  If there's a test (and there generally should be) then
I think the requirement is unduly onerous in those cases.  I think
there's an open question about what QA considers "verified" where
automated tests don't exist, but I don't think we need to define that
across the project. I think QA needs to define that per-area since it's
their responsibility to enforce a quality bar.  dbaron's comments also
suggest that we at least know some bugs are "unverifiable" in various
ways (i.e. race conditions and other difficult to reproduce issues) and
we can/should identify and mark those bugs so QA doesn't spend time
trying to sort those out.

> [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]

could change to:

[all bugs assigned to you that are open, or resolved/fixed if it doesn't have one of verifyme, unverifiable, in-testsuite+]

This would:

* help focus QA effort on the class of bugs where they are most needed and where we'll get the best ROI
* give us an (imprecise) idea of how we're doing in terms of automated testing and where we probably need better frameworks
* minimize the effort involved for developers

I don't think this categorization would take a developer more than 30 seconds per bug they fix.

-- Mike



_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Alexander Keybl
In reply to this post by Benjamin Smedberg
> Developers are likely to just test the use cases that they already had in mind when they were developing the patch in the first place.

One thing that concerns me when we talk about these sorts of changes is that we feel generally powerless to request a bit more process from devs. Verifying that bugs landed as intended is not atypical as a developer responsibility. I agree it's not as good as QAing every landing, but we're trying to optimize here.

> I also don't think that the proposed technical change is going to change anyone's behavior. I may be mistaken, but I don't think that changing something in the BMO dashboard is going to change behaviors, since I haven't heard of anyone using those dashboards. I didn't even know that dashboard existed until your message, and the existing tools I have present better information.

What's the best way to put a process change in from of devs in your opinion? BMO seemed like a good candidate because I was under the impression more were using its assigned/needinfo dash. I don't think verification can be something that we roll out more organically, since it's obviously something nobody really wants to spend their time doing.

-Alex

On Jun 12, 2013, at 7:59 AM, Benjamin Smedberg <[hidden email]> wrote:

> On 6/11/2013 6:56 PM, Alex Keybl wrote:
>> I've been considering ways to ensure that all bugs landing to mozilla-central get some sort of manual testing prior to release. I'm curious if others would be in favor of changing the "Assigned to You" query in BMO's "My Dashboard" from being
>>
>> [all open bugs assigned to you]
>>
>> to
>>
>> [all bugs assigned to you that are open, or resolved/fixed if it doesn't have the verifyme keyword]
>>
>> Developers could then use their dashboard to ensure that all of their landed bugs have gotten the proper testing after the fact in an actual Nightly build, at which point it would be verified/fixed. The verifyme keyword would only be used when there needed to be complex/large-scale manual testing or the verification required hardware that a developer didn't have access to.
>>
>> I suspect that this will lead to an increase in quality, and a more obvious delineation between bugs QA or developers validate post-landing.
>
> I do *not* think that developers should be responsible for verifying their own bugs, if we actually do need a verification step. Developers are likely to just test the use cases that they already had in mind when they were developing the patch in the first place. The whole point of external verification is that you have somebody with fewer preconceived notions about a change who can report on behaviors that the original developer and reviewer were not considering and were not included in any automated tests.
>
> I also don't think that the proposed technical change is going to change anyone's behavior. I may be mistaken, but I don't think that changing something in the BMO dashboard is going to change behaviors, since I haven't heard of anyone using those dashboards. I didn't even know that dashboard existed until your message, and the existing tools I have present better information.
>
> --BDS
>

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Ehsan Akhgari
On 2013-06-12 12:55 PM, Alex Keybl wrote:
>> I also don't think that the proposed technical change is going to change anyone's behavior. I may be mistaken, but I don't think that changing something in the BMO dashboard is going to change behaviors, since I haven't heard of anyone using those dashboards. I didn't even know that dashboard existed until your message, and the existing tools I have present better information.
>
> What's the best way to put a process change in from of devs in your opinion? BMO seemed like a good candidate because I was under the impression more were using its assigned/needinfo dash. I don't think verification can be something that we roll out more organically, since it's obviously something nobody really wants to spend their time doing.

Trying to find the best way to do this is putting the cart before the
horse.  If you convince developers that doing this is actually useful,
I'm sure they will come up with ways to find the list of bugs that they
need to verify.  That's not a very hard problem to solve.

Cheers,
Ehsan
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Alexander Keybl
> Trying to find the best way to do this is putting the cart before the horse.  If you convince developers that doing this is actually useful, I'm sure they will come up with ways to find the list of bugs that they need to verify.  That's not a very hard problem to solve.

What if it's not something that anyone really wants to do, it's just something that needs to happen to ensure a minimum quality bar? No pain no gain. The suggestion of socializing verification through BMO would only be a method of setting expectations, similar to how we use a patch r+ as part of normal bug process.

-Alex

On Jun 12, 2013, at 10:28 AM, Ehsan Akhgari <[hidden email]> wrote:

> On 2013-06-12 12:55 PM, Alex Keybl wrote:
>>> I also don't think that the proposed technical change is going to change anyone's behavior. I may be mistaken, but I don't think that changing something in the BMO dashboard is going to change behaviors, since I haven't heard of anyone using those dashboards. I didn't even know that dashboard existed until your message, and the existing tools I have present better information.
>>
>> What's the best way to put a process change in from of devs in your opinion? BMO seemed like a good candidate because I was under the impression more were using its assigned/needinfo dash. I don't think verification can be something that we roll out more organically, since it's obviously something nobody really wants to spend their time doing.
>
> Trying to find the best way to do this is putting the cart before the horse.  If you convince developers that doing this is actually useful, I'm sure they will come up with ways to find the list of bugs that they need to verify.  That's not a very hard problem to solve.
>
> Cheers,
> Ehsan

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

mhoye
In reply to this post by Mike Connor-4


----- Original Message -----

> From: "Mike Connor" <[hidden email]>
>
> [all bugs assigned to you that are open, or resolved/fixed if it
> doesn't have one of verifyme, unverifiable, in-testsuite+]
>
> This would:
>
> * help focus QA effort on the class of bugs where they are most
> needed and where we'll get the best ROI
> * give us an (imprecise) idea of how we're doing in terms of
> automated testing and where we probably need better frameworks
> * minimize the effort involved for developers

In addition, a flag indicating that a fix needs to be manually verified would be a very nice, low-barrier onramp for community contribution. The next step from a community perspective is a dashboard that says, if you've got $FOO OS on $BAR hardware, please pick one of these bugs to verify manually and report back.



- mhoye
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Ehsan Akhgari
In reply to this post by Alexander Keybl
On 2013-06-12 1:32 PM, Alex Keybl wrote:

>> Trying to find the best way to do this is putting the cart before the horse.  If you convince developers that doing this is actually useful, I'm sure they will come up with ways to find the list of bugs that they need to verify.  That's not a very hard problem to solve.
>
> What if it's not something that anyone really wants to do, it's just something that needs to happen to ensure a minimum quality bar? No pain no gain. The suggestion of socializing verification through BMO would only be a method of setting expectations, similar to how we use a patch r+ as part of normal bug process.
>
> -Alex
>
> On Jun 12, 2013, at 10:28 AM, Ehsan Akhgari <[hidden email]> wrote:
>
>> On 2013-06-12 12:55 PM, Alex Keybl wrote:
>>>> I also don't think that the proposed technical change is going to change anyone's behavior. I may be mistaken, but I don't think that changing something in the BMO dashboard is going to change behaviors, since I haven't heard of anyone using those dashboards. I didn't even know that dashboard existed until your message, and the existing tools I have present better information.
>>>
>>> What's the best way to put a process change in from of devs in your opinion? BMO seemed like a good candidate because I was under the impression more were using its assigned/needinfo dash. I don't think verification can be something that we roll out more organically, since it's obviously something nobody really wants to spend their time doing.
>>
>> Trying to find the best way to do this is putting the cart before the horse.  If you convince developers that doing this is actually useful, I'm sure they will come up with ways to find the list of bugs that they need to verify.  That's not a very hard problem to solve.

That won't work, because there is not one single Bugzilla workflow.  I
for example don't remember the last time that I looked at the "Bugs
assigned to me" query, and in general the mere fact that the assigned to
field on a bug is set to my email address says nothing about whether I'm
actually working on it.  I can name quite a few other developers in the
same boat.

Cheers,
Ehsan

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Kyle Huey-2
In reply to this post by Alexander Keybl
Maybe we should step back a bit.  What is the actual problem we are trying
to solve here?  I have yet to see that mentioned in this thread.

- Kyle
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Alexander Keybl
There are many major issues (B2G comes to mind, but this is true everywhere) that could have been avoided by doing a simple high-level pass on landed code. This is especially true of code landing without an automated testcase, and bugs that are left unverified by QA.

I'd like us to change our standard of verification, such that all landings are looked at post-landing by somebody. And that person notes in the bug that it has been verified. My suggestion for scaling this would be to ask the developers to spend the couple of minutes it takes to fire up the browser and run through a user story, test case, etc.

-Alex

On Jun 12, 2013, at 10:42 AM, Kyle Huey <[hidden email]> wrote:

> Maybe we should step back a bit.  What is the actual problem we are trying to solve here?  I have yet to see that mentioned in this thread.
>
> - Kyle

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Changing our Standard for Bug Verification

Ehsan Akhgari
On 2013-06-12 1:45 PM, Alex Keybl wrote:
> There are many major issues (B2G comes to mind, but this is true everywhere) that could have been avoided by doing a simple high-level pass on landed code. This is especially true of code landing without an automated testcase, and bugs that are left unverified by QA.
>
> I'd like us to change our standard of verification, such that all landings are looked at post-landing by somebody. And that person notes in the bug that it has been verified. My suggestion for scaling this would be to ask the developers to spend the couple of minutes it takes to fire up the browser and run through a user story, test case, etc.

Why are the developers the best people to do this kind of verification?

Ehsan

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
12345