Improving Fix Verification (v2)

classic Classic list List threaded Threaded
45 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Improving Fix Verification (v2)

Mike Connor-4
Hi all,

Many of you will recall that Alex Keybl proposed a change recently that
would have asked developers to manually verify all of their patches
after landing.  Understandably, this was met with significant concerns
regarding effective utilization of developer time, and the thread
quickly became very hard to follow.  In the very long discussion, a
heavily revised concept seemed to gain consensus, and it was suggested
that a new thread be started to discuss that revised proposal.

The proposal I'm going to link to below is intended to be
straightforward, and seeks to build upon the following three core
principles:

* For all fixed bugs, the state of test coverage should be clear and
unambiguous
* Developers have the best insight into which bugs carry the most risk
and require attention from QA
* QA is the best group to effectively test fixes, and has the best
insight into which bugs require additional testing and verification

The thrust of the proposal is simple: we should replace the current
assortment of keywords, flags, and whiteboard annotations with a single
multi-state flag (testing-status), and make setting this flag part of
the required workflow when resolving a bug in a product component as
FIXED.   Developers will be asked to assess whether a bug is either: a)
not testable, b) covered sufficiently by automated tests or c) requires
QA to assess the bug.  QA then will determine the amount and type of
testing required, and take action as appropriate.  It is intended that
this formalization of the process will provide us visibility into our
testing coverage and resource investment across our products.

The full details, including a visual workflow diagram, can be found at
https://wiki.mozilla.org/User:Mconnor/BugVerification

Any question or comments are welcome here, or you can find me on IRC.

-- Mike

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Gavin Sharp-3
https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow
looks pretty good. One thing that comes to mind, though, is most of
the time developers can determine "automated-test-needed" without help
from QA, so there should be a path to that state that doesn't involve
QA. In practice this is often "could be tested, but don't have time to
write test now". In fact I can't think of a scenario in which QAs help
would be required to determine automated-testability, and so it seems
to me like testing-needed could be renamed manual-testing-needed, and
the "QA: Is there a framework?" section of the chart could be tacked
on to the Developer "Is verification needed?" branch.

Gavin

On Mon, Jul 8, 2013 at 11:56 AM, Mike Connor <[hidden email]> wrote:

> Hi all,
>
> Many of you will recall that Alex Keybl proposed a change recently that
> would have asked developers to manually verify all of their patches after
> landing.  Understandably, this was met with significant concerns regarding
> effective utilization of developer time, and the thread quickly became very
> hard to follow.  In the very long discussion, a heavily revised concept
> seemed to gain consensus, and it was suggested that a new thread be started
> to discuss that revised proposal.
>
> The proposal I'm going to link to below is intended to be straightforward,
> and seeks to build upon the following three core principles:
>
> * For all fixed bugs, the state of test coverage should be clear and
> unambiguous
> * Developers have the best insight into which bugs carry the most risk and
> require attention from QA
> * QA is the best group to effectively test fixes, and has the best insight
> into which bugs require additional testing and verification
>
> The thrust of the proposal is simple: we should replace the current
> assortment of keywords, flags, and whiteboard annotations with a single
> multi-state flag (testing-status), and make setting this flag part of the
> required workflow when resolving a bug in a product component as FIXED.
> Developers will be asked to assess whether a bug is either: a) not testable,
> b) covered sufficiently by automated tests or c) requires QA to assess the
> bug.  QA then will determine the amount and type of testing required, and
> take action as appropriate.  It is intended that this formalization of the
> process will provide us visibility into our testing coverage and resource
> investment across our products.
>
> The full details, including a visual workflow diagram, can be found at
> https://wiki.mozilla.org/User:Mconnor/BugVerification
>
> Any question or comments are welcome here, or you can find me on IRC.
>
> -- Mike
>
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Gavin Sharp-3
On Mon, Jul 8, 2013 at 5:00 PM, Gavin Sharp <[hidden email]> wrote:
> https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow

One other comment: in our development process, I think it's most often
true that the developer is in the best position to make a judgement
about the potential utility of QA verification, but I don't think
that's ideal. They can be biased by their closeness to the problem
and/or laziness and end up making the wrong call. In an ideal world I
think a separate QA person would actually be responsible for
determining "is verification needed" (first question in the workflow
diagram), rather than the developer. But that often requires a deeper
understanding of the change that was made than QA people have (and we
don't have enough people doing QA to cover all bugs being fixed).

Given this, I think it's fine for your proposed workflow to put the
responsibility for asking that first question on the developer, but it
should probably also allow for that question to be answered by a QA
person in areas with sufficient QA resources.

Gavin
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Mike Connor-4
In reply to this post by Gavin Sharp-3
On 2013-07-08, at 8:00 PM, Gavin Sharp <[hidden email]> wrote:

> https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow
> looks pretty good. One thing that comes to mind, though, is most of
> the time developers can determine "automated-test-needed" without help
> from QA, so there should be a path to that state that doesn't involve
> QA. In practice this is often "could be tested, but don't have time to
> write test now". In fact I can't think of a scenario in which QAs help
> would be required to determine automated-testability, and so it seems
> to me like testing-needed could be renamed manual-testing-needed, and
> the "QA: Is there a framework?" section of the chart could be tacked
> on to the Developer "Is verification needed?" branch.

That's what I started with as well!  The reason I ended up with this particular workflow is that, in the cases where automated testing is not yet sufficient, I explicitly want QA to be looking at those bugs to assess/verify as needed.  There's probably some cases where developers will assert that manual testing cannot be useful, so they can set that value directly, but I suspect those are a relatively small minority.

-- Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Mike Connor-4
In reply to this post by Gavin Sharp-3
On 2013-07-10, at 3:45 PM, Gavin Sharp <[hidden email]> wrote:

> On Mon, Jul 8, 2013 at 5:00 PM, Gavin Sharp <[hidden email]> wrote:
>> https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow
>
> One other comment: in our development process, I think it's most often
> true that the developer is in the best position to make a judgement
> about the potential utility of QA verification, but I don't think
> that's ideal. They can be biased by their closeness to the problem
> and/or laziness and end up making the wrong call. In an ideal world I
> think a separate QA person would actually be responsible for
> determining "is verification needed" (first question in the workflow
> diagram), rather than the developer. But that often requires a deeper
> understanding of the change that was made than QA people have (and we
> don't have enough people doing QA to cover all bugs being fixed).
>
> Given this, I think it's fine for your proposed workflow to put the
> responsibility for asking that first question on the developer, but it
> should probably also allow for that question to be answered by a QA
> person in areas with sufficient QA resources.

I suspect/hope that in any areas with sufficient QA resources to make these calls, those QA engineers are watching bugs and can change those values.  I believe QA should always feel empowered to push developers on quality issues where they feel there's insufficient rigour on the part of developers.

-- Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

beltzner
In reply to this post by Gavin Sharp-3
Could this be a role for the code reviewer? Or perhaps when submitted for
review, the author should indicate their intention for the flag and give
rationale.

(Amongst other things, this would also ensure the flag gets set!)

On Wednesday, July 10, 2013, Gavin Sharp wrote:

> On Mon, Jul 8, 2013 at 5:00 PM, Gavin Sharp <[hidden email]<javascript:;>>
> wrote:
> > https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow
>
> One other comment: in our development process, I think it's most often
> true that the developer is in the best position to make a judgement
> about the potential utility of QA verification, but I don't think
> that's ideal. They can be biased by their closeness to the problem
> and/or laziness and end up making the wrong call. In an ideal world I
> think a separate QA person would actually be responsible for
> determining "is verification needed" (first question in the workflow
> diagram), rather than the developer. But that often requires a deeper
> understanding of the change that was made than QA people have (and we
> don't have enough people doing QA to cover all bugs being fixed).
>
> Given this, I think it's fine for your proposed workflow to put the
> responsibility for asking that first question on the developer, but it
> should probably also allow for that question to be answered by a QA
> person in areas with sufficient QA resources.
>
> Gavin
> _______________________________________________
> dev-planning mailing list
> [hidden email] <javascript:;>
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Zack Weinberg-2
In reply to this post by Mike Connor-4
On 2013-07-08 8:00 PM, Gavin Sharp wrote:
> ... most of the time developers can determine "automated-test-needed"
> without help from QA, so there should be a path to that state that
> doesn't involve QA. In practice this is often "could be tested, but
> don't have time to write test now".

Closely related thing: the kind of work I do, often I find it is very
difficult for *me* to write the automated tests because I am a little
*too* familiar with the guts of the thing being tested.  (Case in point:
some of the CSS "does this parse correctly?" tests were written with
intimate knowledge of the control flow in the parser, and this actually
led to a spec bug because I didn't notice an edge case that gets lumped
together with a more common case in *our* parser, but needs special
wording in the spec.  (Handling of bad-url right before stylesheet EOF,
if you're really curious.))

So it would be nice if this process included a way for devs to request
that *someone else* write an automated, black-box test for something,
working from the spec rather than the code.  (Naturally, "sorry we don't
have the manpower" would be an acceptable response to this request.)

zw
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Naoki Hirata
In response to "devs to request that *someone else* write an automated,
black-box test for something, working from the spec rather than the code."

I agree with the dev requesting testing.

I would like to say that I tend to find more bugs when I look at the
spec and the code rather than just the spec alone or the code alone.

I think Zack makes a lot of good points in his statements.

Regards,
Naoki


On 7/10/13 4:44 PM, Zack Weinberg wrote:

> On 2013-07-08 8:00 PM, Gavin Sharp wrote:
>> ... most of the time developers can determine "automated-test-needed"
>> without help from QA, so there should be a path to that state that
>> doesn't involve QA. In practice this is often "could be tested, but
>> don't have time to write test now".
>
> Closely related thing: the kind of work I do, often I find it is very
> difficult for *me* to write the automated tests because I am a little
> *too* familiar with the guts of the thing being tested. (Case in
> point: some of the CSS "does this parse correctly?" tests were written
> with intimate knowledge of the control flow in the parser, and this
> actually led to a spec bug because I didn't notice an edge case that
> gets lumped together with a more common case in *our* parser, but
> needs special wording in the spec. (Handling of bad-url right before
> stylesheet EOF, if you're really curious.))
>
> So it would be nice if this process included a way for devs to request
> that *someone else* write an automated, black-box test for something,
> working from the spec rather than the code. (Naturally, "sorry we
> don't have the manpower" would be an acceptable response to this
> request.)
>
> zw
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Martin Best
I would highly recommend that we get QA involved in this discussion very early before this discussion goes much further. I've had several similar discussions and their resource limitations often have a significant impact on what can be done. It's important to have their insight during these early discussions.

Good to see this discussion.

Martin

----- Original Message -----

From: "平田修樹 (Naoki Hirata)" <[hidden email]>
To: [hidden email]
Sent: Thursday, July 11, 2013 3:21:15 AM
Subject: Re: Improving Fix Verification (v2)

In response to "devs to request that *someone else* write an automated,
black-box test for something, working from the spec rather than the code."

I agree with the dev requesting testing.

I would like to say that I tend to find more bugs when I look at the
spec and the code rather than just the spec alone or the code alone.

I think Zack makes a lot of good points in his statements.

Regards,
Naoki


On 7/10/13 4:44 PM, Zack Weinberg wrote:

> On 2013-07-08 8:00 PM, Gavin Sharp wrote:
>> ... most of the time developers can determine "automated-test-needed"
>> without help from QA, so there should be a path to that state that
>> doesn't involve QA. In practice this is often "could be tested, but
>> don't have time to write test now".
>
> Closely related thing: the kind of work I do, often I find it is very
> difficult for *me* to write the automated tests because I am a little
> *too* familiar with the guts of the thing being tested. (Case in
> point: some of the CSS "does this parse correctly?" tests were written
> with intimate knowledge of the control flow in the parser, and this
> actually led to a spec bug because I didn't notice an edge case that
> gets lumped together with a more common case in *our* parser, but
> needs special wording in the spec. (Handling of bad-url right before
> stylesheet EOF, if you're really curious.))
>
> So it would be nice if this process included a way for devs to request
> that *someone else* write an automated, black-box test for something,
> working from the spec rather than the code. (Naturally, "sorry we
> don't have the manpower" would be an acceptable response to this
> request.)
>
> zw
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning 

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning 

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Clint Talbert-3
I think this is a great step forward with verifications, and I'm all for
getting rid of the confusing mess of flags and whiteboard fields we
currently use to track this stuff. Most of the time when we get
questions about automated tests that need to be written or frameworks
that are needed for such tests, those tend to come from development. I
think this is a function of the fact that development outnumbers QA 3 -4
to 1 here.  So, as long as developers can just as easily set the
"automated-test-needed"/"framework-needed" values, I think we have a
great strategy for moving forward.

I'd also like to see this folded into the review process as a means to
ensure that the flag is properly set and that the issue is discussed.

Thanks for putting this together, Mike.


On 7/11/2013 1:36 AM, Martin Best wrote:
> I would highly recommend that we get QA involved in this discussion very early before this discussion goes much further. I've had several similar discussions and their resource limitations often have a significant impact on what can be done. It's important to have their insight during these early discussions.
>
>
I've sent a note to the internal QA alias asking them to look over this
thread and post comments/questions as necessary.

Clint
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Matt Wobensmith
In reply to this post by Martin Best
I think this is a great step in the direction of simplification. We may have some bumps when we implement new process, but overall I think this is important. Thanks to all who have had the patience to put this together.

Matt Wobensmith
QA Engineer
Mozilla Corporation
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

jsmith.mozilla
Comments:

I would get rid of the immediate step of "testing-needed" and just have the developer directly set either framework-needed, automated-test-needed, or manual-test-needed directly. There will be cases where I think there will be disagreement in the case where say, a QA comes through analyzing the bugs and disagrees with the resolution. But in that case, I think the developer and QA can just work together to come to agreement when there's disagreement on the testing-status state.

QA is limited on resources, so I don't expect we'll be able to triage every single fixed bug that comes through. Each QA team does look through a subset of the fixed bugs though regularly - on Firefox OS for example, I'll usually watch the fixed blockers for a release that come in weekly.

I do think we need to clarify some of the special cases in this workflow. For example, let's say framework-needed gets set because this can be best verified through automation, but the framework isn't available. Manual testing could be done as an option on this bug as a short-term solution. How do we handle the workflow here? Should I triage and notice framework-needed is present with a comment saying "Manual testing would be helpful given that we cannot automate this in the short-term?" Should we have a workflow that moves between a one-off verification to framework-needed in this case?
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

jsmith.mozilla
One other comment:

I think there's a missing keyword in here that specifies the one-off verification status. Maybe manual-verification-needed?
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Alexander Keybl
In reply to this post by Mike Connor-4
I just wanted to chime in and say that I support Mike's plan, and I think the spirit of this is still in line with my original intent - to do a better job of verifying those bugs that are not already covered by automated testing.

I'm very interested to see how many bugs fall into the manual-test-needed/testing-needed and framework-needed buckets, to better understand the load being asked of QA and the A-Team.

-Alex

On Jul 8, 2013, at 11:56 AM, Mike Connor <[hidden email]> wrote:

> Hi all,
>
> Many of you will recall that Alex Keybl proposed a change recently that would have asked developers to manually verify all of their patches after landing.  Understandably, this was met with significant concerns regarding effective utilization of developer time, and the thread quickly became very hard to follow.  In the very long discussion, a heavily revised concept seemed to gain consensus, and it was suggested that a new thread be started to discuss that revised proposal.
>
> The proposal I'm going to link to below is intended to be straightforward, and seeks to build upon the following three core principles:
>
> * For all fixed bugs, the state of test coverage should be clear and unambiguous
> * Developers have the best insight into which bugs carry the most risk and require attention from QA
> * QA is the best group to effectively test fixes, and has the best insight into which bugs require additional testing and verification
>
> The thrust of the proposal is simple: we should replace the current assortment of keywords, flags, and whiteboard annotations with a single multi-state flag (testing-status), and make setting this flag part of the required workflow when resolving a bug in a product component as FIXED.   Developers will be asked to assess whether a bug is either: a) not testable, b) covered sufficiently by automated tests or c) requires QA to assess the bug.  QA then will determine the amount and type of testing required, and take action as appropriate.  It is intended that this formalization of the process will provide us visibility into our testing coverage and resource investment across our products.
>
> The full details, including a visual workflow diagram, can be found at https://wiki.mozilla.org/User:Mconnor/BugVerification
>
> Any question or comments are welcome here, or you can find me on IRC.
>
> -- Mike
>
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Anthony Hughes-3
In reply to this post by Mike Connor-4
Hi Mike,

Thanks for taking initiative on this. I fully support this proposal but I have a couple of follow up questions.

> in-automated-testsuite, automated-test-needed
How would this affect the present workflow and usage of the in-testsuite and in-qa-testsuite flags?

> manual-test-needed , in-manual-testsuite
How would this affect the present workflow and usage of the in-moztrap flag?

Thanks

Anthony Hughes
QA Engineer, Desktop Firefox
Mozilla Corporation


----- Original Message -----

> From: "Mike Connor" <[hidden email]>
> To: "[hidden email] group" <[hidden email]>
> Sent: Monday, July 8, 2013 11:56:57 AM
> Subject: Improving Fix Verification (v2)
>
> Hi all,
>
> Many of you will recall that Alex Keybl proposed a change recently that
> would have asked developers to manually verify all of their patches
> after landing.  Understandably, this was met with significant concerns
> regarding effective utilization of developer time, and the thread
> quickly became very hard to follow.  In the very long discussion, a
> heavily revised concept seemed to gain consensus, and it was suggested
> that a new thread be started to discuss that revised proposal.
>
> The proposal I'm going to link to below is intended to be
> straightforward, and seeks to build upon the following three core
> principles:
>
> * For all fixed bugs, the state of test coverage should be clear and
> unambiguous
> * Developers have the best insight into which bugs carry the most risk
> and require attention from QA
> * QA is the best group to effectively test fixes, and has the best
> insight into which bugs require additional testing and verification
>
> The thrust of the proposal is simple: we should replace the current
> assortment of keywords, flags, and whiteboard annotations with a single
> multi-state flag (testing-status), and make setting this flag part of
> the required workflow when resolving a bug in a product component as
> FIXED.   Developers will be asked to assess whether a bug is either: a)
> not testable, b) covered sufficiently by automated tests or c) requires
> QA to assess the bug.  QA then will determine the amount and type of
> testing required, and take action as appropriate.  It is intended that
> this formalization of the process will provide us visibility into our
> testing coverage and resource investment across our products.
>
> The full details, including a visual workflow diagram, can be found at
> https://wiki.mozilla.org/User:Mconnor/BugVerification
>
> Any question or comments are welcome here, or you can find me on IRC.
>
> -- Mike
>
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Mike Connor-4
On 2013-07-11 5:04 PM, Anthony Hughes wrote:
> Hi Mike,
>
> Thanks for taking initiative on this. I fully support this proposal but I have a couple of follow up questions.
>
>> in-automated-testsuite, automated-test-needed
> How would this affect the present workflow and usage of the in-testsuite and in-qa-testsuite flags?
>
>> manual-test-needed , in-manual-testsuite
> How would this affect the present workflow and usage of the in-moztrap flag?

My intent is that this process would replace both of those flags, as
well as verifyme/[qa-] and anything else in common use for tracking
testing.  The idea is that there should be one canonical and consistent
status for testing.

-- Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Mike Connor-4
In reply to this post by jsmith.mozilla
On 2013-07-11 4:44 PM, [hidden email] wrote:
> Comments:
>
> I would get rid of the immediate step of "testing-needed" and just have the developer directly set either framework-needed, automated-test-needed, or manual-test-needed directly. There will be cases where I think there will be disagreement in the case where say, a QA comes through analyzing the bugs and disagrees with the resolution. But in that case, I think the developer and QA can just work together to come to agreement when there's disagreement on the testing-status state.

The rationale behind "testing-needed" is to ensure that QA is aware of
all untested bugs landing in our products, and is able to take action as
they see fit.  If a developer just marks a bug as framework-needed, that
fix is effectively untested and unverified, which is an unmanaged risk.  
My goal here is to better manage the risk of fixes not covered by
automation.

> QA is limited on resources, so I don't expect we'll be able to triage every single fixed bug that comes through. Each QA team does look through a subset of the fixed bugs though regularly - on Firefox OS for example, I'll usually watch the fixed blockers for a release that come in weekly.

To be clear, QA would only be asked to look at the bugs that aren't
covered by tests, but are usefully testable.  Hopefully that's a
significantly smaller subset than all fixed bugs!  I understand
resources are tight, how else would you propose we identify the working
set where QA intervention is desirable?

> I do think we need to clarify some of the special cases in this workflow. For example, let's say framework-needed gets set because this can be best verified through automation, but the framework isn't available. Manual testing could be done as an option on this bug as a short-term solution. How do we handle the workflow here? Should I triage and notice framework-needed is present with a comment saying "Manual testing would be helpful given that we cannot automate this in the short-term?" Should we have a workflow that moves between a one-off verification to framework-needed in this case?

This type of case is why the proposal has a testing-needed state, rather
than a complex workflow or multiple flags.  The idea is that for the
subset of fixed bugs that are untested/unverified, QA will determine the
correct course of action.  I don't think the process should be
prescriptive for how QA makes those decisions, nor do I think we should
require any action other than that which QA deems necessary appropriate.

-- Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Ehsan Akhgari
In reply to this post by Mike Connor-4
Under this proposal, do developers need to set this flag on every bug
that they fix?

Cheers,
Ehsan
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Mike Connor-4
On 2013-07-11, at 6:13 PM, Ehsan Akhgari <[hidden email]> wrote:

> Under this proposal, do developers need to set this flag on every bug that they fix?

That's the expectation, yes.  I think it should be fairly easy to determine the right value for a given bug, if you've thought about how something should be tested…

-- Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Improving Fix Verification (v2)

Ehsan Akhgari
On 2013-07-11 6:16 PM, Mike Connor wrote:
> On 2013-07-11, at 6:13 PM, Ehsan Akhgari <[hidden email]> wrote:
>
>> Under this proposal, do developers need to set this flag on every bug that they fix?
>
> That's the expectation, yes.  I think it should be fairly easy to determine the right value for a given bug, if you've thought about how something should be tested…

Can we have pick a default value so that not every developer has to set
one flag per each bug they fix?  That is a huge cost, and at least most
of the bugs that I have personally worked on in the past few years will
not really benefit from this extra step at all, since they're either not
directly actionable by QA, or QA had already been looped in before the
bug is fixed.

Ehsan

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
123