Re: Test Case Management

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Axel Hecht
Hi,

sorry for chiming in late. It'd be great if our test case management
supported running tests on localized versions, and had some reporting
support for that.

CCing Delphine for the insights we gathered from our FxOS testing programs.

Axel

On 25/02/16 00:30, John Dorlus wrote:

> Greetings Mozillians,
>
> The quest for a test case management system has been a topic for a while
> now amongst several teams. Moztrap is a great tool but it's maintenance is
> slow and we have a need now to leverage someone else's code so we can focus
> on the new and shiny. There are many options out there and we want to find
> one that will support our Open Sourceness. We found a tools seems
> interesting and want to get some opinions on this or other alternatives.
>
> TestRail - TestRail is one of the most widely used Test Case Management
> systems amongst top tier tech companies. Comes with all of the bells and
> whistles (including integration with Bugzilla and Github) but is normally
> super expensive. I found a page on the TestRail site saying they give free
> licenses to open source projects.1
>
> Have any of you heard of something better? Better support? Better
> integration? Is TestRail a good choice? Let us know your thoughts.
>
> 1 - https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>
> [image: photo]
> *John Dorlus*
> Senior Software Engineer in Test, Mozilla Corporation
> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
> E. Evelyn Avenue Mountain View, CA 94041 USA
> <http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus>
> <http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> <http://www.linkedin.com/company/mozilla-corporation>
> Get a signature like this: Click here!
> <http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478>
>

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Naoki Hirata
+1

I hope that it's asystem that supports showing screenshots and comparisons
as well.  I find that it helps for l10n testing as well as other things.

Regards,
Naoki

On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:

> Hi,
>
> sorry for chiming in late. It'd be great if our test case management
> supported running tests on localized versions, and had some reporting
> support for that.
>
> CCing Delphine for the insights we gathered from our FxOS testing programs.
>
> Axel
>
> On 25/02/16 00:30, John Dorlus wrote:
>
>> Greetings Mozillians,
>>
>> The quest for a test case management system has been a topic for a while
>> now amongst several teams. Moztrap is a great tool but it's maintenance is
>> slow and we have a need now to leverage someone else's code so we can
>> focus
>> on the new and shiny. There are many options out there and we want to find
>> one that will support our Open Sourceness. We found a tools seems
>> interesting and want to get some opinions on this or other alternatives.
>>
>> TestRail - TestRail is one of the most widely used Test Case Management
>> systems amongst top tier tech companies. Comes with all of the bells and
>> whistles (including integration with Bugzilla and Github) but is normally
>> super expensive. I found a page on the TestRail site saying they give free
>> licenses to open source projects.1
>>
>> Have any of you heard of something better? Better support? Better
>> integration? Is TestRail a good choice? Let us know your thoughts.
>>
>> 1 -
>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>>
>> [image: photo]
>> *John Dorlus*
>> Senior Software Engineer in Test, Mozilla Corporation
>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
>> E. Evelyn Avenue Mountain View, CA 94041 USA
>> <http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus
>> >
>> <http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>> <http://www.linkedin.com/company/mozilla-corporation>
>> Get a signature like this: Click here!
>> <
>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
>> >
>>
>>
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Stuart Philp
Yeah fair points. We've been assessing it so far mostly for our needs,
which don't include a lot of l10n testing. But it looks to have the
flexibility in the test case meta data to assign tests to locales,
versions, releases, etc.


On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:

> +1
>
> I hope that it's asystem that supports showing screenshots and comparisons
> as well.  I find that it helps for l10n testing as well as other things.
>
> Regards,
> Naoki
>
> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]
> <javascript:;>> wrote:
>
> > Hi,
> >
> > sorry for chiming in late. It'd be great if our test case management
> > supported running tests on localized versions, and had some reporting
> > support for that.
> >
> > CCing Delphine for the insights we gathered from our FxOS testing
> programs.
> >
> > Axel
> >
> > On 25/02/16 00:30, John Dorlus wrote:
> >
> >> Greetings Mozillians,
> >>
> >> The quest for a test case management system has been a topic for a while
> >> now amongst several teams. Moztrap is a great tool but it's maintenance
> is
> >> slow and we have a need now to leverage someone else's code so we can
> >> focus
> >> on the new and shiny. There are many options out there and we want to
> find
> >> one that will support our Open Sourceness. We found a tools seems
> >> interesting and want to get some opinions on this or other alternatives.
> >>
> >> TestRail - TestRail is one of the most widely used Test Case Management
> >> systems amongst top tier tech companies. Comes with all of the bells and
> >> whistles (including integration with Bugzilla and Github) but is
> normally
> >> super expensive. I found a page on the TestRail site saying they give
> free
> >> licenses to open source projects.1
> >>
> >> Have any of you heard of something better? Better support? Better
> >> integration? Is TestRail a good choice? Let us know your thoughts.
> >>
> >> 1 -
> >>
> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
> >>
> >> [image: photo]
> >> *John Dorlus*
> >> Senior Software Engineer in Test, Mozilla Corporation
> >> m:407.780.8031 | e:[hidden email] <javascript:;> | w:
> http://www.mozilla.org | a:331
> >> E. Evelyn Avenue Mountain View, CA 94041 USA
> >> <http://www.facebook.com/mozilla>  <
> http://www.linkedin.com/in/johndorlus
> >> >
> >> <http://www.twitter.com/the_power_coder>  <
> http://www.github.com/silne30>
> >> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> >> <http://www.linkedin.com/company/mozilla-corporation>
> >> Get a signature like this: Click here!
> >> <
> >>
> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
> >> >
> >>
> >>
> > _______________________________________________
> > dev-quality mailing list
> > [hidden email] <javascript:;>
> > https://lists.mozilla.org/listinfo/dev-quality
> >
> _______________________________________________
> dev-quality mailing list
> [hidden email] <javascript:;>
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Delphine Lebédel
It's been more than a year that I myself haven't executed l10n testing on
Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
Marigold would have had good feedback for this - but I guess we can't
really ask them anymore.

What I do remember that is really important for l10n has been highlighted
by Naoki and Stuart: the need for screenshots (really necessary) and the
cited meta data. Something that would have been helpful (but not only from
l10n point of view) would have been the ability to file bugs directly
within moztrap, and not have to go back and forth between moztrap and
bugzilla.

I think I'd need to dig into the tool again to see if I can come with
something else useful. Thanks for getting the conversation started :)

delphine

On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]> wrote:

> Yeah fair points. We've been assessing it so far mostly for our needs,
> which don't include a lot of l10n testing. But it looks to have the
> flexibility in the test case meta data to assign tests to locales,
> versions, releases, etc.
>
>
>
> On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
>
>> +1
>>
>> I hope that it's asystem that supports showing screenshots and comparisons
>> as well.  I find that it helps for l10n testing as well as other things.
>>
>> Regards,
>> Naoki
>>
>> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
>>
>> > Hi,
>> >
>> > sorry for chiming in late. It'd be great if our test case management
>> > supported running tests on localized versions, and had some reporting
>> > support for that.
>> >
>> > CCing Delphine for the insights we gathered from our FxOS testing
>> programs.
>> >
>> > Axel
>> >
>> > On 25/02/16 00:30, John Dorlus wrote:
>> >
>> >> Greetings Mozillians,
>> >>
>> >> The quest for a test case management system has been a topic for a
>> while
>> >> now amongst several teams. Moztrap is a great tool but it's
>> maintenance is
>> >> slow and we have a need now to leverage someone else's code so we can
>> >> focus
>> >> on the new and shiny. There are many options out there and we want to
>> find
>> >> one that will support our Open Sourceness. We found a tools seems
>> >> interesting and want to get some opinions on this or other
>> alternatives.
>> >>
>> >> TestRail - TestRail is one of the most widely used Test Case Management
>> >> systems amongst top tier tech companies. Comes with all of the bells
>> and
>> >> whistles (including integration with Bugzilla and Github) but is
>> normally
>> >> super expensive. I found a page on the TestRail site saying they give
>> free
>> >> licenses to open source projects.1
>> >>
>> >> Have any of you heard of something better? Better support? Better
>> >> integration? Is TestRail a good choice? Let us know your thoughts.
>> >>
>> >> 1 -
>> >>
>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>> >>
>> >> [image: photo]
>> >> *John Dorlus*
>> >> Senior Software Engineer in Test, Mozilla Corporation
>> >> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org |
>> a:331
>> >> E. Evelyn Avenue Mountain View, CA 94041 USA
>> >> <http://www.facebook.com/mozilla>  <
>> http://www.linkedin.com/in/johndorlus
>> >> >
>> >> <http://www.twitter.com/the_power_coder>  <
>> http://www.github.com/silne30>
>> >> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>> >> <http://www.linkedin.com/company/mozilla-corporation>
>> >> Get a signature like this: Click here!
>> >> <
>> >>
>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
>> >> >
>> >>
>> >>
>> > _______________________________________________
>> > dev-quality mailing list
>> > [hidden email]
>> > https://lists.mozilla.org/listinfo/dev-quality
>> >
>> _______________________________________________
>> dev-quality mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-quality
>>
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Lawrence Mandel-2
Hi all,

I would like to proceed with TestRail (or another system but from what I
gather this one seems to meet our needs). Specifically, I do not think that
we should continue to develop our own test case management tool in house.
Has anyone spoken with TestRail about their open source license? If not, I
volunteer Stuart and myself to do so.

Lawrence

On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <[hidden email]
> wrote:

> Hi guys,
>
> I think it was John who mentioned that we may be able to get a free license
> for Test Rail (based on this:
> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/).
> Was wondering if anyone has contacted them to ask about this.
>
> I'm also adding Lawrence to this thread, as we've been discussing this
> internally, and our QA teams are very interested in getting this to replace
> MozTrap, gdocs.
>
> Regards,
> Florin.
>
> -----Original Message-----
> From: dev-quality
> [mailto:dev-quality-bounces+florin.mezei=
> [hidden email]]
> On Behalf Of Delphine Lebédel
> Sent: Saturday, April 02, 2016 3:50 AM
> To: Stuart Philp <[hidden email]>
> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
> [hidden email]; Naoki Hirata <[hidden email]>
> Subject: Re: Test Case Management
>
> It's been more than a year that I myself haven't executed l10n testing on
> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
> Marigold would have had good feedback for this - but I guess we can't
> really
> ask them anymore.
>
> What I do remember that is really important for l10n has been highlighted
> by
> Naoki and Stuart: the need for screenshots (really necessary) and the cited
> meta data. Something that would have been helpful (but not only from l10n
> point of view) would have been the ability to file bugs directly within
> moztrap, and not have to go back and forth between moztrap and bugzilla.
>
> I think I'd need to dig into the tool again to see if I can come with
> something else useful. Thanks for getting the conversation started :)
>
> delphine
>
> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]> wrote:
>
> > Yeah fair points. We've been assessing it so far mostly for our needs,
> > which don't include a lot of l10n testing. But it looks to have the
> > flexibility in the test case meta data to assign tests to locales,
> > versions, releases, etc.
> >
> >
> >
> > On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
> >
> >> +1
> >>
> >> I hope that it's asystem that supports showing screenshots and
> >> comparisons as well.  I find that it helps for l10n testing as well as
> other things.
> >>
> >> Regards,
> >> Naoki
> >>
> >> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
> >>
> >> > Hi,
> >> >
> >> > sorry for chiming in late. It'd be great if our test case
> >> > management supported running tests on localized versions, and had
> >> > some reporting support for that.
> >> >
> >> > CCing Delphine for the insights we gathered from our FxOS testing
> >> programs.
> >> >
> >> > Axel
> >> >
> >> > On 25/02/16 00:30, John Dorlus wrote:
> >> >
> >> >> Greetings Mozillians,
> >> >>
> >> >> The quest for a test case management system has been a topic for a
> >> while
> >> >> now amongst several teams. Moztrap is a great tool but it's
> >> maintenance is
> >> >> slow and we have a need now to leverage someone else's code so we
> >> >> can focus on the new and shiny. There are many options out there
> >> >> and we want to
> >> find
> >> >> one that will support our Open Sourceness. We found a tools seems
> >> >> interesting and want to get some opinions on this or other
> >> alternatives.
> >> >>
> >> >> TestRail - TestRail is one of the most widely used Test Case
> >> >> Management systems amongst top tier tech companies. Comes with all
> >> >> of the bells
> >> and
> >> >> whistles (including integration with Bugzilla and Github) but is
> >> normally
> >> >> super expensive. I found a page on the TestRail site saying they
> >> >> give
> >> free
> >> >> licenses to open source projects.1
> >> >>
> >> >> Have any of you heard of something better? Better support? Better
> >> >> integration? Is TestRail a good choice? Let us know your thoughts.
> >> >>
> >> >> 1 -
> >> >>
> >> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
> >> ts/
> >> >>
> >> >> [image: photo]
> >> >> *John Dorlus*
> >> >> Senior Software Engineer in Test, Mozilla Corporation
> >> >> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
> >> >> |
> >> a:331
> >> >> E. Evelyn Avenue Mountain View, CA 94041 USA
> >> >> <http://www.facebook.com/mozilla>  <
> >> http://www.linkedin.com/in/johndorlus
> >> >> >
> >> >> <http://www.twitter.com/the_power_coder>  <
> >> http://www.github.com/silne30>
> >> >> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> >> >> <http://www.linkedin.com/company/mozilla-corporation>
> >> >> Get a signature like this: Click here!
> >> >> <
> >> >>
> >> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
> >> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
> >> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
> >> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
> >> 478
> >> >> >
> >> >>
> >> >>
> >> > _______________________________________________
> >> > dev-quality mailing list
> >> > [hidden email]
> >> > https://lists.mozilla.org/listinfo/dev-quality
> >> >
> >> _______________________________________________
> >> dev-quality mailing list
> >> [hidden email]
> >> https://lists.mozilla.org/listinfo/dev-quality
> >>
> >
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
>
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Cameron Dawson
As the owner of MozTrap (who is deployed full-time on Treeherder), this sounds great!  :)  Please let me know any assistance I can provide with migrating data over from MozTrap to TestRail.

Thanks,
-Cam


> On Apr 6, 2016, at 8:38 AM, Lawrence Mandel <[hidden email]> wrote:
>
> Hi all,
>
> I would like to proceed with TestRail (or another system but from what I
> gather this one seems to meet our needs). Specifically, I do not think that
> we should continue to develop our own test case management tool in house.
> Has anyone spoken with TestRail about their open source license? If not, I
> volunteer Stuart and myself to do so.
>
> Lawrence
>
> On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <[hidden email]
>> wrote:
>
>> Hi guys,
>>
>> I think it was John who mentioned that we may be able to get a free license
>> for Test Rail (based on this:
>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/).
>> Was wondering if anyone has contacted them to ask about this.
>>
>> I'm also adding Lawrence to this thread, as we've been discussing this
>> internally, and our QA teams are very interested in getting this to replace
>> MozTrap, gdocs.
>>
>> Regards,
>> Florin.
>>
>> -----Original Message-----
>> From: dev-quality
>> [mailto:dev-quality-bounces+florin.mezei=
>> [hidden email]]
>> On Behalf Of Delphine Lebédel
>> Sent: Saturday, April 02, 2016 3:50 AM
>> To: Stuart Philp <[hidden email]>
>> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
>> [hidden email]; Naoki Hirata <[hidden email]>
>> Subject: Re: Test Case Management
>>
>> It's been more than a year that I myself haven't executed l10n testing on
>> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
>> Marigold would have had good feedback for this - but I guess we can't
>> really
>> ask them anymore.
>>
>> What I do remember that is really important for l10n has been highlighted
>> by
>> Naoki and Stuart: the need for screenshots (really necessary) and the cited
>> meta data. Something that would have been helpful (but not only from l10n
>> point of view) would have been the ability to file bugs directly within
>> moztrap, and not have to go back and forth between moztrap and bugzilla.
>>
>> I think I'd need to dig into the tool again to see if I can come with
>> something else useful. Thanks for getting the conversation started :)
>>
>> delphine
>>
>> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]> wrote:
>>
>>> Yeah fair points. We've been assessing it so far mostly for our needs,
>>> which don't include a lot of l10n testing. But it looks to have the
>>> flexibility in the test case meta data to assign tests to locales,
>>> versions, releases, etc.
>>>
>>>
>>>
>>> On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
>>>
>>>> +1
>>>>
>>>> I hope that it's asystem that supports showing screenshots and
>>>> comparisons as well.  I find that it helps for l10n testing as well as
>> other things.
>>>>
>>>> Regards,
>>>> Naoki
>>>>
>>>> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> sorry for chiming in late. It'd be great if our test case
>>>>> management supported running tests on localized versions, and had
>>>>> some reporting support for that.
>>>>>
>>>>> CCing Delphine for the insights we gathered from our FxOS testing
>>>> programs.
>>>>>
>>>>> Axel
>>>>>
>>>>> On 25/02/16 00:30, John Dorlus wrote:
>>>>>
>>>>>> Greetings Mozillians,
>>>>>>
>>>>>> The quest for a test case management system has been a topic for a
>>>> while
>>>>>> now amongst several teams. Moztrap is a great tool but it's
>>>> maintenance is
>>>>>> slow and we have a need now to leverage someone else's code so we
>>>>>> can focus on the new and shiny. There are many options out there
>>>>>> and we want to
>>>> find
>>>>>> one that will support our Open Sourceness. We found a tools seems
>>>>>> interesting and want to get some opinions on this or other
>>>> alternatives.
>>>>>>
>>>>>> TestRail - TestRail is one of the most widely used Test Case
>>>>>> Management systems amongst top tier tech companies. Comes with all
>>>>>> of the bells
>>>> and
>>>>>> whistles (including integration with Bugzilla and Github) but is
>>>> normally
>>>>>> super expensive. I found a page on the TestRail site saying they
>>>>>> give
>>>> free
>>>>>> licenses to open source projects.1
>>>>>>
>>>>>> Have any of you heard of something better? Better support? Better
>>>>>> integration? Is TestRail a good choice? Let us know your thoughts.
>>>>>>
>>>>>> 1 -
>>>>>>
>>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
>>>> ts/
>>>>>>
>>>>>> [image: photo]
>>>>>> *John Dorlus*
>>>>>> Senior Software Engineer in Test, Mozilla Corporation
>>>>>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
>>>>>> |
>>>> a:331
>>>>>> E. Evelyn Avenue Mountain View, CA 94041 USA
>>>>>> <http://www.facebook.com/mozilla>  <
>>>> http://www.linkedin.com/in/johndorlus
>>>>>>>
>>>>>> <http://www.twitter.com/the_power_coder>  <
>>>> http://www.github.com/silne30>
>>>>>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>>>>>> <http://www.linkedin.com/company/mozilla-corporation>
>>>>>> Get a signature like this: Click here!
>>>>>> <
>>>>>>
>>>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
>>>> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
>>>> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
>>>> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
>>>> 478
>>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> dev-quality mailing list
>>>>> [hidden email]
>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>
>>>> _______________________________________________
>>>> dev-quality mailing list
>>>> [hidden email]
>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>
>>>
>> _______________________________________________
>> dev-quality mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-quality
>>
>>
>>
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Stuart Philp
In reply to this post by Lawrence Mandel-2
My team has a free open-source license from gurock for 50 seats which we've
been using to evaluate testrail, it is running here:
http://testrail.stage.mozaws.net/

We have also decided to go forward with trying it more permanently as our
early feedback was mostly positive. We do however have a few questions we
want to answer, you can see our discussion/questions if you're curious here:

https://docs.google.com/document/d/1WkZMjYxEyEWHqmjKpE2idOLUCmQG6Fa5E6TQQiAL-BU/edit

Once we have answered those questions we will move forward with planning
how the transition will happen. I would note we think that if we were to
ask for a larger seat count (tbd what that number would be, 100-500 would
be a rough range for all QA, dev, and community needs), we should also look
at paying for the service as it would no longer be a small side project but
rather a large, key piece of QA operations at Mozilla.



On Wed, Apr 6, 2016 at 11:38 AM, Lawrence Mandel <[hidden email]>
wrote:

> Hi all,
>
> I would like to proceed with TestRail (or another system but from what I
> gather this one seems to meet our needs). Specifically, I do not think that
> we should continue to develop our own test case management tool in house.
> Has anyone spoken with TestRail about their open source license? If not, I
> volunteer Stuart and myself to do so.
>
> Lawrence
>
> On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <
> [hidden email]> wrote:
>
>> Hi guys,
>>
>> I think it was John who mentioned that we may be able to get a free
>> license
>> for Test Rail (based on this:
>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>> ).
>> Was wondering if anyone has contacted them to ask about this.
>>
>> I'm also adding Lawrence to this thread, as we've been discussing this
>> internally, and our QA teams are very interested in getting this to
>> replace
>> MozTrap, gdocs.
>>
>> Regards,
>> Florin.
>>
>> -----Original Message-----
>> From: dev-quality
>> [mailto:dev-quality-bounces+florin.mezei=
>> [hidden email]]
>> On Behalf Of Delphine Lebédel
>> Sent: Saturday, April 02, 2016 3:50 AM
>> To: Stuart Philp <[hidden email]>
>> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
>> [hidden email]; Naoki Hirata <[hidden email]>
>> Subject: Re: Test Case Management
>>
>> It's been more than a year that I myself haven't executed l10n testing on
>> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
>> Marigold would have had good feedback for this - but I guess we can't
>> really
>> ask them anymore.
>>
>> What I do remember that is really important for l10n has been highlighted
>> by
>> Naoki and Stuart: the need for screenshots (really necessary) and the
>> cited
>> meta data. Something that would have been helpful (but not only from l10n
>> point of view) would have been the ability to file bugs directly within
>> moztrap, and not have to go back and forth between moztrap and bugzilla.
>>
>> I think I'd need to dig into the tool again to see if I can come with
>> something else useful. Thanks for getting the conversation started :)
>>
>> delphine
>>
>> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]> wrote:
>>
>> > Yeah fair points. We've been assessing it so far mostly for our needs,
>> > which don't include a lot of l10n testing. But it looks to have the
>> > flexibility in the test case meta data to assign tests to locales,
>> > versions, releases, etc.
>> >
>> >
>> >
>> > On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
>> >
>> >> +1
>> >>
>> >> I hope that it's asystem that supports showing screenshots and
>> >> comparisons as well.  I find that it helps for l10n testing as well as
>> other things.
>> >>
>> >> Regards,
>> >> Naoki
>> >>
>> >> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > sorry for chiming in late. It'd be great if our test case
>> >> > management supported running tests on localized versions, and had
>> >> > some reporting support for that.
>> >> >
>> >> > CCing Delphine for the insights we gathered from our FxOS testing
>> >> programs.
>> >> >
>> >> > Axel
>> >> >
>> >> > On 25/02/16 00:30, John Dorlus wrote:
>> >> >
>> >> >> Greetings Mozillians,
>> >> >>
>> >> >> The quest for a test case management system has been a topic for a
>> >> while
>> >> >> now amongst several teams. Moztrap is a great tool but it's
>> >> maintenance is
>> >> >> slow and we have a need now to leverage someone else's code so we
>> >> >> can focus on the new and shiny. There are many options out there
>> >> >> and we want to
>> >> find
>> >> >> one that will support our Open Sourceness. We found a tools seems
>> >> >> interesting and want to get some opinions on this or other
>> >> alternatives.
>> >> >>
>> >> >> TestRail - TestRail is one of the most widely used Test Case
>> >> >> Management systems amongst top tier tech companies. Comes with all
>> >> >> of the bells
>> >> and
>> >> >> whistles (including integration with Bugzilla and Github) but is
>> >> normally
>> >> >> super expensive. I found a page on the TestRail site saying they
>> >> >> give
>> >> free
>> >> >> licenses to open source projects.1
>> >> >>
>> >> >> Have any of you heard of something better? Better support? Better
>> >> >> integration? Is TestRail a good choice? Let us know your thoughts.
>> >> >>
>> >> >> 1 -
>> >> >>
>> >> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
>> >> ts/
>> >> >>
>> >> >> [image: photo]
>> >> >> *John Dorlus*
>> >> >> Senior Software Engineer in Test, Mozilla Corporation
>> >> >> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
>> >> >> |
>> >> a:331
>> >> >> E. Evelyn Avenue Mountain View, CA 94041 USA
>> >> >> <http://www.facebook.com/mozilla>  <
>> >> http://www.linkedin.com/in/johndorlus
>> >> >> >
>> >> >> <http://www.twitter.com/the_power_coder>  <
>> >> http://www.github.com/silne30>
>> >> >> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>> >> >> <http://www.linkedin.com/company/mozilla-corporation>
>> >> >> Get a signature like this: Click here!
>> >> >> <
>> >> >>
>> >> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
>> >> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
>> >> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
>> >> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
>> >> 478
>> >> >> >
>> >> >>
>> >> >>
>> >> > _______________________________________________
>> >> > dev-quality mailing list
>> >> > [hidden email]
>> >> > https://lists.mozilla.org/listinfo/dev-quality
>> >> >
>> >> _______________________________________________
>> >> dev-quality mailing list
>> >> [hidden email]
>> >> https://lists.mozilla.org/listinfo/dev-quality
>> >>
>> >
>> _______________________________________________
>> dev-quality mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-quality
>>
>>
>>
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Stuart Philp
In reply to this post by Cameron Dawson
Oh nice! One of the questions we had is what happens to test rail when it
reaches the scale that Moztrap has been running at, i.e. many thousands of
test cases. If we had a way to export/import some cases from Moztrap we
could quickly answer that, it likely requires some data manipulation but
would still save a bunch of time. Cameron, perhaps you can coordinate that
with Stephen Donner?

On Wed, Apr 6, 2016 at 11:50 AM, Cameron Dawson <[hidden email]> wrote:

> As the owner of MozTrap (who is deployed full-time on Treeherder), this
> sounds great!  :)  Please let me know any assistance I can provide with
> migrating data over from MozTrap to TestRail.
>
> Thanks,
> -Cam
>
>
> > On Apr 6, 2016, at 8:38 AM, Lawrence Mandel <[hidden email]> wrote:
> >
> > Hi all,
> >
> > I would like to proceed with TestRail (or another system but from what I
> > gather this one seems to meet our needs). Specifically, I do not think
> that
> > we should continue to develop our own test case management tool in house.
> > Has anyone spoken with TestRail about their open source license? If not,
> I
> > volunteer Stuart and myself to do so.
> >
> > Lawrence
> >
> > On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <
> [hidden email]
> >> wrote:
> >
> >> Hi guys,
> >>
> >> I think it was John who mentioned that we may be able to get a free
> license
> >> for Test Rail (based on this:
> >>
> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/).
> >> Was wondering if anyone has contacted them to ask about this.
> >>
> >> I'm also adding Lawrence to this thread, as we've been discussing this
> >> internally, and our QA teams are very interested in getting this to
> replace
> >> MozTrap, gdocs.
> >>
> >> Regards,
> >> Florin.
> >>
> >> -----Original Message-----
> >> From: dev-quality
> >> [mailto:dev-quality-bounces+florin.mezei=
> >> [hidden email]]
> >> On Behalf Of Delphine Lebédel
> >> Sent: Saturday, April 02, 2016 3:50 AM
> >> To: Stuart Philp <[hidden email]>
> >> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
> >> [hidden email]; Naoki Hirata <[hidden email]>
> >> Subject: Re: Test Case Management
> >>
> >> It's been more than a year that I myself haven't executed l10n testing
> on
> >> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
> >> Marigold would have had good feedback for this - but I guess we can't
> >> really
> >> ask them anymore.
> >>
> >> What I do remember that is really important for l10n has been
> highlighted
> >> by
> >> Naoki and Stuart: the need for screenshots (really necessary) and the
> cited
> >> meta data. Something that would have been helpful (but not only from
> l10n
> >> point of view) would have been the ability to file bugs directly within
> >> moztrap, and not have to go back and forth between moztrap and bugzilla.
> >>
> >> I think I'd need to dig into the tool again to see if I can come with
> >> something else useful. Thanks for getting the conversation started :)
> >>
> >> delphine
> >>
> >> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]>
> wrote:
> >>
> >>> Yeah fair points. We've been assessing it so far mostly for our needs,
> >>> which don't include a lot of l10n testing. But it looks to have the
> >>> flexibility in the test case meta data to assign tests to locales,
> >>> versions, releases, etc.
> >>>
> >>>
> >>>
> >>> On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
> >>>
> >>>> +1
> >>>>
> >>>> I hope that it's asystem that supports showing screenshots and
> >>>> comparisons as well.  I find that it helps for l10n testing as well as
> >> other things.
> >>>>
> >>>> Regards,
> >>>> Naoki
> >>>>
> >>>> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> sorry for chiming in late. It'd be great if our test case
> >>>>> management supported running tests on localized versions, and had
> >>>>> some reporting support for that.
> >>>>>
> >>>>> CCing Delphine for the insights we gathered from our FxOS testing
> >>>> programs.
> >>>>>
> >>>>> Axel
> >>>>>
> >>>>> On 25/02/16 00:30, John Dorlus wrote:
> >>>>>
> >>>>>> Greetings Mozillians,
> >>>>>>
> >>>>>> The quest for a test case management system has been a topic for a
> >>>> while
> >>>>>> now amongst several teams. Moztrap is a great tool but it's
> >>>> maintenance is
> >>>>>> slow and we have a need now to leverage someone else's code so we
> >>>>>> can focus on the new and shiny. There are many options out there
> >>>>>> and we want to
> >>>> find
> >>>>>> one that will support our Open Sourceness. We found a tools seems
> >>>>>> interesting and want to get some opinions on this or other
> >>>> alternatives.
> >>>>>>
> >>>>>> TestRail - TestRail is one of the most widely used Test Case
> >>>>>> Management systems amongst top tier tech companies. Comes with all
> >>>>>> of the bells
> >>>> and
> >>>>>> whistles (including integration with Bugzilla and Github) but is
> >>>> normally
> >>>>>> super expensive. I found a page on the TestRail site saying they
> >>>>>> give
> >>>> free
> >>>>>> licenses to open source projects.1
> >>>>>>
> >>>>>> Have any of you heard of something better? Better support? Better
> >>>>>> integration? Is TestRail a good choice? Let us know your thoughts.
> >>>>>>
> >>>>>> 1 -
> >>>>>>
> >>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
> >>>> ts/
> >>>>>>
> >>>>>> [image: photo]
> >>>>>> *John Dorlus*
> >>>>>> Senior Software Engineer in Test, Mozilla Corporation
> >>>>>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
> >>>>>> |
> >>>> a:331
> >>>>>> E. Evelyn Avenue Mountain View, CA 94041 USA
> >>>>>> <http://www.facebook.com/mozilla>  <
> >>>> http://www.linkedin.com/in/johndorlus
> >>>>>>>
> >>>>>> <http://www.twitter.com/the_power_coder>  <
> >>>> http://www.github.com/silne30>
> >>>>>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> >>>>>> <http://www.linkedin.com/company/mozilla-corporation>
> >>>>>> Get a signature like this: Click here!
> >>>>>> <
> >>>>>>
> >>>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
> >>>> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
> >>>> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
> >>>> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
> >>>> 478
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>> _______________________________________________
> >>>>> dev-quality mailing list
> >>>>> [hidden email]
> >>>>> https://lists.mozilla.org/listinfo/dev-quality
> >>>>>
> >>>> _______________________________________________
> >>>> dev-quality mailing list
> >>>> [hidden email]
> >>>> https://lists.mozilla.org/listinfo/dev-quality
> >>>>
> >>>
> >> _______________________________________________
> >> dev-quality mailing list
> >> [hidden email]
> >> https://lists.mozilla.org/listinfo/dev-quality
> >>
> >>
> >>
> > _______________________________________________
> > dev-quality mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-quality
>
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Naoki Hirata
There's ways to export using various sql queries.  I recall that Matt
Brandt had an etherpad at one point; I lost the link to it.

BTW, I guarantee that you will run into the issue of having thousands of
test cases.  The reason why we switch to moztrap was because litmus was
having that and such; instead of cleaning up the test cases, we switch
systems.  Moztrap only has a fraction of those test cases.

The maintenence of the test cases is going to continue to be a problem if
you don't come up with a maintenance strategy and a test case strategy of
what type of tests are to be placed in the test cases and allocate the time
needed for those test cases to be managed.

I highly recommend one of two things :
1) Place all test cases in the test suite -> mark all the automated cases
as automated and seperate all the manual tests; have the results of the
automated in the test cases.
2) Remove all automated cases (as you go along automating test cases) and
keep all the nonautomated test cases in.

Maintenance of test cases is required and the test case management shouldl
have good UX to support this.  This include screen captures and such for
l10n purposes.  You cannot blame the test case management system for
unmanaged test cases like litmus was scapegoated to be.



On Wed, Apr 6, 2016 at 8:57 AM, Stuart Philp <[hidden email]> wrote:

> Oh nice! One of the questions we had is what happens to test rail when it
> reaches the scale that Moztrap has been running at, i.e. many thousands of
> test cases. If we had a way to export/import some cases from Moztrap we
> could quickly answer that, it likely requires some data manipulation but
> would still save a bunch of time. Cameron, perhaps you can coordinate that
> with Stephen Donner?
>
> On Wed, Apr 6, 2016 at 11:50 AM, Cameron Dawson <[hidden email]>
> wrote:
>
>> As the owner of MozTrap (who is deployed full-time on Treeherder), this
>> sounds great!  :)  Please let me know any assistance I can provide with
>> migrating data over from MozTrap to TestRail.
>>
>> Thanks,
>> -Cam
>>
>>
>> > On Apr 6, 2016, at 8:38 AM, Lawrence Mandel <[hidden email]>
>> wrote:
>> >
>> > Hi all,
>> >
>> > I would like to proceed with TestRail (or another system but from what I
>> > gather this one seems to meet our needs). Specifically, I do not think
>> that
>> > we should continue to develop our own test case management tool in
>> house.
>> > Has anyone spoken with TestRail about their open source license? If
>> not, I
>> > volunteer Stuart and myself to do so.
>> >
>> > Lawrence
>> >
>> > On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <
>> [hidden email]
>> >> wrote:
>> >
>> >> Hi guys,
>> >>
>> >> I think it was John who mentioned that we may be able to get a free
>> license
>> >> for Test Rail (based on this:
>> >>
>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>> ).
>> >> Was wondering if anyone has contacted them to ask about this.
>> >>
>> >> I'm also adding Lawrence to this thread, as we've been discussing this
>> >> internally, and our QA teams are very interested in getting this to
>> replace
>> >> MozTrap, gdocs.
>> >>
>> >> Regards,
>> >> Florin.
>> >>
>> >> -----Original Message-----
>> >> From: dev-quality
>> >> [mailto:dev-quality-bounces+florin.mezei=
>> >> [hidden email]]
>> >> On Behalf Of Delphine Lebédel
>> >> Sent: Saturday, April 02, 2016 3:50 AM
>> >> To: Stuart Philp <[hidden email]>
>> >> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
>> >> [hidden email]; Naoki Hirata <[hidden email]>
>> >> Subject: Re: Test Case Management
>> >>
>> >> It's been more than a year that I myself haven't executed l10n testing
>> on
>> >> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
>> >> Marigold would have had good feedback for this - but I guess we can't
>> >> really
>> >> ask them anymore.
>> >>
>> >> What I do remember that is really important for l10n has been
>> highlighted
>> >> by
>> >> Naoki and Stuart: the need for screenshots (really necessary) and the
>> cited
>> >> meta data. Something that would have been helpful (but not only from
>> l10n
>> >> point of view) would have been the ability to file bugs directly within
>> >> moztrap, and not have to go back and forth between moztrap and
>> bugzilla.
>> >>
>> >> I think I'd need to dig into the tool again to see if I can come with
>> >> something else useful. Thanks for getting the conversation started :)
>> >>
>> >> delphine
>> >>
>> >> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]>
>> wrote:
>> >>
>> >>> Yeah fair points. We've been assessing it so far mostly for our needs,
>> >>> which don't include a lot of l10n testing. But it looks to have the
>> >>> flexibility in the test case meta data to assign tests to locales,
>> >>> versions, releases, etc.
>> >>>
>> >>>
>> >>>
>> >>> On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
>> >>>
>> >>>> +1
>> >>>>
>> >>>> I hope that it's asystem that supports showing screenshots and
>> >>>> comparisons as well.  I find that it helps for l10n testing as well
>> as
>> >> other things.
>> >>>>
>> >>>> Regards,
>> >>>> Naoki
>> >>>>
>> >>>> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> sorry for chiming in late. It'd be great if our test case
>> >>>>> management supported running tests on localized versions, and had
>> >>>>> some reporting support for that.
>> >>>>>
>> >>>>> CCing Delphine for the insights we gathered from our FxOS testing
>> >>>> programs.
>> >>>>>
>> >>>>> Axel
>> >>>>>
>> >>>>> On 25/02/16 00:30, John Dorlus wrote:
>> >>>>>
>> >>>>>> Greetings Mozillians,
>> >>>>>>
>> >>>>>> The quest for a test case management system has been a topic for a
>> >>>> while
>> >>>>>> now amongst several teams. Moztrap is a great tool but it's
>> >>>> maintenance is
>> >>>>>> slow and we have a need now to leverage someone else's code so we
>> >>>>>> can focus on the new and shiny. There are many options out there
>> >>>>>> and we want to
>> >>>> find
>> >>>>>> one that will support our Open Sourceness. We found a tools seems
>> >>>>>> interesting and want to get some opinions on this or other
>> >>>> alternatives.
>> >>>>>>
>> >>>>>> TestRail - TestRail is one of the most widely used Test Case
>> >>>>>> Management systems amongst top tier tech companies. Comes with all
>> >>>>>> of the bells
>> >>>> and
>> >>>>>> whistles (including integration with Bugzilla and Github) but is
>> >>>> normally
>> >>>>>> super expensive. I found a page on the TestRail site saying they
>> >>>>>> give
>> >>>> free
>> >>>>>> licenses to open source projects.1
>> >>>>>>
>> >>>>>> Have any of you heard of something better? Better support? Better
>> >>>>>> integration? Is TestRail a good choice? Let us know your thoughts.
>> >>>>>>
>> >>>>>> 1 -
>> >>>>>>
>> >>>>
>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
>> >>>> ts/
>> >>>>>>
>> >>>>>> [image: photo]
>> >>>>>> *John Dorlus*
>> >>>>>> Senior Software Engineer in Test, Mozilla Corporation
>> >>>>>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
>> >>>>>> |
>> >>>> a:331
>> >>>>>> E. Evelyn Avenue Mountain View, CA 94041 USA
>> >>>>>> <http://www.facebook.com/mozilla>  <
>> >>>> http://www.linkedin.com/in/johndorlus
>> >>>>>>>
>> >>>>>> <http://www.twitter.com/the_power_coder>  <
>> >>>> http://www.github.com/silne30>
>> >>>>>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>> >>>>>> <http://www.linkedin.com/company/mozilla-corporation>
>> >>>>>> Get a signature like this: Click here!
>> >>>>>> <
>> >>>>>>
>> >>>>
>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
>> >>>> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
>> >>>> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
>> >>>> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
>> >>>> 478
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>> _______________________________________________
>> >>>>> dev-quality mailing list
>> >>>>> [hidden email]
>> >>>>> https://lists.mozilla.org/listinfo/dev-quality
>> >>>>>
>> >>>> _______________________________________________
>> >>>> dev-quality mailing list
>> >>>> [hidden email]
>> >>>> https://lists.mozilla.org/listinfo/dev-quality
>> >>>>
>> >>>
>> >> _______________________________________________
>> >> dev-quality mailing list
>> >> [hidden email]
>> >> https://lists.mozilla.org/listinfo/dev-quality
>> >>
>> >>
>> >>
>> > _______________________________________________
>> > dev-quality mailing list
>> > [hidden email]
>> > https://lists.mozilla.org/listinfo/dev-quality
>>
>>
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Naoki Hirata
Found an old email referencing things from an old email I sent out :

"There is a script that was put together to export moztrap tests, it hasn't
been updated : This is Geo's old script:
http://people.mozilla.com/~gmealer/moz2jira/

As far as connecting to moztrap here are some queries :
https://etherpad.mozilla.org/b2g-queries

And the following should give you information.  If you don't have access to
the server you will have to ask for access via bugzilla/IT."


On Wed, Apr 6, 2016 at 9:20 AM, Naoki Hirata <[hidden email]> wrote:

> There's ways to export using various sql queries.  I recall that Matt
> Brandt had an etherpad at one point; I lost the link to it.
>
> BTW, I guarantee that you will run into the issue of having thousands of
> test cases.  The reason why we switch to moztrap was because litmus was
> having that and such; instead of cleaning up the test cases, we switch
> systems.  Moztrap only has a fraction of those test cases.
>
> The maintenence of the test cases is going to continue to be a problem if
> you don't come up with a maintenance strategy and a test case strategy of
> what type of tests are to be placed in the test cases and allocate the time
> needed for those test cases to be managed.
>
> I highly recommend one of two things :
> 1) Place all test cases in the test suite -> mark all the automated cases
> as automated and seperate all the manual tests; have the results of the
> automated in the test cases.
> 2) Remove all automated cases (as you go along automating test cases) and
> keep all the nonautomated test cases in.
>
> Maintenance of test cases is required and the test case management shouldl
> have good UX to support this.  This include screen captures and such for
> l10n purposes.  You cannot blame the test case management system for
> unmanaged test cases like litmus was scapegoated to be.
>
>
>
> On Wed, Apr 6, 2016 at 8:57 AM, Stuart Philp <[hidden email]> wrote:
>
>> Oh nice! One of the questions we had is what happens to test rail when it
>> reaches the scale that Moztrap has been running at, i.e. many thousands of
>> test cases. If we had a way to export/import some cases from Moztrap we
>> could quickly answer that, it likely requires some data manipulation but
>> would still save a bunch of time. Cameron, perhaps you can coordinate that
>> with Stephen Donner?
>>
>> On Wed, Apr 6, 2016 at 11:50 AM, Cameron Dawson <[hidden email]>
>> wrote:
>>
>>> As the owner of MozTrap (who is deployed full-time on Treeherder), this
>>> sounds great!  :)  Please let me know any assistance I can provide with
>>> migrating data over from MozTrap to TestRail.
>>>
>>> Thanks,
>>> -Cam
>>>
>>>
>>> > On Apr 6, 2016, at 8:38 AM, Lawrence Mandel <[hidden email]>
>>> wrote:
>>> >
>>> > Hi all,
>>> >
>>> > I would like to proceed with TestRail (or another system but from what
>>> I
>>> > gather this one seems to meet our needs). Specifically, I do not think
>>> that
>>> > we should continue to develop our own test case management tool in
>>> house.
>>> > Has anyone spoken with TestRail about their open source license? If
>>> not, I
>>> > volunteer Stuart and myself to do so.
>>> >
>>> > Lawrence
>>> >
>>> > On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <
>>> [hidden email]
>>> >> wrote:
>>> >
>>> >> Hi guys,
>>> >>
>>> >> I think it was John who mentioned that we may be able to get a free
>>> license
>>> >> for Test Rail (based on this:
>>> >>
>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>>> ).
>>> >> Was wondering if anyone has contacted them to ask about this.
>>> >>
>>> >> I'm also adding Lawrence to this thread, as we've been discussing this
>>> >> internally, and our QA teams are very interested in getting this to
>>> replace
>>> >> MozTrap, gdocs.
>>> >>
>>> >> Regards,
>>> >> Florin.
>>> >>
>>> >> -----Original Message-----
>>> >> From: dev-quality
>>> >> [mailto:dev-quality-bounces+florin.mezei=
>>> >> [hidden email]]
>>> >> On Behalf Of Delphine Lebédel
>>> >> Sent: Saturday, April 02, 2016 3:50 AM
>>> >> To: Stuart Philp <[hidden email]>
>>> >> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
>>> >> [hidden email]; Naoki Hirata <[hidden email]>
>>> >> Subject: Re: Test Case Management
>>> >>
>>> >> It's been more than a year that I myself haven't executed l10n
>>> testing on
>>> >> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
>>> >> Marigold would have had good feedback for this - but I guess we can't
>>> >> really
>>> >> ask them anymore.
>>> >>
>>> >> What I do remember that is really important for l10n has been
>>> highlighted
>>> >> by
>>> >> Naoki and Stuart: the need for screenshots (really necessary) and the
>>> cited
>>> >> meta data. Something that would have been helpful (but not only from
>>> l10n
>>> >> point of view) would have been the ability to file bugs directly
>>> within
>>> >> moztrap, and not have to go back and forth between moztrap and
>>> bugzilla.
>>> >>
>>> >> I think I'd need to dig into the tool again to see if I can come with
>>> >> something else useful. Thanks for getting the conversation started :)
>>> >>
>>> >> delphine
>>> >>
>>> >> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]>
>>> wrote:
>>> >>
>>> >>> Yeah fair points. We've been assessing it so far mostly for our
>>> needs,
>>> >>> which don't include a lot of l10n testing. But it looks to have the
>>> >>> flexibility in the test case meta data to assign tests to locales,
>>> >>> versions, releases, etc.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
>>> >>>
>>> >>>> +1
>>> >>>>
>>> >>>> I hope that it's asystem that supports showing screenshots and
>>> >>>> comparisons as well.  I find that it helps for l10n testing as well
>>> as
>>> >> other things.
>>> >>>>
>>> >>>> Regards,
>>> >>>> Naoki
>>> >>>>
>>> >>>> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]>
>>> wrote:
>>> >>>>
>>> >>>>> Hi,
>>> >>>>>
>>> >>>>> sorry for chiming in late. It'd be great if our test case
>>> >>>>> management supported running tests on localized versions, and had
>>> >>>>> some reporting support for that.
>>> >>>>>
>>> >>>>> CCing Delphine for the insights we gathered from our FxOS testing
>>> >>>> programs.
>>> >>>>>
>>> >>>>> Axel
>>> >>>>>
>>> >>>>> On 25/02/16 00:30, John Dorlus wrote:
>>> >>>>>
>>> >>>>>> Greetings Mozillians,
>>> >>>>>>
>>> >>>>>> The quest for a test case management system has been a topic for a
>>> >>>> while
>>> >>>>>> now amongst several teams. Moztrap is a great tool but it's
>>> >>>> maintenance is
>>> >>>>>> slow and we have a need now to leverage someone else's code so we
>>> >>>>>> can focus on the new and shiny. There are many options out there
>>> >>>>>> and we want to
>>> >>>> find
>>> >>>>>> one that will support our Open Sourceness. We found a tools seems
>>> >>>>>> interesting and want to get some opinions on this or other
>>> >>>> alternatives.
>>> >>>>>>
>>> >>>>>> TestRail - TestRail is one of the most widely used Test Case
>>> >>>>>> Management systems amongst top tier tech companies. Comes with all
>>> >>>>>> of the bells
>>> >>>> and
>>> >>>>>> whistles (including integration with Bugzilla and Github) but is
>>> >>>> normally
>>> >>>>>> super expensive. I found a page on the TestRail site saying they
>>> >>>>>> give
>>> >>>> free
>>> >>>>>> licenses to open source projects.1
>>> >>>>>>
>>> >>>>>> Have any of you heard of something better? Better support? Better
>>> >>>>>> integration? Is TestRail a good choice? Let us know your thoughts.
>>> >>>>>>
>>> >>>>>> 1 -
>>> >>>>>>
>>> >>>>
>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
>>> >>>> ts/
>>> >>>>>>
>>> >>>>>> [image: photo]
>>> >>>>>> *John Dorlus*
>>> >>>>>> Senior Software Engineer in Test, Mozilla Corporation
>>> >>>>>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
>>> >>>>>> |
>>> >>>> a:331
>>> >>>>>> E. Evelyn Avenue Mountain View, CA 94041 USA
>>> >>>>>> <http://www.facebook.com/mozilla>  <
>>> >>>> http://www.linkedin.com/in/johndorlus
>>> >>>>>>>
>>> >>>>>> <http://www.twitter.com/the_power_coder>  <
>>> >>>> http://www.github.com/silne30>
>>> >>>>>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>>> >>>>>> <http://www.linkedin.com/company/mozilla-corporation>
>>> >>>>>> Get a signature like this: Click here!
>>> >>>>>> <
>>> >>>>>>
>>> >>>>
>>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
>>> >>>>
>>> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
>>> >>>>
>>> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
>>> >>>>
>>> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
>>> >>>> 478
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>> _______________________________________________
>>> >>>>> dev-quality mailing list
>>> >>>>> [hidden email]
>>> >>>>> https://lists.mozilla.org/listinfo/dev-quality
>>> >>>>>
>>> >>>> _______________________________________________
>>> >>>> dev-quality mailing list
>>> >>>> [hidden email]
>>> >>>> https://lists.mozilla.org/listinfo/dev-quality
>>> >>>>
>>> >>>
>>> >> _______________________________________________
>>> >> dev-quality mailing list
>>> >> [hidden email]
>>> >> https://lists.mozilla.org/listinfo/dev-quality
>>> >>
>>> >>
>>> >>
>>> > _______________________________________________
>>> > dev-quality mailing list
>>> > [hidden email]
>>> > https://lists.mozilla.org/listinfo/dev-quality
>>>
>>>
>>
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Axel Hecht
In reply to this post by Lawrence Mandel-2
If there's need/opportunity: the Gurock brothers are in Berlin, and we
have folks in Berlin, so maybe we'll be able to get a in-person
conversation.

Re Stuart's

> Yeah fair points. We've been assessing it so far mostly for our needs,
> which don't include a lot of l10n testing. But it looks to have the
> flexibility in the test case meta data to assign tests to locales,
> versions, releases, etc.
I'm concerned about that from an l10n POV. I'd like to see practical
experience.

With moztrap, we were in a similar boat, and I think we paid quite some
price. Features like testing speed projections etc really need to be
designed to support testing products that are localized.

Side-stepping, on the l10n side, we're quite happy about categorizing
test results, and they're in a hierarchy of results. In l10n, a plain
"yes/no" isn't good enough.
http://www.qt21.eu/mqm-definition/issues-list-2015-12-30.html#issues_list 
is looking promising, in categorizing test failures. Those issue types
at a corse level allow to understand our quality, and at a deeper level
instruct us on how to improve and coach.

I'd personally really like to hear some war stories from the Gurocks or
their clients in the field of l10n.

Axel

On 06/04/16 17:54, Stuart Philp wrote:

> My team has a free open-source license from gurock for 50 seats which we've
> been using to evaluate testrail, it is running here:
> http://testrail.stage.mozaws.net/
>
> We have also decided to go forward with trying it more permanently as our
> early feedback was mostly positive. We do however have a few questions we
> want to answer, you can see our discussion/questions if you're curious here:
>
> https://docs.google.com/document/d/1WkZMjYxEyEWHqmjKpE2idOLUCmQG6Fa5E6TQQiAL-BU/edit
>
> Once we have answered those questions we will move forward with planning
> how the transition will happen. I would note we think that if we were to
> ask for a larger seat count (tbd what that number would be, 100-500 would
> be a rough range for all QA, dev, and community needs), we should also look
> at paying for the service as it would no longer be a small side project but
> rather a large, key piece of QA operations at Mozilla.
>
>
>
> On Wed, Apr 6, 2016 at 11:38 AM, Lawrence Mandel <[hidden email]>
> wrote:
>
>> Hi all,
>>
>> I would like to proceed with TestRail (or another system but from what I
>> gather this one seems to meet our needs). Specifically, I do not think that
>> we should continue to develop our own test case management tool in house.
>> Has anyone spoken with TestRail about their open source license? If not, I
>> volunteer Stuart and myself to do so.
>>
>> Lawrence
>>
>> On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <
>> [hidden email]> wrote:
>>
>>> Hi guys,
>>>
>>> I think it was John who mentioned that we may be able to get a free
>>> license
>>> for Test Rail (based on this:
>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>>> ).
>>> Was wondering if anyone has contacted them to ask about this.
>>>
>>> I'm also adding Lawrence to this thread, as we've been discussing this
>>> internally, and our QA teams are very interested in getting this to
>>> replace
>>> MozTrap, gdocs.
>>>
>>> Regards,
>>> Florin.
>>>
>>> -----Original Message-----
>>> From: dev-quality
>>> [mailto:dev-quality-bounces+florin.mezei=
>>> [hidden email]]
>>> On Behalf Of Delphine Lebédel
>>> Sent: Saturday, April 02, 2016 3:50 AM
>>> To: Stuart Philp <[hidden email]>
>>> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
>>> [hidden email]; Naoki Hirata <[hidden email]>
>>> Subject: Re: Test Case Management
>>>
>>> It's been more than a year that I myself haven't executed l10n testing on
>>> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
>>> Marigold would have had good feedback for this - but I guess we can't
>>> really
>>> ask them anymore.
>>>
>>> What I do remember that is really important for l10n has been highlighted
>>> by
>>> Naoki and Stuart: the need for screenshots (really necessary) and the
>>> cited
>>> meta data. Something that would have been helpful (but not only from l10n
>>> point of view) would have been the ability to file bugs directly within
>>> moztrap, and not have to go back and forth between moztrap and bugzilla.
>>>
>>> I think I'd need to dig into the tool again to see if I can come with
>>> something else useful. Thanks for getting the conversation started :)
>>>
>>> delphine
>>>
>>> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]> wrote:
>>>
>>>> Yeah fair points. We've been assessing it so far mostly for our needs,
>>>> which don't include a lot of l10n testing. But it looks to have the
>>>> flexibility in the test case meta data to assign tests to locales,
>>>> versions, releases, etc.
>>>>
>>>>
>>>>
>>>> On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> I hope that it's asystem that supports showing screenshots and
>>>>> comparisons as well.  I find that it helps for l10n testing as well as
>>> other things.
>>>>> Regards,
>>>>> Naoki
>>>>>
>>>>> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> sorry for chiming in late. It'd be great if our test case
>>>>>> management supported running tests on localized versions, and had
>>>>>> some reporting support for that.
>>>>>>
>>>>>> CCing Delphine for the insights we gathered from our FxOS testing
>>>>> programs.
>>>>>> Axel
>>>>>>
>>>>>> On 25/02/16 00:30, John Dorlus wrote:
>>>>>>
>>>>>>> Greetings Mozillians,
>>>>>>>
>>>>>>> The quest for a test case management system has been a topic for a
>>>>> while
>>>>>>> now amongst several teams. Moztrap is a great tool but it's
>>>>> maintenance is
>>>>>>> slow and we have a need now to leverage someone else's code so we
>>>>>>> can focus on the new and shiny. There are many options out there
>>>>>>> and we want to
>>>>> find
>>>>>>> one that will support our Open Sourceness. We found a tools seems
>>>>>>> interesting and want to get some opinions on this or other
>>>>> alternatives.
>>>>>>> TestRail - TestRail is one of the most widely used Test Case
>>>>>>> Management systems amongst top tier tech companies. Comes with all
>>>>>>> of the bells
>>>>> and
>>>>>>> whistles (including integration with Bugzilla and Github) but is
>>>>> normally
>>>>>>> super expensive. I found a page on the TestRail site saying they
>>>>>>> give
>>>>> free
>>>>>>> licenses to open source projects.1
>>>>>>>
>>>>>>> Have any of you heard of something better? Better support? Better
>>>>>>> integration? Is TestRail a good choice? Let us know your thoughts.
>>>>>>>
>>>>>>> 1 -
>>>>>>>
>>>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
>>>>> ts/
>>>>>>> [image: photo]
>>>>>>> *John Dorlus*
>>>>>>> Senior Software Engineer in Test, Mozilla Corporation
>>>>>>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
>>>>>>> |
>>>>> a:331
>>>>>>> E. Evelyn Avenue Mountain View, CA 94041 USA
>>>>>>> <http://www.facebook.com/mozilla>  <
>>>>> http://www.linkedin.com/in/johndorlus
>>>>>>> <http://www.twitter.com/the_power_coder>  <
>>>>> http://www.github.com/silne30>
>>>>>>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>>>>>>> <http://www.linkedin.com/company/mozilla-corporation>
>>>>>>> Get a signature like this: Click here!
>>>>>>> <
>>>>>>>
>>>>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
>>>>> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
>>>>> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
>>>>> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
>>>>> 478
>>>>>>>
>>>>>> _______________________________________________
>>>>>> dev-quality mailing list
>>>>>> [hidden email]
>>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>>
>>>>> _______________________________________________
>>>>> dev-quality mailing list
>>>>> [hidden email]
>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>
>>> _______________________________________________
>>> dev-quality mailing list
>>> [hidden email]
>>> https://lists.mozilla.org/listinfo/dev-quality
>>>
>>>
>>>

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

'chandrakant dhutadmal' via qa@mozilla.com
Hi,
Is there any way in which one can take a look at TestRail system?  I would appreciate if someone has the instance deployed and could add me to the system.
My Email id is - [hidden email]
Regards,Chandrakant Dhutadmal.

    On Friday, April 8, 2016 3:05 AM, Axel Hecht <[hidden email]> wrote:
 

 If there's need/opportunity: the Gurock brothers are in Berlin, and we
have folks in Berlin, so maybe we'll be able to get a in-person
conversation.

Re Stuart's

> Yeah fair points. We've been assessing it so far mostly for our needs,
> which don't include a lot of l10n testing. But it looks to have the
> flexibility in the test case meta data to assign tests to locales,
> versions, releases, etc.
I'm concerned about that from an l10n POV. I'd like to see practical
experience.

With moztrap, we were in a similar boat, and I think we paid quite some
price. Features like testing speed projections etc really need to be
designed to support testing products that are localized.

Side-stepping, on the l10n side, we're quite happy about categorizing
test results, and they're in a hierarchy of results. In l10n, a plain
"yes/no" isn't good enough.
http://www.qt21.eu/mqm-definition/issues-list-2015-12-30.html#issues_list 
is looking promising, in categorizing test failures. Those issue types
at a corse level allow to understand our quality, and at a deeper level
instruct us on how to improve and coach.

I'd personally really like to hear some war stories from the Gurocks or
their clients in the field of l10n.

Axel

On 06/04/16 17:54, Stuart Philp wrote:

> My team has a free open-source license from gurock for 50 seats which we've
> been using to evaluate testrail, it is running here:
> http://testrail.stage.mozaws.net/
>
> We have also decided to go forward with trying it more permanently as our
> early feedback was mostly positive. We do however have a few questions we
> want to answer, you can see our discussion/questions if you're curious here:
>
> https://docs.google.com/document/d/1WkZMjYxEyEWHqmjKpE2idOLUCmQG6Fa5E6TQQiAL-BU/edit
>
> Once we have answered those questions we will move forward with planning
> how the transition will happen. I would note we think that if we were to
> ask for a larger seat count (tbd what that number would be, 100-500 would
> be a rough range for all QA, dev, and community needs), we should also look
> at paying for the service as it would no longer be a small side project but
> rather a large, key piece of QA operations at Mozilla.
>
>
>
> On Wed, Apr 6, 2016 at 11:38 AM, Lawrence Mandel <[hidden email]>
> wrote:
>
>> Hi all,
>>
>> I would like to proceed with TestRail (or another system but from what I
>> gather this one seems to meet our needs). Specifically, I do not think that
>> we should continue to develop our own test case management tool in house.
>> Has anyone spoken with TestRail about their open source license? If not, I
>> volunteer Stuart and myself to do so.
>>
>> Lawrence
>>
>> On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <
>> [hidden email]> wrote:
>>
>>> Hi guys,
>>>
>>> I think it was John who mentioned that we may be able to get a free
>>> license
>>> for Test Rail (based on this:
>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>>> ).
>>> Was wondering if anyone has contacted them to ask about this.
>>>
>>> I'm also adding Lawrence to this thread, as we've been discussing this
>>> internally, and our QA teams are very interested in getting this to
>>> replace
>>> MozTrap, gdocs.
>>>
>>> Regards,
>>> Florin.
>>>
>>> -----Original Message-----
>>> From: dev-quality
>>> [mailto:dev-quality-bounces+florin.mezei=
>>> [hidden email]]
>>> On Behalf Of Delphine Lebédel
>>> Sent: Saturday, April 02, 2016 3:50 AM
>>> To: Stuart Philp <[hidden email]>
>>> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
>>> [hidden email]; Naoki Hirata <[hidden email]>
>>> Subject: Re: Test Case Management
>>>
>>> It's been more than a year that I myself haven't executed l10n testing on
>>> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
>>> Marigold would have had good feedback for this - but I guess we can't
>>> really
>>> ask them anymore.
>>>
>>> What I do remember that is really important for l10n has been highlighted
>>> by
>>> Naoki and Stuart: the need for screenshots (really necessary) and the
>>> cited
>>> meta data. Something that would have been helpful (but not only from l10n
>>> point of view) would have been the ability to file bugs directly within
>>> moztrap, and not have to go back and forth between moztrap and bugzilla.
>>>
>>> I think I'd need to dig into the tool again to see if I can come with
>>> something else useful. Thanks for getting the conversation started :)
>>>
>>> delphine
>>>
>>> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]> wrote:
>>>
>>>> Yeah fair points. We've been assessing it so far mostly for our needs,
>>>> which don't include a lot of l10n testing. But it looks to have the
>>>> flexibility in the test case meta data to assign tests to locales,
>>>> versions, releases, etc.
>>>>
>>>>
>>>>
>>>> On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> I hope that it's asystem that supports showing screenshots and
>>>>> comparisons as well.  I find that it helps for l10n testing as well as
>>> other things.
>>>>> Regards,
>>>>> Naoki
>>>>>
>>>>> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> sorry for chiming in late. It'd be great if our test case
>>>>>> management supported running tests on localized versions, and had
>>>>>> some reporting support for that.
>>>>>>
>>>>>> CCing Delphine for the insights we gathered from our FxOS testing
>>>>> programs.
>>>>>> Axel
>>>>>>
>>>>>> On 25/02/16 00:30, John Dorlus wrote:
>>>>>>
>>>>>>> Greetings Mozillians,
>>>>>>>
>>>>>>> The quest for a test case management system has been a topic for a
>>>>> while
>>>>>>> now amongst several teams. Moztrap is a great tool but it's
>>>>> maintenance is
>>>>>>> slow and we have a need now to leverage someone else's code so we
>>>>>>> can focus on the new and shiny. There are many options out there
>>>>>>> and we want to
>>>>> find
>>>>>>> one that will support our Open Sourceness. We found a tools seems
>>>>>>> interesting and want to get some opinions on this or other
>>>>> alternatives.
>>>>>>> TestRail - TestRail is one of the most widely used Test Case
>>>>>>> Management systems amongst top tier tech companies. Comes with all
>>>>>>> of the bells
>>>>> and
>>>>>>> whistles (including integration with Bugzilla and Github) but is
>>>>> normally
>>>>>>> super expensive. I found a page on the TestRail site saying they
>>>>>>> give
>>>>> free
>>>>>>> licenses to open source projects.1
>>>>>>>
>>>>>>> Have any of you heard of something better? Better support? Better
>>>>>>> integration? Is TestRail a good choice? Let us know your thoughts.
>>>>>>>
>>>>>>> 1 -
>>>>>>>
>>>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
>>>>> ts/
>>>>>>> [image: photo]
>>>>>>> *John Dorlus*
>>>>>>> Senior Software Engineer in Test, Mozilla Corporation
>>>>>>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
>>>>>>> |
>>>>> a:331
>>>>>>> E. Evelyn Avenue Mountain View, CA 94041 USA
>>>>>>> <http://www.facebook.com/mozilla>  <
>>>>> http://www.linkedin.com/in/johndorlus
>>>>>>> <http://www.twitter.com/the_power_coder>  <
>>>>> http://www.github.com/silne30>
>>>>>>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>>>>>>> <http://www.linkedin.com/company/mozilla-corporation>
>>>>>>> Get a signature like this: Click here!
>>>>>>> <
>>>>>>>
>>>>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
>>>>> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
>>>>> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
>>>>> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
>>>>> 478
>>>>>>>
>>>>>> _______________________________________________
>>>>>> dev-quality mailing list
>>>>>> [hidden email]
>>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>>
>>>>> _______________________________________________
>>>>> dev-quality mailing list
>>>>> [hidden email]
>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>
>>> _______________________________________________
>>> dev-quality mailing list
>>> [hidden email]
>>> https://lists.mozilla.org/listinfo/dev-quality
>>>
>>>
>>>

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality


 
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Lawrence Mandel-2
In reply to this post by Axel Hecht
Looks like there are some open questions for which we can get answers. A
few points that I'd like to make:

1. Support for MozTrap by the Engineering Productivity team will cease at
some point. I would like that point to be soon.
2. I think we should find an existing solution and not build this product
internally. (However, if a team wants to pick up MozTrap, that's up to
them.)
3. Each team can handle their data in their own way. (Each team knows what
needs to be preserved, populated, shared, etc.) Happy for there to be help
migrating. Also happy to just provide data dumps.

What is blocking us from moving forward with TestRail given that there
seems to be decent support for this tool?

For scale, do we all need to use the same instance? Would we be better
served by setting up separate instances?
Is there a reason that we opted for an internal deployment instead of using
TestRail's servers?

Lawrence

On Thu, Apr 7, 2016 at 5:33 PM, Axel Hecht <[hidden email]> wrote:

> If there's need/opportunity: the Gurock brothers are in Berlin, and we
> have folks in Berlin, so maybe we'll be able to get a in-person
> conversation.
>
> Re Stuart's
>
> Yeah fair points. We've been assessing it so far mostly for our needs,
>> which don't include a lot of l10n testing. But it looks to have the
>> flexibility in the test case meta data to assign tests to locales,
>> versions, releases, etc.
>>
> I'm concerned about that from an l10n POV. I'd like to see practical
> experience.
>
> With moztrap, we were in a similar boat, and I think we paid quite some
> price. Features like testing speed projections etc really need to be
> designed to support testing products that are localized.
>
> Side-stepping, on the l10n side, we're quite happy about categorizing test
> results, and they're in a hierarchy of results. In l10n, a plain "yes/no"
> isn't good enough.
> http://www.qt21.eu/mqm-definition/issues-list-2015-12-30.html#issues_list
> is looking promising, in categorizing test failures. Those issue types at a
> corse level allow to understand our quality, and at a deeper level instruct
> us on how to improve and coach.
>
> I'd personally really like to hear some war stories from the Gurocks or
> their clients in the field of l10n.
>
> Axel
>
> On 06/04/16 17:54, Stuart Philp wrote:
>
>> My team has a free open-source license from gurock for 50 seats which
>> we've
>> been using to evaluate testrail, it is running here:
>> http://testrail.stage.mozaws.net/
>>
>> We have also decided to go forward with trying it more permanently as our
>> early feedback was mostly positive. We do however have a few questions we
>> want to answer, you can see our discussion/questions if you're curious
>> here:
>>
>>
>> https://docs.google.com/document/d/1WkZMjYxEyEWHqmjKpE2idOLUCmQG6Fa5E6TQQiAL-BU/edit
>>
>> Once we have answered those questions we will move forward with planning
>> how the transition will happen. I would note we think that if we were to
>> ask for a larger seat count (tbd what that number would be, 100-500 would
>> be a rough range for all QA, dev, and community needs), we should also
>> look
>> at paying for the service as it would no longer be a small side project
>> but
>> rather a large, key piece of QA operations at Mozilla.
>>
>>
>>
>> On Wed, Apr 6, 2016 at 11:38 AM, Lawrence Mandel <[hidden email]>
>> wrote:
>>
>> Hi all,
>>>
>>> I would like to proceed with TestRail (or another system but from what I
>>> gather this one seems to meet our needs). Specifically, I do not think
>>> that
>>> we should continue to develop our own test case management tool in house.
>>> Has anyone spoken with TestRail about their open source license? If not,
>>> I
>>> volunteer Stuart and myself to do so.
>>>
>>> Lawrence
>>>
>>> On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <
>>> [hidden email]> wrote:
>>>
>>> Hi guys,
>>>>
>>>> I think it was John who mentioned that we may be able to get a free
>>>> license
>>>> for Test Rail (based on this:
>>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>>>> ).
>>>> Was wondering if anyone has contacted them to ask about this.
>>>>
>>>> I'm also adding Lawrence to this thread, as we've been discussing this
>>>> internally, and our QA teams are very interested in getting this to
>>>> replace
>>>> MozTrap, gdocs.
>>>>
>>>> Regards,
>>>> Florin.
>>>>
>>>> -----Original Message-----
>>>> From: dev-quality
>>>> [mailto:dev-quality-bounces+florin.mezei=
>>>> [hidden email]]
>>>> On Behalf Of Delphine Lebédel
>>>> Sent: Saturday, April 02, 2016 3:50 AM
>>>> To: Stuart Philp <[hidden email]>
>>>> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
>>>> [hidden email]; Naoki Hirata <[hidden email]>
>>>> Subject: Re: Test Case Management
>>>>
>>>> It's been more than a year that I myself haven't executed l10n testing
>>>> on
>>>> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
>>>> Marigold would have had good feedback for this - but I guess we can't
>>>> really
>>>> ask them anymore.
>>>>
>>>> What I do remember that is really important for l10n has been
>>>> highlighted
>>>> by
>>>> Naoki and Stuart: the need for screenshots (really necessary) and the
>>>> cited
>>>> meta data. Something that would have been helpful (but not only from
>>>> l10n
>>>> point of view) would have been the ability to file bugs directly within
>>>> moztrap, and not have to go back and forth between moztrap and bugzilla.
>>>>
>>>> I think I'd need to dig into the tool again to see if I can come with
>>>> something else useful. Thanks for getting the conversation started :)
>>>>
>>>> delphine
>>>>
>>>> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]>
>>>> wrote:
>>>>
>>>> Yeah fair points. We've been assessing it so far mostly for our needs,
>>>>> which don't include a lot of l10n testing. But it looks to have the
>>>>> flexibility in the test case meta data to assign tests to locales,
>>>>> versions, releases, etc.
>>>>>
>>>>>
>>>>>
>>>>> On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
>>>>>
>>>>> +1
>>>>>>
>>>>>> I hope that it's asystem that supports showing screenshots and
>>>>>> comparisons as well.  I find that it helps for l10n testing as well as
>>>>>>
>>>>> other things.
>>>>
>>>>> Regards,
>>>>>> Naoki
>>>>>>
>>>>>> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>>
>>>>>>> sorry for chiming in late. It'd be great if our test case
>>>>>>> management supported running tests on localized versions, and had
>>>>>>> some reporting support for that.
>>>>>>>
>>>>>>> CCing Delphine for the insights we gathered from our FxOS testing
>>>>>>>
>>>>>> programs.
>>>>>>
>>>>>>> Axel
>>>>>>>
>>>>>>> On 25/02/16 00:30, John Dorlus wrote:
>>>>>>>
>>>>>>> Greetings Mozillians,
>>>>>>>>
>>>>>>>> The quest for a test case management system has been a topic for a
>>>>>>>>
>>>>>>> while
>>>>>>
>>>>>>> now amongst several teams. Moztrap is a great tool but it's
>>>>>>>>
>>>>>>> maintenance is
>>>>>>
>>>>>>> slow and we have a need now to leverage someone else's code so we
>>>>>>>> can focus on the new and shiny. There are many options out there
>>>>>>>> and we want to
>>>>>>>>
>>>>>>> find
>>>>>>
>>>>>>> one that will support our Open Sourceness. We found a tools seems
>>>>>>>> interesting and want to get some opinions on this or other
>>>>>>>>
>>>>>>> alternatives.
>>>>>>
>>>>>>> TestRail - TestRail is one of the most widely used Test Case
>>>>>>>> Management systems amongst top tier tech companies. Comes with all
>>>>>>>> of the bells
>>>>>>>>
>>>>>>> and
>>>>>>
>>>>>>> whistles (including integration with Bugzilla and Github) but is
>>>>>>>>
>>>>>>> normally
>>>>>>
>>>>>>> super expensive. I found a page on the TestRail site saying they
>>>>>>>> give
>>>>>>>>
>>>>>>> free
>>>>>>
>>>>>>> licenses to open source projects.1
>>>>>>>>
>>>>>>>> Have any of you heard of something better? Better support? Better
>>>>>>>> integration? Is TestRail a good choice? Let us know your thoughts.
>>>>>>>>
>>>>>>>> 1 -
>>>>>>>>
>>>>>>>>
>>>>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
>>>>>> ts/
>>>>>>
>>>>>>> [image: photo]
>>>>>>>> *John Dorlus*
>>>>>>>> Senior Software Engineer in Test, Mozilla Corporation
>>>>>>>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
>>>>>>>> |
>>>>>>>>
>>>>>>> a:331
>>>>>>
>>>>>>> E. Evelyn Avenue Mountain View, CA 94041 USA
>>>>>>>> <http://www.facebook.com/mozilla>  <
>>>>>>>>
>>>>>>> http://www.linkedin.com/in/johndorlus
>>>>>>
>>>>>>> <http://www.twitter.com/the_power_coder>  <
>>>>>>>>
>>>>>>> http://www.github.com/silne30>
>>>>>>
>>>>>>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>>>>>>>> <http://www.linkedin.com/company/mozilla-corporation>
>>>>>>>> Get a signature like this: Click here!
>>>>>>>> <
>>>>>>>>
>>>>>>>>
>>>>>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
>>>>>> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
>>>>>> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
>>>>>> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
>>>>>> 478
>>>>>>
>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>> dev-quality mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>>>
>>>>>>> _______________________________________________
>>>>>> dev-quality mailing list
>>>>>> [hidden email]
>>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>>
>>>>>> _______________________________________________
>>>> dev-quality mailing list
>>>> [hidden email]
>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>
>>>>
>>>>
>>>>
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Stuart Philp
On Wed, Apr 13, 2016 at 2:07 PM, Lawrence Mandel <[hidden email]>
wrote:

> Looks like there are some open questions for which we can get answers. A
> few points that I'd like to make:
>
> 1. Support for MozTrap by the Engineering Productivity team will cease at
> some point. I would like that point to be soon.
>
2. I think we should find an existing solution and not build this product

> internally. (However, if a team wants to pick up MozTrap, that's up to
> them.)
> 3. Each team can handle their data in their own way. (Each team knows what
> needs to be preserved, populated, shared, etc.) Happy for there to be help
> migrating. Also happy to just provide data dumps.
>
> What is blocking us from moving forward with TestRail given that there
> seems to be decent support for this tool?
>
> I think we're very close to being able to move after the load/scale
testing is complete. We look to have means to extract data from MozTrap and
to import into testrail, which should tell us how it handles large data
sets, and make transitioning easier for everyone. Based on that we can
determine how we want to proceed. We will update here when it is complete.


> For scale, do we all need to use the same instance? Would we be better
> served by setting up separate instances?
> Is there a reason that we opted for an internal deployment instead of
> using TestRail's servers?
>

I believe open source license requires us to self-host, but that is another
reason why if we decide to put our full fledged support behind this, we
could talk to them about paying + have them host/manage.

>
> Lawrence
>
> On Thu, Apr 7, 2016 at 5:33 PM, Axel Hecht <[hidden email]> wrote:
>
>> If there's need/opportunity: the Gurock brothers are in Berlin, and we
>> have folks in Berlin, so maybe we'll be able to get a in-person
>> conversation.
>>
>> Re Stuart's
>>
>> Yeah fair points. We've been assessing it so far mostly for our needs,
>>> which don't include a lot of l10n testing. But it looks to have the
>>> flexibility in the test case meta data to assign tests to locales,
>>> versions, releases, etc.
>>>
>> I'm concerned about that from an l10n POV. I'd like to see practical
>> experience.
>>
>> With moztrap, we were in a similar boat, and I think we paid quite some
>> price. Features like testing speed projections etc really need to be
>> designed to support testing products that are localized.
>>
>> Side-stepping, on the l10n side, we're quite happy about categorizing
>> test results, and they're in a hierarchy of results. In l10n, a plain
>> "yes/no" isn't good enough.
>> http://www.qt21.eu/mqm-definition/issues-list-2015-12-30.html#issues_list
>> is looking promising, in categorizing test failures. Those issue types at a
>> corse level allow to understand our quality, and at a deeper level instruct
>> us on how to improve and coach.
>>
>> I'd personally really like to hear some war stories from the Gurocks or
>> their clients in the field of l10n.
>>
>> Axel
>>
>> On 06/04/16 17:54, Stuart Philp wrote:
>>
>>> My team has a free open-source license from gurock for 50 seats which
>>> we've
>>> been using to evaluate testrail, it is running here:
>>> http://testrail.stage.mozaws.net/
>>>
>>> We have also decided to go forward with trying it more permanently as our
>>> early feedback was mostly positive. We do however have a few questions we
>>> want to answer, you can see our discussion/questions if you're curious
>>> here:
>>>
>>>
>>> https://docs.google.com/document/d/1WkZMjYxEyEWHqmjKpE2idOLUCmQG6Fa5E6TQQiAL-BU/edit
>>>
>>> Once we have answered those questions we will move forward with planning
>>> how the transition will happen. I would note we think that if we were to
>>> ask for a larger seat count (tbd what that number would be, 100-500 would
>>> be a rough range for all QA, dev, and community needs), we should also
>>> look
>>> at paying for the service as it would no longer be a small side project
>>> but
>>> rather a large, key piece of QA operations at Mozilla.
>>>
>>>
>>>
>>> On Wed, Apr 6, 2016 at 11:38 AM, Lawrence Mandel <[hidden email]>
>>> wrote:
>>>
>>> Hi all,
>>>>
>>>> I would like to proceed with TestRail (or another system but from what I
>>>> gather this one seems to meet our needs). Specifically, I do not think
>>>> that
>>>> we should continue to develop our own test case management tool in
>>>> house.
>>>> Has anyone spoken with TestRail about their open source license? If
>>>> not, I
>>>> volunteer Stuart and myself to do so.
>>>>
>>>> Lawrence
>>>>
>>>> On Wed, Apr 6, 2016 at 11:31 AM, Florin Mezei <
>>>> [hidden email]> wrote:
>>>>
>>>> Hi guys,
>>>>>
>>>>> I think it was John who mentioned that we may be able to get a free
>>>>> license
>>>>> for Test Rail (based on this:
>>>>>
>>>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>>>>> ).
>>>>> Was wondering if anyone has contacted them to ask about this.
>>>>>
>>>>> I'm also adding Lawrence to this thread, as we've been discussing this
>>>>> internally, and our QA teams are very interested in getting this to
>>>>> replace
>>>>> MozTrap, gdocs.
>>>>>
>>>>> Regards,
>>>>> Florin.
>>>>>
>>>>> -----Original Message-----
>>>>> From: dev-quality
>>>>> [mailto:dev-quality-bounces+florin.mezei=
>>>>> [hidden email]]
>>>>> On Behalf Of Delphine Lebédel
>>>>> Sent: Saturday, April 02, 2016 3:50 AM
>>>>> To: Stuart Philp <[hidden email]>
>>>>> Cc: John Dorlus <[hidden email]>; Axel Hecht <[hidden email]>;
>>>>> [hidden email]; Naoki Hirata <[hidden email]>
>>>>> Subject: Re: Test Case Management
>>>>>
>>>>> It's been more than a year that I myself haven't executed l10n testing
>>>>> on
>>>>> Moztrap, so it's become a bit vague in my head. I'm sure QAnalysts and
>>>>> Marigold would have had good feedback for this - but I guess we can't
>>>>> really
>>>>> ask them anymore.
>>>>>
>>>>> What I do remember that is really important for l10n has been
>>>>> highlighted
>>>>> by
>>>>> Naoki and Stuart: the need for screenshots (really necessary) and the
>>>>> cited
>>>>> meta data. Something that would have been helpful (but not only from
>>>>> l10n
>>>>> point of view) would have been the ability to file bugs directly within
>>>>> moztrap, and not have to go back and forth between moztrap and
>>>>> bugzilla.
>>>>>
>>>>> I think I'd need to dig into the tool again to see if I can come with
>>>>> something else useful. Thanks for getting the conversation started :)
>>>>>
>>>>> delphine
>>>>>
>>>>> On Fri, Apr 1, 2016 at 5:08 PM, Stuart Philp <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> Yeah fair points. We've been assessing it so far mostly for our needs,
>>>>>> which don't include a lot of l10n testing. But it looks to have the
>>>>>> flexibility in the test case meta data to assign tests to locales,
>>>>>> versions, releases, etc.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Friday, 1 April 2016, Naoki Hirata <[hidden email]> wrote:
>>>>>>
>>>>>> +1
>>>>>>>
>>>>>>> I hope that it's asystem that supports showing screenshots and
>>>>>>> comparisons as well.  I find that it helps for l10n testing as well
>>>>>>> as
>>>>>>>
>>>>>> other things.
>>>>>
>>>>>> Regards,
>>>>>>> Naoki
>>>>>>>
>>>>>>> On Fri, Apr 1, 2016 at 3:52 PM, Axel Hecht <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>>
>>>>>>>> sorry for chiming in late. It'd be great if our test case
>>>>>>>> management supported running tests on localized versions, and had
>>>>>>>> some reporting support for that.
>>>>>>>>
>>>>>>>> CCing Delphine for the insights we gathered from our FxOS testing
>>>>>>>>
>>>>>>> programs.
>>>>>>>
>>>>>>>> Axel
>>>>>>>>
>>>>>>>> On 25/02/16 00:30, John Dorlus wrote:
>>>>>>>>
>>>>>>>> Greetings Mozillians,
>>>>>>>>>
>>>>>>>>> The quest for a test case management system has been a topic for a
>>>>>>>>>
>>>>>>>> while
>>>>>>>
>>>>>>>> now amongst several teams. Moztrap is a great tool but it's
>>>>>>>>>
>>>>>>>> maintenance is
>>>>>>>
>>>>>>>> slow and we have a need now to leverage someone else's code so we
>>>>>>>>> can focus on the new and shiny. There are many options out there
>>>>>>>>> and we want to
>>>>>>>>>
>>>>>>>> find
>>>>>>>
>>>>>>>> one that will support our Open Sourceness. We found a tools seems
>>>>>>>>> interesting and want to get some opinions on this or other
>>>>>>>>>
>>>>>>>> alternatives.
>>>>>>>
>>>>>>>> TestRail - TestRail is one of the most widely used Test Case
>>>>>>>>> Management systems amongst top tier tech companies. Comes with all
>>>>>>>>> of the bells
>>>>>>>>>
>>>>>>>> and
>>>>>>>
>>>>>>>> whistles (including integration with Bugzilla and Github) but is
>>>>>>>>>
>>>>>>>> normally
>>>>>>>
>>>>>>>> super expensive. I found a page on the TestRail site saying they
>>>>>>>>> give
>>>>>>>>>
>>>>>>>> free
>>>>>>>
>>>>>>>> licenses to open source projects.1
>>>>>>>>>
>>>>>>>>> Have any of you heard of something better? Better support? Better
>>>>>>>>> integration? Is TestRail a good choice? Let us know your thoughts.
>>>>>>>>>
>>>>>>>>> 1 -
>>>>>>>>>
>>>>>>>>>
>>>>>>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projec
>>>>>>> ts/
>>>>>>>
>>>>>>>> [image: photo]
>>>>>>>>> *John Dorlus*
>>>>>>>>> Senior Software Engineer in Test, Mozilla Corporation
>>>>>>>>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
>>>>>>>>> |
>>>>>>>>>
>>>>>>>> a:331
>>>>>>>
>>>>>>>> E. Evelyn Avenue Mountain View, CA 94041 USA
>>>>>>>>> <http://www.facebook.com/mozilla>  <
>>>>>>>>>
>>>>>>>> http://www.linkedin.com/in/johndorlus
>>>>>>>
>>>>>>>> <http://www.twitter.com/the_power_coder>  <
>>>>>>>>>
>>>>>>>> http://www.github.com/silne30>
>>>>>>>
>>>>>>>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>>>>>>>>> <http://www.linkedin.com/company/mozilla-corporation>
>>>>>>>>> Get a signature like this: Click here!
>>>>>>>>> <
>>>>>>>>>
>>>>>>>>>
>>>>>>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lz
>>>>>>> ZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY
>>>>>>> 2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3Mz
>>>>>>> I1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350
>>>>>>> 478
>>>>>>>
>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>> dev-quality mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>> dev-quality mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>>>
>>>>>>> _______________________________________________
>>>>> dev-quality mailing list
>>>>> [hidden email]
>>>>> https://lists.mozilla.org/listinfo/dev-quality
>>>>>
>>>>>
>>>>>
>>>>>
>>
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality