Test Case Management

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Test Case Management

John Dorlus
Greetings Mozillians,

The quest for a test case management system has been a topic for a while
now amongst several teams. Moztrap is a great tool but it's maintenance is
slow and we have a need now to leverage someone else's code so we can focus
on the new and shiny. There are many options out there and we want to find
one that will support our Open Sourceness. We found a tools seems
interesting and want to get some opinions on this or other alternatives.

TestRail - TestRail is one of the most widely used Test Case Management
systems amongst top tier tech companies. Comes with all of the bells and
whistles (including integration with Bugzilla and Github) but is normally
super expensive. I found a page on the TestRail site saying they give free
licenses to open source projects.1

Have any of you heard of something better? Better support? Better
integration? Is TestRail a good choice? Let us know your thoughts.

1 - https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/

[image: photo]
*John Dorlus*
Senior Software Engineer in Test, Mozilla Corporation
m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
E. Evelyn Avenue Mountain View, CA 94041 USA
<http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus>
<http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
<http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
<http://www.linkedin.com/company/mozilla-corporation>
Get a signature like this: Click here!
<http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Stuart Philp
I've used TestRail previously, it takes a bit to set up and learn, but it
is very detailed, and nicely separates "runs" from "plans", and can
automatically file/comment on bugs (or at least did when I used it with
Jira, I'm sure bugzilla is the same). I wasn't aware of the free open
source licenses, but it's definitely worth looking at.

On Wed, Feb 24, 2016 at 6:30 PM, John Dorlus <[hidden email]> wrote:

> Greetings Mozillians,
>
> The quest for a test case management system has been a topic for a while
> now amongst several teams. Moztrap is a great tool but it's maintenance is
> slow and we have a need now to leverage someone else's code so we can focus
> on the new and shiny. There are many options out there and we want to find
> one that will support our Open Sourceness. We found a tools seems
> interesting and want to get some opinions on this or other alternatives.
>
> TestRail - TestRail is one of the most widely used Test Case Management
> systems amongst top tier tech companies. Comes with all of the bells and
> whistles (including integration with Bugzilla and Github) but is normally
> super expensive. I found a page on the TestRail site saying they give free
> licenses to open source projects.1
>
> Have any of you heard of something better? Better support? Better
> integration? Is TestRail a good choice? Let us know your thoughts.
>
> 1 -
> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>
> [image: photo]
> *John Dorlus*
> Senior Software Engineer in Test, Mozilla Corporation
> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
> E. Evelyn Avenue Mountain View, CA 94041 USA
> <http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus>
> <http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> <http://www.linkedin.com/company/mozilla-corporation>
> Get a signature like this: Click here!
> <
> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
> >
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Kevin Brosnan
In reply to this post by John Dorlus
From a quick look it is a slick interface. Is there any limit to the number
of projects we can create? Limits on the number of users for their open
source plan? How would it look with multiple dispersed teams?

If they require us to go through their pricing structure. I am a bit
concerned that it makes community testing quite difficult. With a per user
pricing model we would need to decide if we want to pay for community
access. If we do then there will be a need to prune inactive community
members. There is also the problem of how to split the costs over the
several teams reporting to different parts of the org.

Kevin

On Wed, Feb 24, 2016 at 3:30 PM, John Dorlus <[hidden email]> wrote:

> Greetings Mozillians,
>
> The quest for a test case management system has been a topic for a while
> now amongst several teams. Moztrap is a great tool but it's maintenance is
> slow and we have a need now to leverage someone else's code so we can focus
> on the new and shiny. There are many options out there and we want to find
> one that will support our Open Sourceness. We found a tools seems
> interesting and want to get some opinions on this or other alternatives.
>
> TestRail - TestRail is one of the most widely used Test Case Management
> systems amongst top tier tech companies. Comes with all of the bells and
> whistles (including integration with Bugzilla and Github) but is normally
> super expensive. I found a page on the TestRail site saying they give free
> licenses to open source projects.1
>
> Have any of you heard of something better? Better support? Better
> integration? Is TestRail a good choice? Let us know your thoughts.
>
> 1 -
> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>
> [image: photo]
> *John Dorlus*
> Senior Software Engineer in Test, Mozilla Corporation
> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
> E. Evelyn Avenue Mountain View, CA 94041 USA
> <http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus>
> <http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> <http://www.linkedin.com/company/mozilla-corporation>
> Get a signature like this: Click here!
> <
> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
> >
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

John Dorlus
Hi Kevin,

Very good question about the teams. From the wording, it says open source teams and open source projects. It seems like the criteria is that the source is open, period. I will reach out to the company and find out what the guidelines are.

J.D.

> On Feb 24, 2016, at 7:04 PM, Kevin Brosnan <[hidden email]> wrote:
>
> From a quick look it is a slick interface. Is there any limit to the number of projects we can create? Limits on the number of users for their open source plan? How would it look with multiple dispersed teams?
>
> If they require us to go through their pricing structure. I am a bit concerned that it makes community testing quite difficult. With a per user pricing model we would need to decide if we want to pay for community access. If we do then there will be a need to prune inactive community members. There is also the problem of how to split the costs over the several teams reporting to different parts of the org.
>
> Kevin
>
>> On Wed, Feb 24, 2016 at 3:30 PM, John Dorlus <[hidden email]> wrote:
>> Greetings Mozillians,
>>
>> The quest for a test case management system has been a topic for a while
>> now amongst several teams. Moztrap is a great tool but it's maintenance is
>> slow and we have a need now to leverage someone else's code so we can focus
>> on the new and shiny. There are many options out there and we want to find
>> one that will support our Open Sourceness. We found a tools seems
>> interesting and want to get some opinions on this or other alternatives.
>>
>> TestRail - TestRail is one of the most widely used Test Case Management
>> systems amongst top tier tech companies. Comes with all of the bells and
>> whistles (including integration with Bugzilla and Github) but is normally
>> super expensive. I found a page on the TestRail site saying they give free
>> licenses to open source projects.1
>>
>> Have any of you heard of something better? Better support? Better
>> integration? Is TestRail a good choice? Let us know your thoughts.
>>
>> 1 - https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>>
>> [image: photo]
>> *John Dorlus*
>> Senior Software Engineer in Test, Mozilla Corporation
>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
>> E. Evelyn Avenue Mountain View, CA 94041 USA
>> <http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus>
>> <http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>> <http://www.linkedin.com/company/mozilla-corporation>
>> Get a signature like this: Click here!
>> <http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478>
>> _______________________________________________
>> dev-quality mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Dave Hunt-3
In reply to this post by John Dorlus
I like that there’s a pytest plugin for it!
https://github.com/allankilpatrick/pytest-testrail/ <https://github.com/allankilpatrick/pytest-testrail/>

Dave

> On 24 Feb 2016, at 23:30, John Dorlus <[hidden email]> wrote:
>
> Greetings Mozillians,
>
> The quest for a test case management system has been a topic for a while
> now amongst several teams. Moztrap is a great tool but it's maintenance is
> slow and we have a need now to leverage someone else's code so we can focus
> on the new and shiny. There are many options out there and we want to find
> one that will support our Open Sourceness. We found a tools seems
> interesting and want to get some opinions on this or other alternatives.
>
> TestRail - TestRail is one of the most widely used Test Case Management
> systems amongst top tier tech companies. Comes with all of the bells and
> whistles (including integration with Bugzilla and Github) but is normally
> super expensive. I found a page on the TestRail site saying they give free
> licenses to open source projects.1
>
> Have any of you heard of something better? Better support? Better
> integration? Is TestRail a good choice? Let us know your thoughts.
>
> 1 - https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>
> [image: photo]
> *John Dorlus*
> Senior Software Engineer in Test, Mozilla Corporation
> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
> E. Evelyn Avenue Mountain View, CA 94041 USA
> <http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus>
> <http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> <http://www.linkedin.com/company/mozilla-corporation>
> Get a signature like this: Click here!
> <http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478>
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

John Dorlus
That's all it takes to sell Dave. 😆

J.D.

> On Feb 25, 2016, at 5:06 AM, Dave Hunt <[hidden email]> wrote:
>
> I like that there’s a pytest plugin for it!
> https://github.com/allankilpatrick/pytest-testrail/
>
> Dave
>
>> On 24 Feb 2016, at 23:30, John Dorlus <[hidden email]> wrote:
>>
>> Greetings Mozillians,
>>
>> The quest for a test case management system has been a topic for a while
>> now amongst several teams. Moztrap is a great tool but it's maintenance is
>> slow and we have a need now to leverage someone else's code so we can focus
>> on the new and shiny. There are many options out there and we want to find
>> one that will support our Open Sourceness. We found a tools seems
>> interesting and want to get some opinions on this or other alternatives.
>>
>> TestRail - TestRail is one of the most widely used Test Case Management
>> systems amongst top tier tech companies. Comes with all of the bells and
>> whistles (including integration with Bugzilla and Github) but is normally
>> super expensive. I found a page on the TestRail site saying they give free
>> licenses to open source projects.1
>>
>> Have any of you heard of something better? Better support? Better
>> integration? Is TestRail a good choice? Let us know your thoughts.
>>
>> 1 - https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>>
>> [image: photo]
>> *John Dorlus*
>> Senior Software Engineer in Test, Mozilla Corporation
>> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
>> E. Evelyn Avenue Mountain View, CA 94041 USA
>> <http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus>
>> <http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
>> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>> <http://www.linkedin.com/company/mozilla-corporation>
>> Get a signature like this: Click here!
>> <http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478>
>> _______________________________________________
>> dev-quality mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Stuart Philp
Thanks to Mr. Dorlus, we have an account set up with Gurock to give
testrail a spin. If anyone is interested in playing around with it, I've
set it up in our staging AWS environment. Shoot me a message and I'll add
you. We have 50 seats so should be plenty of room for now.

Cheers

On Thu, Feb 25, 2016 at 11:43 AM, John Dorlus <[hidden email]> wrote:

> That's all it takes to sell Dave. 😆
>
> J.D.
>
> > On Feb 25, 2016, at 5:06 AM, Dave Hunt <[hidden email]> wrote:
> >
> > I like that there’s a pytest plugin for it!
> > https://github.com/allankilpatrick/pytest-testrail/
> >
> > Dave
> >
> >> On 24 Feb 2016, at 23:30, John Dorlus <[hidden email]> wrote:
> >>
> >> Greetings Mozillians,
> >>
> >> The quest for a test case management system has been a topic for a while
> >> now amongst several teams. Moztrap is a great tool but it's maintenance
> is
> >> slow and we have a need now to leverage someone else's code so we can
> focus
> >> on the new and shiny. There are many options out there and we want to
> find
> >> one that will support our Open Sourceness. We found a tools seems
> >> interesting and want to get some opinions on this or other alternatives.
> >>
> >> TestRail - TestRail is one of the most widely used Test Case Management
> >> systems amongst top tier tech companies. Comes with all of the bells and
> >> whistles (including integration with Bugzilla and Github) but is
> normally
> >> super expensive. I found a page on the TestRail site saying they give
> free
> >> licenses to open source projects.1
> >>
> >> Have any of you heard of something better? Better support? Better
> >> integration? Is TestRail a good choice? Let us know your thoughts.
> >>
> >> 1 -
> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
> >>
> >> [image: photo]
> >> *John Dorlus*
> >> Senior Software Engineer in Test, Mozilla Corporation
> >> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org |
> a:331
> >> E. Evelyn Avenue Mountain View, CA 94041 USA
> >> <http://www.facebook.com/mozilla>  <
> http://www.linkedin.com/in/johndorlus>
> >> <http://www.twitter.com/the_power_coder>  <
> http://www.github.com/silne30>
> >> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> >> <http://www.linkedin.com/company/mozilla-corporation>
> >> Get a signature like this: Click here!
> >> <
> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
> >
> >> _______________________________________________
> >> dev-quality mailing list
> >> [hidden email]
> >> https://lists.mozilla.org/listinfo/dev-quality
> >
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Kamil Jozwiak
If we decide to start using TestRail rather than MozTrap, will the current
tests that are still being used be migrated over? Will the open source
license cover SV and our community? I might be wrong here, but I remember
that some of our community members were using the MozTrap platform during
test days. Is that still the case? Or have we moved away from MozTrap all
together?

On Mon, Feb 29, 2016 at 5:36 PM, Stuart Philp <[hidden email]> wrote:

> Thanks to Mr. Dorlus, we have an account set up with Gurock to give
> testrail a spin. If anyone is interested in playing around with it, I've
> set it up in our staging AWS environment. Shoot me a message and I'll add
> you. We have 50 seats so should be plenty of room for now.
>
> Cheers
>
> On Thu, Feb 25, 2016 at 11:43 AM, John Dorlus <[hidden email]> wrote:
>
> > That's all it takes to sell Dave. 😆
> >
> > J.D.
> >
> > > On Feb 25, 2016, at 5:06 AM, Dave Hunt <[hidden email]> wrote:
> > >
> > > I like that there’s a pytest plugin for it!
> > > https://github.com/allankilpatrick/pytest-testrail/
> > >
> > > Dave
> > >
> > >> On 24 Feb 2016, at 23:30, John Dorlus <[hidden email]> wrote:
> > >>
> > >> Greetings Mozillians,
> > >>
> > >> The quest for a test case management system has been a topic for a
> while
> > >> now amongst several teams. Moztrap is a great tool but it's
> maintenance
> > is
> > >> slow and we have a need now to leverage someone else's code so we can
> > focus
> > >> on the new and shiny. There are many options out there and we want to
> > find
> > >> one that will support our Open Sourceness. We found a tools seems
> > >> interesting and want to get some opinions on this or other
> alternatives.
> > >>
> > >> TestRail - TestRail is one of the most widely used Test Case
> Management
> > >> systems amongst top tier tech companies. Comes with all of the bells
> and
> > >> whistles (including integration with Bugzilla and Github) but is
> > normally
> > >> super expensive. I found a page on the TestRail site saying they give
> > free
> > >> licenses to open source projects.1
> > >>
> > >> Have any of you heard of something better? Better support? Better
> > >> integration? Is TestRail a good choice? Let us know your thoughts.
> > >>
> > >> 1 -
> > https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
> > >>
> > >> [image: photo]
> > >> *John Dorlus*
> > >> Senior Software Engineer in Test, Mozilla Corporation
> > >> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org |
> > a:331
> > >> E. Evelyn Avenue Mountain View, CA 94041 USA
> > >> <http://www.facebook.com/mozilla>  <
> > http://www.linkedin.com/in/johndorlus>
> > >> <http://www.twitter.com/the_power_coder>  <
> > http://www.github.com/silne30>
> > >> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> > >> <http://www.linkedin.com/company/mozilla-corporation>
> > >> Get a signature like this: Click here!
> > >> <
> >
> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
> > >
> > >> _______________________________________________
> > >> dev-quality mailing list
> > >> [hidden email]
> > >> https://lists.mozilla.org/listinfo/dev-quality
> > >
> > _______________________________________________
> > dev-quality mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-quality
> >
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Naoki Hirata
I personally don't mind moving away from moztrap.

The UX wasn't geared well to someone who works in it nor was the
performance.  I understand that there's been hard work and dedication put
into the project and I apologize if I hurt people's feelings.  In B2G, we
had to use a spreadsheet (for dogfood/smoke tests) and then put the results
in Moztrap which sort of defeats the purpose if we have to use two
different systems.  Filing bugs didn't seem to be prudent after awhile
because there was only a limited number of developers on it.

At the same time, there are people that have made improvements to the UI
for moztrap ( see mozilluminate project:
https://github.com/MozIlluminate/mozilluminate-demo

I think if the UX was made better, there's a chance to revive Moztrap.

Regards,
Naoki

On Tue, Mar 1, 2016 at 5:07 PM, Kamil Jozwiak <[hidden email]> wrote:

> If we decide to start using TestRail rather than MozTrap, will the current
> tests that are still being used be migrated over? Will the open source
> license cover SV and our community? I might be wrong here, but I remember
> that some of our community members were using the MozTrap platform during
> test days. Is that still the case? Or have we moved away from MozTrap all
> together?
>
> On Mon, Feb 29, 2016 at 5:36 PM, Stuart Philp <[hidden email]> wrote:
>
> > Thanks to Mr. Dorlus, we have an account set up with Gurock to give
> > testrail a spin. If anyone is interested in playing around with it, I've
> > set it up in our staging AWS environment. Shoot me a message and I'll add
> > you. We have 50 seats so should be plenty of room for now.
> >
> > Cheers
> >
> > On Thu, Feb 25, 2016 at 11:43 AM, John Dorlus <[hidden email]>
> wrote:
> >
> > > That's all it takes to sell Dave. 😆
> > >
> > > J.D.
> > >
> > > > On Feb 25, 2016, at 5:06 AM, Dave Hunt <[hidden email]> wrote:
> > > >
> > > > I like that there’s a pytest plugin for it!
> > > > https://github.com/allankilpatrick/pytest-testrail/
> > > >
> > > > Dave
> > > >
> > > >> On 24 Feb 2016, at 23:30, John Dorlus <[hidden email]> wrote:
> > > >>
> > > >> Greetings Mozillians,
> > > >>
> > > >> The quest for a test case management system has been a topic for a
> > while
> > > >> now amongst several teams. Moztrap is a great tool but it's
> > maintenance
> > > is
> > > >> slow and we have a need now to leverage someone else's code so we
> can
> > > focus
> > > >> on the new and shiny. There are many options out there and we want
> to
> > > find
> > > >> one that will support our Open Sourceness. We found a tools seems
> > > >> interesting and want to get some opinions on this or other
> > alternatives.
> > > >>
> > > >> TestRail - TestRail is one of the most widely used Test Case
> > Management
> > > >> systems amongst top tier tech companies. Comes with all of the bells
> > and
> > > >> whistles (including integration with Bugzilla and Github) but is
> > > normally
> > > >> super expensive. I found a page on the TestRail site saying they
> give
> > > free
> > > >> licenses to open source projects.1
> > > >>
> > > >> Have any of you heard of something better? Better support? Better
> > > >> integration? Is TestRail a good choice? Let us know your thoughts.
> > > >>
> > > >> 1 -
> > >
> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
> > > >>
> > > >> [image: photo]
> > > >> *John Dorlus*
> > > >> Senior Software Engineer in Test, Mozilla Corporation
> > > >> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org |
> > > a:331
> > > >> E. Evelyn Avenue Mountain View, CA 94041 USA
> > > >> <http://www.facebook.com/mozilla>  <
> > > http://www.linkedin.com/in/johndorlus>
> > > >> <http://www.twitter.com/the_power_coder>  <
> > > http://www.github.com/silne30>
> > > >> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> > > >> <http://www.linkedin.com/company/mozilla-corporation>
> > > >> Get a signature like this: Click here!
> > > >> <
> > >
> >
> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
> > > >
> > > >> _______________________________________________
> > > >> dev-quality mailing list
> > > >> [hidden email]
> > > >> https://lists.mozilla.org/listinfo/dev-quality
> > > >
> > > _______________________________________________
> > > dev-quality mailing list
> > > [hidden email]
> > > https://lists.mozilla.org/listinfo/dev-quality
> > >
> > _______________________________________________
> > dev-quality mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-quality
> >
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Fwd: Test Case Management

John Dorlus
---------- Forwarded message ----------
From: John Dorlus <[hidden email]>
Date: Tue, Mar 1, 2016 at 8:53 PM
Subject: Re: Test Case Management
To: Naoki Hirata <[hidden email]>


Thinking long term, not that I have anything against Moztrap, but do we
want to go back into the cycle of maintenance. Some questions come to mind
when I think about the resurrection of Moztrap:

1.) How much work will it take to bring it up to speed both in performance
and UX?
2.) If we get it up to speed, how long will it be before we outgrow it?
3.) Could the resources that we would allocate to this work be better used?
(Seeing as how we have a ton of new initiatives going on, I think this one
answers itself.)

I know we have a ton of great products that we dogfood but seeing as how we
are struggling to keep this one pertinent, I think this may be a good time
to have someone else do the heavy lifting. That way, we just worry about
what we need to do on other projects rather than always having the support
of Moztrap on the back of our minds.


[image: photo]
*John Dorlus*
Senior Software Engineer in Test, Mozilla Corporation
m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
E. Evelyn Avenue Mountain View, CA 94041 USA
<http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus>
<http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
<http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
<http://www.linkedin.com/company/mozilla-corporation>
Get a signature like this: Click here!
<http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=832472499320283>

On Tue, Mar 1, 2016 at 8:49 PM, Naoki Hirata <[hidden email]> wrote:

> I personally don't mind moving away from moztrap.
>
> The UX wasn't geared well to someone who works in it nor was the
> performance.  I understand that there's been hard work and dedication put
> into the project and I apologize if I hurt people's feelings.  In B2G, we
> had to use a spreadsheet (for dogfood/smoke tests) and then put the results
> in Moztrap which sort of defeats the purpose if we have to use two
> different systems.  Filing bugs didn't seem to be prudent after awhile
> because there was only a limited number of developers on it.
>
> At the same time, there are people that have made improvements to the UI
> for moztrap ( see mozilluminate project:
> https://github.com/MozIlluminate/mozilluminate-demo
>
> I think if the UX was made better, there's a chance to revive Moztrap.
>
> Regards,
> Naoki
>
> On Tue, Mar 1, 2016 at 5:07 PM, Kamil Jozwiak <[hidden email]>
> wrote:
>
>> If we decide to start using TestRail rather than MozTrap, will the current
>> tests that are still being used be migrated over? Will the open source
>> license cover SV and our community? I might be wrong here, but I remember
>> that some of our community members were using the MozTrap platform during
>> test days. Is that still the case? Or have we moved away from MozTrap all
>> together?
>>
>> On Mon, Feb 29, 2016 at 5:36 PM, Stuart Philp <[hidden email]> wrote:
>>
>> > Thanks to Mr. Dorlus, we have an account set up with Gurock to give
>> > testrail a spin. If anyone is interested in playing around with it, I've
>> > set it up in our staging AWS environment. Shoot me a message and I'll
>> add
>> > you. We have 50 seats so should be plenty of room for now.
>> >
>> > Cheers
>> >
>> > On Thu, Feb 25, 2016 at 11:43 AM, John Dorlus <[hidden email]>
>> wrote:
>> >
>> > > That's all it takes to sell Dave. 😆
>> > >
>> > > J.D.
>> > >
>> > > > On Feb 25, 2016, at 5:06 AM, Dave Hunt <[hidden email]> wrote:
>> > > >
>> > > > I like that there’s a pytest plugin for it!
>> > > > https://github.com/allankilpatrick/pytest-testrail/
>> > > >
>> > > > Dave
>> > > >
>> > > >> On 24 Feb 2016, at 23:30, John Dorlus <[hidden email]> wrote:
>> > > >>
>> > > >> Greetings Mozillians,
>> > > >>
>> > > >> The quest for a test case management system has been a topic for a
>> > while
>> > > >> now amongst several teams. Moztrap is a great tool but it's
>> > maintenance
>> > > is
>> > > >> slow and we have a need now to leverage someone else's code so we
>> can
>> > > focus
>> > > >> on the new and shiny. There are many options out there and we want
>> to
>> > > find
>> > > >> one that will support our Open Sourceness. We found a tools seems
>> > > >> interesting and want to get some opinions on this or other
>> > alternatives.
>> > > >>
>> > > >> TestRail - TestRail is one of the most widely used Test Case
>> > Management
>> > > >> systems amongst top tier tech companies. Comes with all of the
>> bells
>> > and
>> > > >> whistles (including integration with Bugzilla and Github) but is
>> > > normally
>> > > >> super expensive. I found a page on the TestRail site saying they
>> give
>> > > free
>> > > >> licenses to open source projects.1
>> > > >>
>> > > >> Have any of you heard of something better? Better support? Better
>> > > >> integration? Is TestRail a good choice? Let us know your thoughts.
>> > > >>
>> > > >> 1 -
>> > >
>> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
>> > > >>
>> > > >> [image: photo]
>> > > >> *John Dorlus*
>> > > >> Senior Software Engineer in Test, Mozilla Corporation
>> > > >> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org
>> |
>> > > a:331
>> > > >> E. Evelyn Avenue Mountain View, CA 94041 USA
>> > > >> <http://www.facebook.com/mozilla>  <
>> > > http://www.linkedin.com/in/johndorlus>
>> > > >> <http://www.twitter.com/the_power_coder>  <
>> > > http://www.github.com/silne30>
>> > > >> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
>> > > >> <http://www.linkedin.com/company/mozilla-corporation>
>> > > >> Get a signature like this: Click here!
>> > > >> <
>> > >
>> >
>> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
>> > > >
>> > > >> _______________________________________________
>> > > >> dev-quality mailing list
>> > > >> [hidden email]
>> > > >> https://lists.mozilla.org/listinfo/dev-quality
>> > > >
>> > > _______________________________________________
>> > > dev-quality mailing list
>> > > [hidden email]
>> > > https://lists.mozilla.org/listinfo/dev-quality
>> > >
>> > _______________________________________________
>> > dev-quality mailing list
>> > [hidden email]
>> > https://lists.mozilla.org/listinfo/dev-quality
>> >
>> _______________________________________________
>> dev-quality mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-quality
>>
>
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Shing Lyu
First of all, I don't mind moving away from MozTrap. I've spent the past ~1
year working on MozTrap improvement and I'm not so positive about its
future given the current resource arrangement. if we can't have someone
dedicated on maintaining it, I would rather use an off-the-shelf product.

Another option is to handover the ownership to LibreOffice foundation
<http://manual-test.libreoffice.org/results/runs/> (if they accept). They
are the only active user for MozTrap other than Mozilla, and they are
currently working on L10N for MozTrap.


2016-03-02 9:54 GMT+08:00 John Dorlus <[hidden email]>:

> ---------- Forwarded message ----------
> From: John Dorlus <[hidden email]>
> Date: Tue, Mar 1, 2016 at 8:53 PM
> Subject: Re: Test Case Management
> To: Naoki Hirata <[hidden email]>
>
>
> Thinking long term, not that I have anything against Moztrap, but do we
> want to go back into the cycle of maintenance. Some questions come to mind
> when I think about the resurrection of Moztrap:
>
> 1.) How much work will it take to bring it up to speed both in performance
> and UX?
>
I've made a new UI for MozTrap: https://shinglyu.github.io/moztrap-new-ui/#/
You can see the performance improved a lot, so a new UI is definitely
feasible. It took us around 8 month (as side project) to finish a test case
managment page, a suite management page and a test result page. I believe a
complete rewrite of the frontend will take ~ 6 month for around 3 engineers
working full time.

2.) If we get it up to speed, how long will it be before we outgrow it?
>
FxOS is a special case, it has 100,000+ cases. Desktop Firefox has only
10,000+. LibreOffice has only 116. I'm not sure our future products will
have so much test cases as FxOS. To make MozTrap scale, some database
schema needs to be updated, especially the Environment part needs to be
refactored and we might want to de-normalize some tables. This will
probably take a quarter or so.


> 3.) Could the resources that we would allocate to this work be better used?
> (Seeing as how we have a ton of new initiatives going on, I think this one
> answers itself.)
>
That's true, I personally wants to see more SBTM in Mozilla. As for
scripted testing, I'm very interested in some framework/tool that can have
higher-level test case reuse. For example I want to be able to extract
certain sub routines as functions or templates and use them to build new
test cases. This can be a new direction for MozIlluminate, which Nakoi
kindly mentioned.


> I know we have a ton of great products that we dogfood but seeing as how we
> are struggling to keep this one pertinent, I think this may be a good time
> to have someone else do the heavy lifting. That way, we just worry about
> what we need to do on other projects rather than always having the support
> of Moztrap on the back of our minds.
>
>
> [image: photo]
> *John Dorlus*
> Senior Software Engineer in Test, Mozilla Corporation
> m:407.780.8031 | e:[hidden email] | w:http://www.mozilla.org | a:331
> E. Evelyn Avenue Mountain View, CA 94041 USA
> <http://www.facebook.com/mozilla>  <http://www.linkedin.com/in/johndorlus>
> <http://www.twitter.com/the_power_coder>  <http://www.github.com/silne30>
> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> <http://www.linkedin.com/company/mozilla-corporation>
> Get a signature like this: Click here!
> <
> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=832472499320283
> >
>
> On Tue, Mar 1, 2016 at 8:49 PM, Naoki Hirata <[hidden email]> wrote:
>
> > I personally don't mind moving away from moztrap.
> >
> > The UX wasn't geared well to someone who works in it nor was the
> > performance.  I understand that there's been hard work and dedication put
> > into the project and I apologize if I hurt people's feelings.  In B2G, we
> > had to use a spreadsheet (for dogfood/smoke tests) and then put the
> results
> > in Moztrap which sort of defeats the purpose if we have to use two
> > different systems.  Filing bugs didn't seem to be prudent after awhile
> > because there was only a limited number of developers on it.
> >
> > At the same time, there are people that have made improvements to the UI
> > for moztrap ( see mozilluminate project:
> > https://github.com/MozIlluminate/mozilluminate-demo
> >
> > I think if the UX was made better, there's a chance to revive Moztrap.
> >
> > Regards,
> > Naoki
> >
> > On Tue, Mar 1, 2016 at 5:07 PM, Kamil Jozwiak <[hidden email]>
> > wrote:
> >
> >> If we decide to start using TestRail rather than MozTrap, will the
> current
> >> tests that are still being used be migrated over? Will the open source
> >> license cover SV and our community? I might be wrong here, but I
> remember
> >> that some of our community members were using the MozTrap platform
> during
> >> test days. Is that still the case? Or have we moved away from MozTrap
> all
> >> together?
> >>
> >> On Mon, Feb 29, 2016 at 5:36 PM, Stuart Philp <[hidden email]>
> wrote:
> >>
> >> > Thanks to Mr. Dorlus, we have an account set up with Gurock to give
> >> > testrail a spin. If anyone is interested in playing around with it,
> I've
> >> > set it up in our staging AWS environment. Shoot me a message and I'll
> >> add
> >> > you. We have 50 seats so should be plenty of room for now.
> >> >
> >> > Cheers
> >> >
> >> > On Thu, Feb 25, 2016 at 11:43 AM, John Dorlus <[hidden email]>
> >> wrote:
> >> >
> >> > > That's all it takes to sell Dave. 😆
> >> > >
> >> > > J.D.
> >> > >
> >> > > > On Feb 25, 2016, at 5:06 AM, Dave Hunt <[hidden email]> wrote:
> >> > > >
> >> > > > I like that there’s a pytest plugin for it!
> >> > > > https://github.com/allankilpatrick/pytest-testrail/
> >> > > >
> >> > > > Dave
> >> > > >
> >> > > >> On 24 Feb 2016, at 23:30, John Dorlus <[hidden email]>
> wrote:
> >> > > >>
> >> > > >> Greetings Mozillians,
> >> > > >>
> >> > > >> The quest for a test case management system has been a topic for
> a
> >> > while
> >> > > >> now amongst several teams. Moztrap is a great tool but it's
> >> > maintenance
> >> > > is
> >> > > >> slow and we have a need now to leverage someone else's code so we
> >> can
> >> > > focus
> >> > > >> on the new and shiny. There are many options out there and we
> want
> >> to
> >> > > find
> >> > > >> one that will support our Open Sourceness. We found a tools seems
> >> > > >> interesting and want to get some opinions on this or other
> >> > alternatives.
> >> > > >>
> >> > > >> TestRail - TestRail is one of the most widely used Test Case
> >> > Management
> >> > > >> systems amongst top tier tech companies. Comes with all of the
> >> bells
> >> > and
> >> > > >> whistles (including integration with Bugzilla and Github) but is
> >> > > normally
> >> > > >> super expensive. I found a page on the TestRail site saying they
> >> give
> >> > > free
> >> > > >> licenses to open source projects.1
> >> > > >>
> >> > > >> Have any of you heard of something better? Better support? Better
> >> > > >> integration? Is TestRail a good choice? Let us know your
> thoughts.
> >> > > >>
> >> > > >> 1 -
> >> > >
> >>
> https://blog.gurock.com/free-testrail-licenses-for-open-source-projects/
> >> > > >>
> >> > > >> [image: photo]
> >> > > >> *John Dorlus*
> >> > > >> Senior Software Engineer in Test, Mozilla Corporation
> >> > > >> m:407.780.8031 | e:[hidden email] | w:
> http://www.mozilla.org
> >> |
> >> > > a:331
> >> > > >> E. Evelyn Avenue Mountain View, CA 94041 USA
> >> > > >> <http://www.facebook.com/mozilla>  <
> >> > > http://www.linkedin.com/in/johndorlus>
> >> > > >> <http://www.twitter.com/the_power_coder>  <
> >> > > http://www.github.com/silne30>
> >> > > >> <http://www.facebook.com/mozilla> <https://twitter.com/mozilla>
> >> > > >> <http://www.linkedin.com/company/mozilla-corporation>
> >> > > >> Get a signature like this: Click here!
> >> > > >> <
> >> > >
> >> >
> >>
> http://ws-promos.appspot.com/r?rdata=eyJydXJsIjogImh0dHA6Ly93d3cud2lzZXN0YW1wLmNvbS9lbWFpbC1pbnN0YWxsP3dzX25jaWQ9NjcyMjk0MDA4JnV0bV9zb3VyY2U9ZXh0ZW5zaW9uJnV0bV9tZWRpdW09ZW1haWwmdXRtX2NhbXBhaWduPXByb21vXzU3MzI1Njg1NDg3Njk3OTIiLCAiZSI6ICI1NzMyNTY4NTQ4NzY5NzkyIn0=&u=129278079350478
> >> > > >
> >> > > >> _______________________________________________
> >> > > >> dev-quality mailing list
> >> > > >> [hidden email]
> >> > > >> https://lists.mozilla.org/listinfo/dev-quality
> >> > > >
> >> > > _______________________________________________
> >> > > dev-quality mailing list
> >> > > [hidden email]
> >> > > https://lists.mozilla.org/listinfo/dev-quality
> >> > >
> >> > _______________________________________________
> >> > dev-quality mailing list
> >> > [hidden email]
> >> > https://lists.mozilla.org/listinfo/dev-quality
> >> >
> >> _______________________________________________
> >> dev-quality mailing list
> >> [hidden email]
> >> https://lists.mozilla.org/listinfo/dev-quality
> >>
> >
> >
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>



--
Shing Lyu
QA, Mozilla Taipei
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Andreas Tolfsen
First let me say that I’m not entirely sure what a “test management
system” is or how one works, but I will begin by acknowledging that
recording tests and their results is a difficult problem.

On 2 March 2016 at 06:29, Shing Lyu <[hidden email]> wrote:
> 2.) If we get it up to speed, how long will it be before we outgrow it?
>
> FxOS is a special case, it has 100,000+ cases. Desktop Firefox has only
> 10,000+. LibreOffice has only 116. I'm not sure our future products will
> have so much test cases as FxOS. To make MozTrap scale, some database
> schema needs to be updated, especially the Environment part needs to be
> refactored and we might want to de-normalize some tables. This will
> probably take a quarter or so.

The definition of “test” is important in this context.  In some
situations an assertion is an appropriate measure to count as a test,
and in others a file would a more suitable metric to use.

I feel like pointing out that the style of testing Firefox OS employs
is radically different from many of the tests written for the web
platform, for Gecko internals, or for testing the Firefox UI.
Comparing how many tests Firefox OS has to Firefox is a pointless
exercise, one not totally unlike comparing apples and oranges.

Some quick and unscientific digging (that should absolutely be taken
with a grain of salt) reveals that central has approximately ~27,000
test files defined in manifests it runs on try.  These files may
contain one or more assertions, making it an extremely coarse metric.
Last week these tests yielded roughly 300,000,000 results.

To illustrate my point, the Web Platform Tests have around 4,000
“parent tests” that are roughly equivalent to a file, but whereas
testharness.js tests may contain multiple test functions “child tests”
(again containing zero or more assertions), reference tests typically
only have a single child, skewing the ratio of parent tests per
testharness- and reference test.  This is substantiated by the fact
that they combined produce between ~250-300,000 child tests.

To add further complexity to this, many tests are internally data- or
table driven, or are generated.  This is the case for many mochitests,
and because of the special way they are written this normally does not
surface to the level of test reporting.

With this I hope I’ve demonstrated that normalising what “a test”
means in a heterogeneous test environment is rather meaningless, to
the point it only causes confusion.

To speak briefly towards Firefox and Gecko, we don’t have a central
database of tests or test results.  We record structured logs in the
Active Data, which is a map-reduce system backed by Elastic Search and
Amazon S3.  It allows us to run meaningful queries for a lot of
things, but is obviously limited by some of the peculiarities of our
test types and reporting.

I won’t go into the discussion about test management, but my suspicion
is that you will encounter tests in the Mozilla codebase that are
difficult – if not impossible – to normalise into a rigid system for a
wide variety of different reasons; only some of which I’ve highlighted
above.
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Cameron Dawson
I am technically the owner of MozTrap, but have had little to no time to maintain it in the last few years.  And I can’t continue to own it.  I’m 100% allocated to other projects.  In fact, my manager and I have been talking about trying to find a “real” owner for it.  MozTrap has lots of scalability and usability issues with it.  It needs attention, but currently there is no one to give it that attention.  For those teams that would like to continue using MozTrap, we will need to decide who will become the new development owner.

That being said, if users wanted to move to an off-the-shelf option, I would strongly encourage that.  TestRail looks great, and it integrates with Bugzilla.  Of course, the hardest part of moving to a new product would be data migration.  But there are several ways to get data out of MozTrap.  You can get it either as JSON, or via a DB query, etc.  :)

If everyone moves to products other than MozTrap, then we can just decom our servers and leave the project for use as open source.  Or perhaps give ownership to LibreOffice, if they like.

-Cam

> On Mar 2, 2016, at 8:11 AM, Andreas Tolfsen <[hidden email]> wrote:
>
> First let me say that I’m not entirely sure what a “test management
> system” is or how one works, but I will begin by acknowledging that
> recording tests and their results is a difficult problem.
>
> On 2 March 2016 at 06:29, Shing Lyu <[hidden email]> wrote:
>> 2.) If we get it up to speed, how long will it be before we outgrow it?
>>
>> FxOS is a special case, it has 100,000+ cases. Desktop Firefox has only
>> 10,000+. LibreOffice has only 116. I'm not sure our future products will
>> have so much test cases as FxOS. To make MozTrap scale, some database
>> schema needs to be updated, especially the Environment part needs to be
>> refactored and we might want to de-normalize some tables. This will
>> probably take a quarter or so.
>
> The definition of “test” is important in this context.  In some
> situations an assertion is an appropriate measure to count as a test,
> and in others a file would a more suitable metric to use.
>
> I feel like pointing out that the style of testing Firefox OS employs
> is radically different from many of the tests written for the web
> platform, for Gecko internals, or for testing the Firefox UI.
> Comparing how many tests Firefox OS has to Firefox is a pointless
> exercise, one not totally unlike comparing apples and oranges.
>
> Some quick and unscientific digging (that should absolutely be taken
> with a grain of salt) reveals that central has approximately ~27,000
> test files defined in manifests it runs on try.  These files may
> contain one or more assertions, making it an extremely coarse metric.
> Last week these tests yielded roughly 300,000,000 results.
>
> To illustrate my point, the Web Platform Tests have around 4,000
> “parent tests” that are roughly equivalent to a file, but whereas
> testharness.js tests may contain multiple test functions “child tests”
> (again containing zero or more assertions), reference tests typically
> only have a single child, skewing the ratio of parent tests per
> testharness- and reference test.  This is substantiated by the fact
> that they combined produce between ~250-300,000 child tests.
>
> To add further complexity to this, many tests are internally data- or
> table driven, or are generated.  This is the case for many mochitests,
> and because of the special way they are written this normally does not
> surface to the level of test reporting.
>
> With this I hope I’ve demonstrated that normalising what “a test”
> means in a heterogeneous test environment is rather meaningless, to
> the point it only causes confusion.
>
> To speak briefly towards Firefox and Gecko, we don’t have a central
> database of tests or test results.  We record structured logs in the
> Active Data, which is a map-reduce system backed by Elastic Search and
> Amazon S3.  It allows us to run meaningful queries for a lot of
> things, but is obviously limited by some of the peculiarities of our
> test types and reporting.
>
> I won’t go into the discussion about test management, but my suspicion
> is that you will encounter tests in the Mozilla codebase that are
> difficult – if not impossible – to normalise into a rigid system for a
> wide variety of different reasons; only some of which I’ve highlighted
> above.
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality

_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: Test Case Management

Shing Lyu
@Cameron,

I feel quite connected to the project, but our team's resource plan doesn't
allow me to share ownership of it.

@Andreas & Florin,

I believe we are talking about GUI-level manual testing on MozTrap (see:
https://moztrap.mozilla.org/manage/cases/), so nothing to do with
automation and all the Gecko internal tests.

Shing

2016-03-03 9:15 GMT+08:00 Cameron Dawson <[hidden email]>:

> I am technically the owner of MozTrap, but have had little to no time to
> maintain it in the last few years.  And I can’t continue to own it.  I’m
> 100% allocated to other projects.  In fact, my manager and I have been
> talking about trying to find a “real” owner for it.  MozTrap has lots of
> scalability and usability issues with it.  It needs attention, but
> currently there is no one to give it that attention.  For those teams that
> would like to continue using MozTrap, we will need to decide who will
> become the new development owner.
>
> That being said, if users wanted to move to an off-the-shelf option, I
> would strongly encourage that.  TestRail looks great, and it integrates
> with Bugzilla.  Of course, the hardest part of moving to a new product
> would be data migration.  But there are several ways to get data out of
> MozTrap.  You can get it either as JSON, or via a DB query, etc.  :)
>
> If everyone moves to products other than MozTrap, then we can just decom
> our servers and leave the project for use as open source.  Or perhaps give
> ownership to LibreOffice, if they like.
>
> -Cam
>
> > On Mar 2, 2016, at 8:11 AM, Andreas Tolfsen <[hidden email]> wrote:
> >
> > First let me say that I’m not entirely sure what a “test management
> > system” is or how one works, but I will begin by acknowledging that
> > recording tests and their results is a difficult problem.
> >
> > On 2 March 2016 at 06:29, Shing Lyu <[hidden email]> wrote:
> >> 2.) If we get it up to speed, how long will it be before we outgrow it?
> >>
> >> FxOS is a special case, it has 100,000+ cases. Desktop Firefox has only
> >> 10,000+. LibreOffice has only 116. I'm not sure our future products will
> >> have so much test cases as FxOS. To make MozTrap scale, some database
> >> schema needs to be updated, especially the Environment part needs to be
> >> refactored and we might want to de-normalize some tables. This will
> >> probably take a quarter or so.
> >
> > The definition of “test” is important in this context.  In some
> > situations an assertion is an appropriate measure to count as a test,
> > and in others a file would a more suitable metric to use.
> >
> > I feel like pointing out that the style of testing Firefox OS employs
> > is radically different from many of the tests written for the web
> > platform, for Gecko internals, or for testing the Firefox UI.
> > Comparing how many tests Firefox OS has to Firefox is a pointless
> > exercise, one not totally unlike comparing apples and oranges.
> >
> > Some quick and unscientific digging (that should absolutely be taken
> > with a grain of salt) reveals that central has approximately ~27,000
> > test files defined in manifests it runs on try.  These files may
> > contain one or more assertions, making it an extremely coarse metric.
> > Last week these tests yielded roughly 300,000,000 results.
> >
> > To illustrate my point, the Web Platform Tests have around 4,000
> > “parent tests” that are roughly equivalent to a file, but whereas
> > testharness.js tests may contain multiple test functions “child tests”
> > (again containing zero or more assertions), reference tests typically
> > only have a single child, skewing the ratio of parent tests per
> > testharness- and reference test.  This is substantiated by the fact
> > that they combined produce between ~250-300,000 child tests.
> >
> > To add further complexity to this, many tests are internally data- or
> > table driven, or are generated.  This is the case for many mochitests,
> > and because of the special way they are written this normally does not
> > surface to the level of test reporting.
> >
> > With this I hope I’ve demonstrated that normalising what “a test”
> > means in a heterogeneous test environment is rather meaningless, to
> > the point it only causes confusion.
> >
> > To speak briefly towards Firefox and Gecko, we don’t have a central
> > database of tests or test results.  We record structured logs in the
> > Active Data, which is a map-reduce system backed by Elastic Search and
> > Amazon S3.  It allows us to run meaningful queries for a lot of
> > things, but is obviously limited by some of the peculiarities of our
> > test types and reporting.
> >
> > I won’t go into the discussion about test management, but my suspicion
> > is that you will encounter tests in the Mozilla codebase that are
> > difficult – if not impossible – to normalise into a rigid system for a
> > wide variety of different reasons; only some of which I’ve highlighted
> > above.
> > _______________________________________________
> > dev-quality mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-quality
>
>


--
Shing Lyu
QA, Mozilla Taipei
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality