automated optical regression tests

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

automated optical regression tests

Svante Schubert
Hi,

I am in need of a way to trigger Firefox for printing a HTML document.
This is required for an automated optical regression test of an
OpenOffice.org XHTML export filter.

The general approach might be of interest for you as well. Using a
similar automated testing approach for your Firefox application.
Although our test tool is not yet available/supported, it is planned to
be open sourced.

Basically what we do, we print to a postscript file, change it to JPEG
and compare this JPEG with an earlier positive version, we know was o.k.

By doing a pixel based XOR comparison, we are producing an black/white
image for the tester, showing all differences white, the rest black.

Is there a way of doing a Firefox print from an external program or do I
have to add some JavaScript in the XHTML to enable this?

Regards,
Svante

--
Svante Schubert <[hidden email]>             Sun Microsystems
Software Engineer - StarOffice                           Sachsenfeld 4
Phone:  +49 40 23646 965                               D-20097 Hamburg
Fax:    +49 40 23646 550                 http://www.sun.com/staroffice
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: automated optical regression tests

Axel Hecht
Svante Schubert wrote:

> Hi,
>
> I am in need of a way to trigger Firefox for printing a HTML document.
> This is required for an automated optical regression test of an
> OpenOffice.org XHTML export filter.
>
> The general approach might be of interest for you as well. Using a
> similar automated testing approach for your Firefox application.
> Although our test tool is not yet available/supported, it is planned to
> be open sourced.
>
> Basically what we do, we print to a postscript file, change it to JPEG
> and compare this JPEG with an earlier positive version, we know was o.k.
>
> By doing a pixel based XOR comparison, we are producing an black/white
> image for the tester, showing all differences white, the rest black.
>
> Is there a way of doing a Firefox print from an external program or do I
> have to add some JavaScript in the XHTML to enable this?
>

I know that we're looking into this, too. As long as you're exporting to
  XML (ie a file with a .xml ending), you might be ok, even.
For HTML, we have long standing bugs about layout not being
deterministic, which may cause false positives.

I bet that our approach would be based on canvas, I think that canvas
was enhanced to do something like that in particular. But that's just a
shady memory.

davel would know, I guess.

Axel
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: automated optical regression tests

Svante Schubert
Axel Hecht wrote:

> Svante Schubert wrote:
>> Hi,
>>
>> I am in need of a way to trigger Firefox for printing a HTML document.
>> This is required for an automated optical regression test of an
>> OpenOffice.org XHTML export filter.
>>
>> The general approach might be of interest for you as well. Using a
>> similar automated testing approach for your Firefox application.
>> Although our test tool is not yet available/supported, it is planned to
>> be open sourced.
>>
>> Basically what we do, we print to a postscript file, change it to JPEG
>> and compare this JPEG with an earlier positive version, we know was o.k.
>>
>> By doing a pixel based XOR comparison, we are producing an black/white
>> image for the tester, showing all differences white, the rest black.
>>
>> Is there a way of doing a Firefox print from an external program or do I
>> have to add some JavaScript in the XHTML to enable this?
>>
>
> I know that we're looking into this, too. As long as you're exporting to
>  XML (ie a file with a .xml ending), you might be ok, even.
> For HTML, we have long standing bugs about layout not being
> deterministic, which may cause false positives.
>
> I bet that our approach would be based on canvas, I think that canvas
> was enhanced to do something like that in particular. But that's just a
> shady memory.
>
> davel would know, I guess.
>
> Axel

Have you got further information on this, Davel?

Kind Regards,
Svante
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: automated optical regression tests

Adam Guthrie
In reply to this post by Svante Schubert
Svante Schubert wrote:

> Hi,
>
> I am in need of a way to trigger Firefox for printing a HTML document.
> This is required for an automated optical regression test of an
> OpenOffice.org XHTML export filter.
>
> The general approach might be of interest for you as well. Using a
> similar automated testing approach for your Firefox application.
> Although our test tool is not yet available/supported, it is planned to
> be open sourced.
>
> Basically what we do, we print to a postscript file, change it to JPEG
> and compare this JPEG with an earlier positive version, we know was o.k.
>
> By doing a pixel based XOR comparison, we are producing an black/white
> image for the tester, showing all differences white, the rest black.
>
> Is there a way of doing a Firefox print from an external program or do I
> have to add some JavaScript in the XHTML to enable this?

Martijn Wargers, Michael Wu, and myself have been working on
making a tool that would allow for visual regression testing.

Michael wrote an extension that loads URLs specified on the
command line, takes screenshots of those and writes them to
files. It's still sort of in a primitive state; for
instance, we don't have a way to diff the files yet, but
this could probably be done using Robert O'Callahan's Perl
script [0].

I can e-mail the extension to you personally if you'd like.

-Adam
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: automated optical regression tests

Svante Schubert
Adam Guthrie wrote:

> Svante Schubert wrote:
>  
>> Hi,
>>
>> I am in need of a way to trigger Firefox for printing a HTML document.
>> This is required for an automated optical regression test of an
>> OpenOffice.org XHTML export filter.
>>
>> The general approach might be of interest for you as well. Using a
>> similar automated testing approach for your Firefox application.
>> Although our test tool is not yet available/supported, it is planned to
>> be open sourced.
>>
>> Basically what we do, we print to a postscript file, change it to JPEG
>> and compare this JPEG with an earlier positive version, we know was o.k.
>>
>> By doing a pixel based XOR comparison, we are producing an black/white
>> image for the tester, showing all differences white, the rest black.
>>
>> Is there a way of doing a Firefox print from an external program or do I
>> have to add some JavaScript in the XHTML to enable this?
>>    
>
> Martijn Wargers, Michael Wu, and myself have been working on
> making a tool that would allow for visual regression testing.
>
> Michael wrote an extension that loads URLs specified on the
> command line, takes screenshots of those and writes them to
> files. It's still sort of in a primitive state; for
> instance, we don't have a way to diff the files yet, but
> this could probably be done using Robert O'Callahan's Perl
> script [0].
>
> I can e-mail the extension to you personally if you'd like.
>
> -Adam
Adam,

that extension you guys made sounds very promising.
Could you please sent me a link to the resources, so I can see it in
action for myself?
Are these saved files postscript files? If so, we might transform them
to graphics and do a visual pixel diff.
And finally have you got pointer to Robert O'Callahan's Perl script you
mentioned?

Thanks in advance,
Svante
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: automated optical regression tests

Martijn-4
On 7/4/06, Svante Schubert <[hidden email]> wrote:

> Adam Guthrie wrote:
> > Svante Schubert wrote:
> >
> >> Hi,
> >>
> >> I am in need of a way to trigger Firefox for printing a HTML document.
> >> This is required for an automated optical regression test of an
> >> OpenOffice.org XHTML export filter.
> >>
> >> The general approach might be of interest for you as well. Using a
> >> similar automated testing approach for your Firefox application.
> >> Although our test tool is not yet available/supported, it is planned to
> >> be open sourced.
> >>
> >> Basically what we do, we print to a postscript file, change it to JPEG
> >> and compare this JPEG with an earlier positive version, we know was o.k.
> >>
> >> By doing a pixel based XOR comparison, we are producing an black/white
> >> image for the tester, showing all differences white, the rest black.
> >>
> >> Is there a way of doing a Firefox print from an external program or do I
> >> have to add some JavaScript in the XHTML to enable this?
> >>
> >
> > Martijn Wargers, Michael Wu, and myself have been working on
> > making a tool that would allow for visual regression testing.
> >
> > Michael wrote an extension that loads URLs specified on the
> > command line, takes screenshots of those and writes them to
> > files. It's still sort of in a primitive state; for
> > instance, we don't have a way to diff the files yet, but
> > this could probably be done using Robert O'Callahan's Perl
> > script [0].
> >
> > I can e-mail the extension to you personally if you'd like.
> >
> > -Adam
> Adam,
>
> that extension you guys made sounds very promising.
> Could you please sent me a link to the resources, so I can see it in
> action for myself?
> Are these saved files postscript files? If so, we might transform them
> to graphics and do a visual pixel diff.
> And finally have you got pointer to Robert O'Callahan's Perl script you
> mentioned?

See here:
http://aluminum.sourmilk.net/autosnapshot.xpi
Usage:
firefox.exe. -snapshot http://google.com

This is the bug where Robert O'Callahan has stored his perl script for
the visual pixel diff.
https://bugzilla.mozilla.org/show_bug.cgi?id=319640

Regards,
Martijn
> Thanks in advance,
> Svante
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: automated optical regression tests

Rob Sayre-2
In reply to this post by Adam Guthrie
Adam Guthrie wrote:
>
> Martijn Wargers, Michael Wu, and myself have been working on
> making a tool that would allow for visual regression testing.

Cool. I have been toying with something similar. Is it to be web-based?
What are the plans to allow it to scale to a very large corpus?

> Michael wrote an extension that loads URLs specified on the
> command line, takes screenshots of those and writes them to
> files. It's still sort of in a primitive state; for
> instance, we don't have a way to diff the files yet, but
> this could probably be done using Robert O'Callahan's Perl
> script [0].

The Python Imaging Library has lots of good stuff for this:
<http://www.pythonware.com/products/pil/>

Image diff scripts:
<http://franklinmint.fm/2006/07/04/imagediff.html>

-Rob
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: automated optical regression tests

Martijn-4
Hi Robert,

On 7/5/06, Robert Sayre <[hidden email]> wrote:
> Adam Guthrie wrote:
> >
> > Martijn Wargers, Michael Wu, and myself have been working on
> > making a tool that would allow for visual regression testing.
>
> Cool. I have been toying with something similar. Is it to be web-based?
> What are the plans to allow it to scale to a very large corpus?

The tool is an extension, which Michael Wu made, it is very basic yet.
It uses canvas methods to create the image and some xpcom to write the
image to disk.
A web-base tool might work very well, but in that case the testcases
need to be selected on one server, since canvas methods only can draw
stuff with the same domain.
There are no plans yet to scale to a very large corpus.

> > Michael wrote an extension that loads URLs specified on the
> > command line, takes screenshots of those and writes them to
> > files. It's still sort of in a primitive state; for
> > instance, we don't have a way to diff the files yet, but
> > this could probably be done using Robert O'Callahan's Perl
> > script [0].
>
> The Python Imaging Library has lots of good stuff for this:
> <http://www.pythonware.com/products/pil/>
>
> Image diff scripts:
> <http://franklinmint.fm/2006/07/04/imagediff.html>

Thanks for those links, Robert.
If you have any further ideas/suggestions about this, I'll be glad to
hear about them.

Regards,
Martijn

> -Rob
> _______________________________________________
> dev-quality mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-quality
>
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality
Reply | Threaded
Open this post in threaded view
|

Re: automated optical regression tests

Rob Sayre-2
In reply to this post by Rob Sayre-2
Martijn wrote:
>
> The tool is an extension, which Michael Wu made, it is very basic yet.
> It uses canvas methods to create the image and some xpcom to write the
> image to disk.

Sounds good. Does it only take command line args, or can it be
controlled through jssh or another remote mechanism? Re-using running
browsers would be good.

> A web-base tool might work very well, but in that case the testcases
> need to be selected on one server, since canvas methods only can draw
> stuff with the same domain.

I don't think that's a problem, since we would want the tests to have
consistent input. Some web archiving would be needed.

> There are no plans yet to scale to a very large corpus.

I wonder if we can get a passworded extension to trigger a web-archiver
that adds a test case to the server. Then we monitor visual changes.
Large deltas go to a Mechanical Turk who picks from three radio buttons:
before/after/dontknow.

-Rob
_______________________________________________
dev-quality mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-quality