dealing with very big (upto 1Gb) svg file

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

dealing with very big (upto 1Gb) svg file

meemsheen
This post was updated on .
Hi,

I have to visualize some values from a big text file with svg rect and with different colors,
but the svg file I generate is so big in size that firefox/safari crashes to open it.

Description about the input text file:

Its a comma separated csv text file which contains float values like:
0.70, 0.50,0.98,0.78,0.61,0.88.........
160 of the above values in one line and more than 100,000 lines.

I have a c script which parse the csv file and writes <rect> tags in svg and with style=fill: rgb( , , ) values filled from the input csv file.

If I just parse some 1000 lines from the input csv file the generated svg is about 50-100MB and I can
view it as I wanted, but when I parse more and more, the generated svg becomes very big and
safari/firefox hangs to open this big svg file.
I have 4GB RAM in my machine.

Any help to deal with this problem or do it in another better way will be much appreciated.

Thanks,

Best regards,

--
meemsheen

Reply | Threaded
Open this post in threaded view
|

Re: dealing with very big (upto 1Gb) svg file

Stanimir Stamenkov-3
Sun, 14 Feb 2010 04:36:26 -0800 (PST), /meemsheen/:

> (...)
> If I just parse some 1000 lines from the input csv file the generated svg is
> about 50-100MB and I can
> view it as I wanted, but when I parse more and more, the generated svg
> becomes very big and
> safari/firefox hangs to open this big svg file.
> I have 4GB RAM in my machine.
>
> Any help to deal with this problem or do it in another better way will be
> much appreciated.

You may ensure you don't use formatting white space in the generated
SVG text which will make the built DOM much smaller, I guess.

If the data is really that big, may be generate and serve a bitmap
image, instead?  Or if it happens on the client side - one could
draw directly on a canvas:

https://developer.mozilla.org/en/HTML/Canvas

--
Stanimir
_______________________________________________
dev-tech-svg mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-svg
Reply | Threaded
Open this post in threaded view
|

Re: dealing with very big (upto 1Gb) svg file

Jeff Schiller
In reply to this post by meemsheen
On Feb 14, 6:36 am, meemsheen <[hidden email]> wrote:

> Hi,
>
> I have to visualize some values from a big text file with svg rect and with
> different colors,
> but the svg file I generate is so big in size that firefox/safari crashes to
> open it.
>
> Description about the input text file:
>
> Its a comma separated csv text file which contains float values like:
> 0.70, 0.50,0.98,0.78,0.61,0.88.........
> 160 of the above values in one line and more than 100,000 lines.
>
> I have a c script which parse the csv file and writes <rect> tags in svg and
> with style=fill: rgb( , , ) values filled from the input csv file.
>
> If I just parse some 1000 lines from the input csv file the generated svg is
> about 50-100MB and I can
> view it as I wanted, but when I parse more and more, the generated svg
> becomes very big and
> safari/firefox hangs to open this big svg file.
> I have 4GB RAM in my machine.
>
> Any help to deal with this problem or do it in another better way will be
> much appreciated.
>

Is it required to turn everything into rectangles and each one having
a different color?  The browsers may be choking on the number of
elements (rects) since each element adds DOM overhead (memory usage).

Have you looked at parsing the data into SVG path commands?  Using
just one path (with subpaths) for each color would drastically cut
down on memory usage in the browser.

<path fill="blue" d="M100,100 L200,200 z"/>

This would be a path that moves the pen to (100,100) then draws a line
to (200,200), then closes the path.

Jeff
_______________________________________________
dev-tech-svg mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-svg
Reply | Threaded
Open this post in threaded view
|

Re: dealing with very big (upto 1Gb) svg file

meemsheen

Jeff Schiller wrote
On Feb 14, 6:36 am, meemsheen <meemsh...@yahoo.com> wrote:

> Any help to deal with this problem or do it in another better way will be
> much appreciated.
>

Is it required to turn everything into rectangles and each one having
a different color?  The browsers may be choking on the number of
elements (rects) since each element adds DOM overhead (memory usage).

Have you looked at parsing the data into SVG path commands?  Using
just one path (with subpaths) for each color would drastically cut
down on memory usage in the browser.

<path fill="blue" d="M100,100 L200,200 z"/>

This would be a path that moves the pen to (100,100) then draws a line
to (200,200), then closes the path.
No, its not required to turn everything into rects, but the different colors are important,
its just similar to visualize such a data with a "heatmap"...but a java program which I've for that
also gives me 'out of mem error', and I also can't use utilities like matrix2png...
 
Your suggestion of using Path and subpaths looks interesting, that it will cut down the load,
but can I use different fill colors for subpaths? (i.e red color to values from 0.50 to green color for values upto 0.99)

--
meemsheen
Reply | Threaded
Open this post in threaded view
|

Re: dealing with very big (upto 1Gb) svg file

Jeff Schiller
In reply to this post by Jeff Schiller
On Feb 14, 2:27 pm, meemsheen <[hidden email]> wrote:
>
> Your suggestion of using Path and subpaths looks interesting, that it will
> cut down the load,
> but can I use different fill colors for subpaths? (i.e red color to values
> from 0.50 to green color for values upto 0.99)
>

No, subpaths are all painted with the same paint of the path element.
However, if you take some time to sort things into the same colors,
then you could group all same-colors into one path.

Jeff
_______________________________________________
dev-tech-svg mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-svg
Reply | Threaded
Open this post in threaded view
|

Re: dealing with very big (upto 1Gb) svg file

Helder Magalhães

> > but can I use different fill colors for subpaths? (i.e red color to values
> > from 0.50 to green color for values upto 0.99)
>
> No, subpaths are all painted with the same paint of the path element.
> However, if you take some time to sort things into the same colors,
> then you could group all same-colors into one path.

Along with the whitespace optimizations already described, I'd also
give emphasis towards property sharing. This will reduce memory and
speed up loading. So, instead of:

  <rect fill="red" stroke="none" x="10" width="10" height="10"/>
  <rect fill="green" stroke="none" x="20" width="10" height="10"/>
  <rect fill="blue" stroke="none" x="30" width="10" height="10"/>

One could simply write:

  <g fill="red" stroke="none">
    <rect stroke="none" x="10" width="10" height="10"/>
    <rect fill="green" x="20" width="10" height="10" />
    <rect fill="blue" x="30" width="10" height="10"/>
  </g>

This works pretty well if you are able to identify a set of shared
property values. The gains become more obvious as the number of child
elements increase.

Also, you may find CSS interesting for avoiding repetitive style-
related attributes: just create a class for them! :-)

Finally, you may also use XML entities for repetitive text, if you can
identify such patterns. ;-)


Hope this helps,
 Helder
_______________________________________________
dev-tech-svg mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-svg