Charts II: NGA - 2700 Charts

[cagney]

The first "Chart" thread was started to discuss and convert all the NZ carts.
Now, through this thread -http://www.cruisersforum.com/forums/...tml#post718487, I have found out that there are ~2700 chart pictures, world wide available on the net. The pictures are in tiles stitched together, to one flash picture, something like a new version of the old hdr files.
Scripts and websites are available to "dezoomify" the pictures on the NOAA/NGA website and download the result as a jpg or png picture.
These pictures are rather large and need to be made smaller.
Then it's just the normal georeferencing etc to make a kap file.
The pictures are "pixelperfect" and a pleasure to work with.
Here is the central website: NOAA's On-Line Chart Viewer
The directories behind the website: Index of /NGAViewer
A Pythonscript that downloads a stitched together Jpg: Help:Zoomable images/dezoomify.py - Wikimedia Commons
A Ruby script to do the same thing: https://gist.github.com/770310
A webscript in html/js/php:https://gist.github.com/770310
A website to do the job:Dezoomify. This site produces a png files.
The format of the links to feed both scripts and the web site is similar to this:

Chart 57440

All this is a very slow process screaming for a cluster of linux boxes.
When the ruby script came to this

Stitching 56 x 76 = 4256 tiles...

my load average was around 13 and the box was totally unresponsive for a long time. Be warned!

Let us together work on this. First the picture wizards ... on how to make these pictures small enough and good enough ...please.

Thanks to NGA for the web site. A worldwide coverage of freely available charts is now within reach. If only BA could take the hint from NOAA, NGA, LINZ and the Brazilian Hydrographer........

Thomas

[brak]

Nice, I didn't realize they had old Defense Agency charts there.

It would not be too difficult to automate download and restitching.
A few notes:
1. I would suggest doing this in two parts - build a download script and locate a repository with enough space, then get the data.

2. Stitch it separately. It's a lengthy process.

3. Georeference (that's a hell of a job)

Step 1 then would go a hell of a lot faster.

[dacust]

Well, I was going to try this.

First I gave the Python script a try:

• Download Python (2.7).
• Install Python
• Download script.
• Script says it needs PIL (Python Image Library).
• Download PIL
• Install Pil
• PIL says it needs registered version of Python.
• Download script to register Python.
• Registration script blows up.
• Give up.

Back to this thread to try the ruby script.

• Notice the link you gave for the ruby script is actually the url for the webscript

OK, I'm going to try the webscript, now...

EDIT: Found Ruby script at https://gist.github.com/59636

-dan

[allanpeda]

I installed Cygwin, Strawberry Perl, and ImageMagick. I used

wget -LNr http://www.charts.noaa.gov/NGAViewer/25609/

to snag all the tile images before placing them in a single directory. Most interesting are the arguments passed to ImageMagick.

Stitcher I write is linked below:

http://goo.gl/4ZPX8

[dougwm10]

Hi

I have had a look at the sample charts available and consider that they are an extremely valuable source for charts and if they can be referenced and made to run under Opencpn then we would all have access to a world wide chart database.

I am not sufficiently computer adept to assist in this conversion task / project but would heartily support anyone who feels they can get involved.

Hopefully this can be organized in a methodical manner so that duplication of effort is avoided and an ongoing database of finished charts is available for individuals to download.

[brak]

There is an issue in these charts that will make building good quality BSB challenging.

The charts appear to be scanned directly from paper charts. This is different from specially crafted images used in raster charts. Consider the tile (zoomed in) below.

What should be a uniform color, is actually a collection of "paint pixels" - artifact of offset printing.
BSB is best encoded from data that has large uniform color surfaces, that can be compressed in RLE. This data would result in terrible RLE - larger than original size.

They need some sort of color/graphic processing to turn pixelated color areas into solid colors.

[allanpeda]

Well, it's certainly more work than it would seem at first blush, but there are projects to assemble street maps from freely available data, (OpenStreetMap) so why not nautical charts?

Yes, I know, the stakes are higher at sea.

[cagney]

Originally Posted by brak

There is an issue in these charts that will make building good quality BSB challenging.

The charts appear to be scanned directly from paper charts. This is different from specially crafted images used in raster charts. Consider the tile (zoomed in) below.

What should be a uniform color, is actually a collection of "paint pixels" - artifact of offset printing.
BSB is best encoded from data that has large uniform color surfaces, that can be compressed in RLE. This data would result in terrible RLE - larger than original size.

They need some sort of color/graphic processing to turn pixelated color areas into solid colors.

Brak

Where did you get that picture tile from?
A quick, look around, and I can't find anything that bad.
What I find looks more like this

from chart 57471. This tile was downloaded with wget.
I suppose the individual charts can have different origins.

Thomas

[cagney]

What is an acceptable chart quality.
Is this good enough?
Starting with a 20.1 MiB chart file, ending with a 6.5 MiB kap file.
This was the process after downloading.

$convert 26342.jpg -rotate -90 out.jpg

The chart was skewed 90 degrees.

$
convert 26342.jpg -resize 35% -depth 4 26342_s.jpg

50% leads to a much bigger size, 25% usable but poor quality.
Test the chart:http://opencpn.avnisoft.co.za/NGA/26342.kap.zip

Thomas

[allanpeda]

Originally Posted by cagney

What is an acceptable chart quality.
Is this good enough?
Starting with a 20.1 MiB chart file, ending with a 6.5 MiB kap file.
This was the process after downloading.

The chart was skewed 90 degrees.

50% leads to a much bigger size, 25% usable but poor quality.
Test the chart:http://opencpn.avnisoft.co.za/NGA/26342.kap.zip

Thomas

Atleast for the NOAA charts I do two passes, first into into strips (kept in native Imagemagick format), then I stitch the strips together:

convert f1.miff f2.miff f3.miff +append -quality 0 +dither -colors 32 -depth 8 -density 600x600 -units pixelsperinch outputfile.png

No resampling, and as suggested by brak, I keep original resolution, but I reduce colors and convert from RGB to indexed. What was originally 50mb, drops to under 6mb.

[brak]

Originally Posted by cagney

Brak

Where did you get that picture tile from?
A quick, look around, and I can't find anything that bad.
What I find looks more like this
Attachment 29138
from chart 57471. This tile was downloaded with wget.
I suppose the individual charts can have different origins.

Thomas

Yes, they do differ - so far I find that about half the charts are paper-scans like the one i posted above, out of a sample of about 100 so far.

Others are "digitally generated" (like modern BSB charts that keep the look of paper charts but appear to be actually rendered as images), like your example.

These pixelated scanned charts would take some serious image processing.

Curiously, this pixelation is such that a PNG image of a reassembled chart had a size of 180Mb whereas straight BMP - only about 60Mb It's a "perfect noise". RLE-encoded BSB chart would be even bigger.

In terms of size, these images appear to be the same size (and DPI) as standard NOAA raster charts. I.e. - they probably should not be resized much.

[brak]

Originally Posted by cagney

What is an acceptable chart quality.
Is this good enough?
Starting with a 20.1 MiB chart file, ending with a 6.5 MiB kap file.
This was the process after downloading.

The chart was skewed 90 degrees.

50% leads to a much bigger size, 25% usable but poor quality.
Test the chart:http://opencpn.avnisoft.co.za/NGA/26342.kap.zip

Thomas

Lots of jpeg artifacts, they are not too visible in original colors, but say in night colors (autoconverted) you can see them more clearly.

This is what's affecting the size - they are not easily compressible and also make palette much larger. This is a result of original files being jpeg (I wish they used png and lossless compression).

See attached.

This would require noise filtering, preferably something that understands about raster charts (I.e. able to come back to smooth surfaces with a small number of colors)

[jonasaberg]

Originally Posted by brak

Lots of jpeg artifacts ... This is a result of original files being jpeg (I wish they used png and lossless compression ).
This would require noise filtering, preferably something that understands about raster charts (I.e. able to come back to smooth surfaces with a small number of colors)

Couldn't agree more.
I have tried earlier to convert scanned line graphics incorrectly saved as jpg to bit mapped formats but it needs higher skills than mine.

I guess some kind of intelligent filter is needed that do something like this:
-convert jpg to png (or some bit mapped format)
-Scan all pixels, group them into ranges.
-Change all colors in each range to one color only.

We now have an image with big fields of colors in different grades with dots in them from the conversion. Some will be connected (=lines/lat long lines/depth and shore curves etc ) and some will be connected groups of dots (shallows, rocks etc)

The algorithm needs to check, with some kind of threshold, and convert single dots to the surrounding color.

This is fiddly to get right... maybe someone good at image analysis could comment if it is indeed impossible.

/J

[jonasaberg]

Maybe this software could do it?
Have not made it work yet, but maybe?

It can according to the help convert a picture like this to line graphics, and if so might be possible to convert back to png with acceptable result.

Inkscape. Draw Freely.

original


converted

[allanpeda]

There are quite a few vectorization packages out there, some already as plugins for Inkscape.

Tools - Inkscape Wiki
Vector Converter | Home