Cruisers Forum
 


Join CruisersForum Today

Reply
 
Thread Tools Rating: Thread Rating: 3 votes, 5.00 average. Display Modes
Old 01-07-2011, 11:29   #1
Registered User

Join Date: Dec 2005
Location: WNA
Boat: Dufour 35
Posts: 3,247
Charts II: NGA - 2700 Charts

The first "Chart" thread was started to discuss and convert all the NZ carts.
Now, through this thread -Chart Images NOAA 25609 BVI, I have found out that there are ~2700 chart pictures, world wide available on the net. The pictures are in tiles stitched together, to one flash picture, something like a new version of the old hdr files.
Scripts and websites are available to "dezoomify" the pictures on the NOAA/NGA website and download the result as a jpg or png picture.
These pictures are rather large and need to be made smaller.
Then it's just the normal georeferencing etc to make a kap file.
The pictures are "pixelperfect" and a pleasure to work with.
Here is the central website: NOAA's On-Line Chart Viewer
The directories behind the website: Index of /NGAViewer
A Pythonscript that downloads a stitched together Jpg: Help:Zoomable images/dezoomify.py - Wikimedia Commons
A Ruby script to do the same thing: https://gist.github.com/770310
A webscript in html/js/php:https://gist.github.com/770310
A website to do the job:Dezoomify. This site produces a png files.
The format of the links to feed both scripts and the web site is similar to this:
Quote:
Chart 57440
All this is a very slow process screaming for a cluster of linux boxes.
When the ruby script came to this
Quote:
Stitching 56 x 76 = 4256 tiles...
my load average was around 13 and the box was totally unresponsive for a long time. Be warned!

Let us together work on this. First the picture wizards ... on how to make these pictures small enough and good enough ...please.

Thanks to NGA for the web site. A worldwide coverage of freely available charts is now within reach. If only BA could take the hint from NOAA, NGA, LINZ and the Brazilian Hydrographer........

Thomas
__________________

__________________
cagney is offline   Reply With Quote
Old 01-07-2011, 11:57   #2
Sponsoring Vendor
 
brak's Avatar

Community Sponsor
Cruisers Forum Supporter

Join Date: Jan 2009
Location: East of West
Posts: 252
Re: Charts II: NGA - 2700 charts

Nice, I didn't realize they had old Defense Agency charts there.

It would not be too difficult to automate download and restitching.
A few notes:
1. I would suggest doing this in two parts - build a download script and locate a repository with enough space, then get the data.

2. Stitch it separately. It's a lengthy process.

3. Georeference (that's a hell of a job)

Step 1 then would go a hell of a lot faster.
__________________
Polar Navy - because life is too short to use ugly navigation software
brak is offline   Reply With Quote
Old 01-07-2011, 18:44   #3
Obsfucator, Second Class
 
dacust's Avatar

Cruisers Forum Supporter

Join Date: Feb 2008
Location: Southeast USA.
Boat: 1982 Sea Ray SRV360
Posts: 1,743
Re: Charts II: NGA - 2700 charts

Well, I was going to try this.

First I gave the Python script a try:
  • Download Python (2.7).
  • Install Python
  • Download script.
  • Script says it needs PIL (Python Image Library).
  • Download PIL
  • Install Pil
  • PIL says it needs registered version of Python.
  • Download script to register Python.
  • Registration script blows up.
  • Give up.

Back to this thread to try the ruby script.
  • Notice the link you gave for the ruby script is actually the url for the webscript

OK, I'm going to try the webscript, now...

EDIT: Found Ruby script at https://gist.github.com/59636

-dan
dacust is offline   Reply With Quote
Old 01-07-2011, 19:43   #4
Registered User

Join Date: Dec 2010
Boat: Catalina 30
Posts: 211
Re: Charts II: NGA - 2700 charts

I installed Cygwin, Strawberry Perl, and ImageMagick. I used

wget -LNr http://www.charts.noaa.gov/NGAViewer/25609/

to snag all the tile images before placing them in a single directory. Most interesting are the arguments passed to ImageMagick.

Stitcher I write is linked below:

http://goo.gl/4ZPX8
__________________
allanpeda is offline   Reply With Quote
Old 01-07-2011, 20:29   #5
Registered User

Join Date: Mar 2009
Posts: 83
Re: Charts II: NGA - 2700 charts

Hi

I have had a look at the sample charts available and consider that they are an extremely valuable source for charts and if they can be referenced and made to run under Opencpn then we would all have access to a world wide chart database.

I am not sufficiently computer adept to assist in this conversion task / project but would heartily support anyone who feels they can get involved.

Hopefully this can be organized in a methodical manner so that duplication of effort is avoided and an ongoing database of finished charts is available for individuals to download.
__________________
dougwm10 is offline   Reply With Quote
Old 01-07-2011, 21:19   #6
Sponsoring Vendor
 
brak's Avatar

Community Sponsor
Cruisers Forum Supporter

Join Date: Jan 2009
Location: East of West
Posts: 252
Re: Charts II: NGA - 2700 charts

There is an issue in these charts that will make building good quality BSB challenging.

The charts appear to be scanned directly from paper charts. This is different from specially crafted images used in raster charts. Consider the tile (zoomed in) below.

What should be a uniform color, is actually a collection of "paint pixels" - artifact of offset printing.
BSB is best encoded from data that has large uniform color surfaces, that can be compressed in RLE. This data would result in terrible RLE - larger than original size.

They need some sort of color/graphic processing to turn pixelated color areas into solid colors.
Attached Thumbnails
Click image for larger version

Name:	Screen shot 2011-07-01 at 11.15.19 PM.png
Views:	307
Size:	208.0 KB
ID:	29132  
__________________
Polar Navy - because life is too short to use ugly navigation software
brak is offline   Reply With Quote
Old 01-07-2011, 22:37   #7
Registered User

Join Date: Dec 2010
Boat: Catalina 30
Posts: 211
Re: Charts II: NGA - 2700 charts

Well, it's certainly more work than it would seem at first blush, but there are projects to assemble street maps from freely available data, (OpenStreetMap) so why not nautical charts?

Yes, I know, the stakes are higher at sea.
__________________
allanpeda is offline   Reply With Quote
Old 02-07-2011, 02:54   #8
Registered User

Join Date: Dec 2005
Location: WNA
Boat: Dufour 35
Posts: 3,247
Re: Charts II: NGA - 2700 charts

Quote:
Originally Posted by brak View Post
There is an issue in these charts that will make building good quality BSB challenging.

The charts appear to be scanned directly from paper charts. This is different from specially crafted images used in raster charts. Consider the tile (zoomed in) below.

What should be a uniform color, is actually a collection of "paint pixels" - artifact of offset printing.
BSB is best encoded from data that has large uniform color surfaces, that can be compressed in RLE. This data would result in terrible RLE - larger than original size.

They need some sort of color/graphic processing to turn pixelated color areas into solid colors.
Brak

Where did you get that picture tile from?
A quick, look around, and I can't find anything that bad.
What I find looks more like this

Name:   57471_6-31-34.jpg
Views: 8346
Size:  8.4 KB
from chart 57471. This tile was downloaded with wget.
I suppose the individual charts can have different origins.

Thomas
__________________
cagney is offline   Reply With Quote
Old 02-07-2011, 05:07   #9
Registered User

Join Date: Dec 2005
Location: WNA
Boat: Dufour 35
Posts: 3,247
Re: Charts II: NGA - 2700 Charts

What is an acceptable chart quality.
Is this good enough?
Starting with a 20.1 MiB chart file, ending with a 6.5 MiB kap file.
This was the process after downloading.
Quote:
$convert 26342.jpg -rotate -90 out.jpg
The chart was skewed 90 degrees.
Quote:
$
convert 26342.jpg -resize 35% -depth 4 26342_s.jpg
50% leads to a much bigger size, 25% usable but poor quality.
Test the chart:http://opencpn.avnisoft.co.za/NGA/26342.kap.zip

Thomas
__________________
cagney is offline   Reply With Quote
Old 02-07-2011, 05:31   #10
Registered User

Join Date: Dec 2010
Boat: Catalina 30
Posts: 211
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by cagney View Post
What is an acceptable chart quality.
Is this good enough?
Starting with a 20.1 MiB chart file, ending with a 6.5 MiB kap file.
This was the process after downloading.

The chart was skewed 90 degrees.

50% leads to a much bigger size, 25% usable but poor quality.
Test the chart:http://opencpn.avnisoft.co.za/NGA/26342.kap.zip

Thomas
Atleast for the NOAA charts I do two passes, first into into strips (kept in native Imagemagick format), then I stitch the strips together:

convert f1.miff f2.miff f3.miff +append -quality 0 +dither -colors 32 -depth 8 -density 600x600 -units pixelsperinch outputfile.png

No resampling, and as suggested by brak, I keep original resolution, but I reduce colors and convert from RGB to indexed. What was originally 50mb, drops to under 6mb.

__________________
allanpeda is offline   Reply With Quote
Old 02-07-2011, 06:56   #11
Sponsoring Vendor
 
brak's Avatar

Community Sponsor
Cruisers Forum Supporter

Join Date: Jan 2009
Location: East of West
Posts: 252
Re: Charts II: NGA - 2700 charts

Quote:
Originally Posted by cagney View Post
Brak

Where did you get that picture tile from?
A quick, look around, and I can't find anything that bad.
What I find looks more like this
Attachment 29138
from chart 57471. This tile was downloaded with wget.
I suppose the individual charts can have different origins.

Thomas
Yes, they do differ - so far I find that about half the charts are paper-scans like the one i posted above, out of a sample of about 100 so far.

Others are "digitally generated" (like modern BSB charts that keep the look of paper charts but appear to be actually rendered as images), like your example.

These pixelated scanned charts would take some serious image processing.

Curiously, this pixelation is such that a PNG image of a reassembled chart had a size of 180Mb whereas straight BMP - only about 60Mb It's a "perfect noise". RLE-encoded BSB chart would be even bigger.

In terms of size, these images appear to be the same size (and DPI) as standard NOAA raster charts. I.e. - they probably should not be resized much.
__________________
Polar Navy - because life is too short to use ugly navigation software
brak is offline   Reply With Quote
Old 02-07-2011, 07:03   #12
Sponsoring Vendor
 
brak's Avatar

Community Sponsor
Cruisers Forum Supporter

Join Date: Jan 2009
Location: East of West
Posts: 252
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by cagney View Post
What is an acceptable chart quality.
Is this good enough?
Starting with a 20.1 MiB chart file, ending with a 6.5 MiB kap file.
This was the process after downloading.

The chart was skewed 90 degrees.

50% leads to a much bigger size, 25% usable but poor quality.
Test the chart:http://opencpn.avnisoft.co.za/NGA/26342.kap.zip

Thomas
Lots of jpeg artifacts, they are not too visible in original colors, but say in night colors (autoconverted) you can see them more clearly.

This is what's affecting the size - they are not easily compressible and also make palette much larger. This is a result of original files being jpeg (I wish they used png and lossless compression).

See attached.

This would require noise filtering, preferably something that understands about raster charts (I.e. able to come back to smooth surfaces with a small number of colors)
Attached Thumbnails
Click image for larger version

Name:	Screen shot 2011-07-02 at 9.02.39 AM.png
Views:	278
Size:	77.1 KB
ID:	29147  
__________________
Polar Navy - because life is too short to use ugly navigation software
brak is offline   Reply With Quote
Old 02-07-2011, 09:35   #13
Registered User
 
jonasaberg's Avatar

Join Date: Jul 2008
Location: Kristiansand, Norway
Boat: Wasa 410
Posts: 305
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by brak View Post
Lots of jpeg artifacts ... This is a result of original files being jpeg (I wish they used png and lossless compression ).
This would require noise filtering, preferably something that understands about raster charts (I.e. able to come back to smooth surfaces with a small number of colors)
Couldn't agree more.
I have tried earlier to convert scanned line graphics incorrectly saved as jpg to bit mapped formats but it needs higher skills than mine.

I guess some kind of intelligent filter is needed that do something like this:
-convert jpg to png (or some bit mapped format)
-Scan all pixels, group them into ranges.
-Change all colors in each range to one color only.

We now have an image with big fields of colors in different grades with dots in them from the conversion. Some will be connected (=lines/lat long lines/depth and shore curves etc ) and some will be connected groups of dots (shallows, rocks etc)

The algorithm needs to check, with some kind of threshold, and convert single dots to the surrounding color.

This is fiddly to get right... maybe someone good at image analysis could comment if it is indeed impossible.

/J
__________________
jonasaberg is offline   Reply With Quote
Old 02-07-2011, 10:40   #14
Registered User
 
jonasaberg's Avatar

Join Date: Jul 2008
Location: Kristiansand, Norway
Boat: Wasa 410
Posts: 305
Re: Charts II: NGA - 2700 Charts

Maybe this software could do it?
Have not made it work yet, but maybe?

It can according to the help convert a picture like this to line graphics, and if so might be possible to convert back to png with acceptable result.

Inkscape. Draw Freely.

original


converted
__________________
jonasaberg is offline   Reply With Quote
Old 02-07-2011, 10:54   #15
Registered User

Join Date: Dec 2010
Boat: Catalina 30
Posts: 211
Re: Charts II: NGA - 2700 Charts

There are quite a few vectorization packages out there, some already as plugins for Inkscape.

Tools - Inkscape Wiki
Vector Converter | Home
__________________

__________________
allanpeda is offline   Reply With Quote
Reply

Tags
charts

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off




Copyright 2002- Social Knowledge, LLC All Rights Reserved.

All times are GMT -7. The time now is 12:07.


Google+
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Social Knowledge Networks
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.

ShowCase vBulletin Plugins by Drive Thru Online, Inc.