Charts II: NGA - 2700 Charts

[cagney]

Originally Posted by Netsurfer

About the question of where to host the files:

I would recommend Dropbox.
The Basic (free) account offers 2GB (and could be enlarged up to 10GB by invitations).

So for hosting 30GB we would need 15 users who create a Dropbox account and host the files in their Public folders.

If you do not have a Dropbox account yet, you could use this link to go to their page and create one. As a side effect I will get more hosting space.

Of course I am willing to host some of the files ... .
So if you have any files ready for sharing, please upload them temporaryly and drop me a note with the download link. I'll then download them and upload them to my Dropbox. After that we'll need at least one place where to put the links for the downloads.

Gunther

Good Gunther, we need to get started with something.
Why don't you make opencpn.info a central hub, where links to charts or char pictures can be published, be it Drop box or somewhere else.
Some means of "booking" intended work area, to avoid double work.

Thomas

[allanpeda]

Originally Posted by cagney

Good Gunther, we need to get started with something.
Why don't you make opencpn.info a central hub, where links to charts or char pictures can be published, be it Drop box or somewhere else.
Some means of "booking" intended work area, to avoid double work.

Thomas

Here, a randomly picked chart:

NOAA-57471-Saldanha Bay - download image

Compresed from 181.43 MB in memory to 8.9 MB using PNG indexed to 32 colors. This was stitched in two hours on a humble notebook (3G RAM, I.3GHz) PC.

[allanpeda]

Here is what I propose:

Get a pool of volunteers to each convert a subset of images. They can post them anywhere (4shared has a nice preview feature), then someone can collect the charts and put them in one place, and/or convert them to torrent files.

Also, if we agree on a fie naming convention, then it will be easier to organize things.

I volunteer for an initial set of charts. I'd use the same settings for the file I uploaded above (no resampling or dithering, just a compressed color space.) Let me know if that chart looks good.

[cagney]

Just so all can see what's going on. Here is allanpedas Saldaha bay as a chart file: http://opencpn.avnisoft.co.za/NGA/NGA57471.kap.zip

A few notes regarding what I've done.
I used the four corners as REF points, easiest way to do it and gives a good georeferencing.
This is an old chart using WGS72. Strictly speaking the corrections to make this chart a WGS72 - DTM - can only be used i relation to WGS84 on OpenCPN, but the difference is probably negligible.
Note that there are other SA charts released as late as Nov 2010.
I gave SD the value LAT but that is just a guess. We only know that the level is 0.26m lower than MLWS!
Finally I used imgkap to create a kap file. A lot of processing is saved by not using tiff2bsb that requires a tiff file. Imgkap is happy with both jpg and png.
There are other advantages with imgkap as well, see the dedicated thread.

Is the quality of this chart good enough?
Is this chart working OK on all software?
OpenCPN, PolarNavy and Seaclear must be OK, Fugawi I think, but the rest....?
Remember that a raster chart is not supposed to be used zoomed by more than a factor 2. I can live with a touch lower quality and a smaller file size. Talking about the end result, not how we should store the pictures.

Thomas

[jonasaberg]

Originally Posted by brak

...The real project would be to georeference all of these images and make them into actual BSB charts. That's a *big* project It also would require someone who's good at doing this sort of thing. These will be more difficult to deal with than NZ charts because like many US charts, most of these have "insets". Each chart would need to be separated into main image (with insets whited-out) and inset images, and those georeferenced individually....

Brak is absolutely right on here. I suggest we need a communal quality check system. We need a way to "vote" on the quality of georeffed charts to warn about incorrect work.

And, I know, it is a bit premature, but it might be good to consider before we have 2000+ charts to sort through...

/J

[Surveyor]

I am tearing through the 9xxxx charts. If all goes as it has been I will have all 9xxxx charts scraped and stitched by Wednesday evening. Lets hope that NOAA doesnt get over their 4th of July hangover angry and notice that we have been raping their bandwidth.

As a reminder, I am grabbing 91004 to 97564. I am using the py script at 100%. The raw stitched images are huge and look great. So far, I have not encountered any charts with missing tiles. Brak, I will let you know if and when I do.

Who is downloading what? I only ask because I am sure that there is already overlapping interests. That is why I picked an out of the way area to start. Well, the PI being my second home has absolutely nothing to do with it!

[allanpeda]

NOAA-57460-Hondeklipbaai-to-Cape-Columbine

I'm doing one at a time when I get the chance. I'll post links here unless there is a registry page.

[LeaseOnLife]

I am working on some of the Region 6 charts, Mauritius to Southafrica/Capetown.

Until agreed otherwise, I am using a folder/naming convention like this.

/NGA-CHARTS/Region-6/61591-Mauritius

61591-Mauritius.jpg

My little GENTOO box still pukes trying to stitch to png (allenpeda perl script). The ruby script from cagney seems to work faster for me. the output file is jpg.

Here is a link to the root of the folders:
http://www.4shared.com/dir/toctSpJ5/NGA-CHARTS.html

Like allenpeda, I'll post progress here, until we have a page to register our uploads.

[allanpeda]

At 25mb, it looks as though 4shared wont preview this one.

NOAA-57440-Orange_River-to-Hondeklipbaai

[LeaseOnLife]

So far I downloaded and stitched the following region-6 charts:

61000-Algoa Bay to Cape Town.jpg
61020-Mozambique Channel-Southern Reaches.jpg
61021-Raas Xaafuun to Hobyo.jpg.4sh
61036-Seychelles.jpg
61040-Yzervarkpunt to Cape of Good Hope.jpg
61050-Cape St. Francis to Ystervarkpunt.jpg
61060-East London to Cape St. Francis.jpg
61061-Approaches to Port Elizabeth.jpg
61070-Mbashe Point to East London.jpg
61080-Port Shepstone to Mbashe Point.jpg
61090-Durnford Point to Port Shepstone.jpg
61100-Ponta do Ouro to Port Durnford.jpg
61110-Limpopo to Ponta do Ouro (Mozambique).jpg
61120-Ponta da Barra to Rio Limpopo (Mozambique).jpg
61470-Madagascar-South-Coast-Nosy Lava to Tanjon Andriamanao.jpg
61510-Madagascar-South-Coast-Tanjon Andriamanao to Tolanaro (Fort Dauphin).jpg
61520-Madagascar-East-Coast-Tolanaro to Faraony.jpg
61522-Madagascar-East-Coast-Faradofay (Fort Dauphin) and Approaches.jpg
61530-Madagascar-East-Coast-Faraony to Toamasina.jpg
61581-Reunion.jpg
61591-Mauritius.jpg
700-Port-Elizabeth-To-Madagascar.jpg

All here:
http://www.4shared.com/dir/__i2G9Rw/Region-6.html
More to come...

[allanpeda]

Wow, I'll never be able to match that rate. Oddly, my OSX system at work takes at least twice as long as my Windows 7 notebook.

NOAA-57400-Conception_Bay-to-Diaz_Point

[dacust]

Is it best to name the charts that way? I know it makes it easy to recognize on disk, but it makes it difficult to to programatically recognize them.

For my own use, I'd rather they just be named the chart number. But I totally realize that is probably just my preference.

Also, putting spaces in the names cause problems for Unix.

-dan

[allanpeda]

Originally Posted by dacust

Is it best to name the charts that way? I know it makes it easy to recognize on disk, but it makes it difficult to to programatically recognize them.

For my own use, I'd rather they just be named the chart number. But I totally realize that is probably just my preference.

Also, putting spaces in the names cause problems for Unix.

-dan

I'm agnostic about it, but regexes can parse anything reasonable, and the textual name is good when humans are perusing the list. My filenames have no spaces, only underscores.

[Amgine]

on *nix (including Mac), nice is your friend for running intensive scripts while not losing all of your cpu...

[cagney]

I think that a correct prefix to these DMA charts (sorry, old habit) should be
NGAxxxxx_short_version_of_name. Don't know where you guys get NOAA from in this connection?
Look at their web site with names as "NGA Chart: 27160 " etc.
Problems with spaces in names can pop up in unexpected places. The last thing I heard of was in the code for windows installer in OpenCPN.... Best to avoid it altogether by using underscores.
Thomas