Charts II: NGA - 2700 Charts

[LeaseOnLife]

Allen

yes, png can compress nicely. But is the proposed reduction to 32 colors and going to indexed colors the final word? If not, we will have to re-download, and re-stitch everything we did before.

I would like to store the downloaded and stitched images without any loss of quality.

As a separate step we then can trial several ways to reduce file sizes.

[allanpeda]

Originally Posted by LeaseOnLife

Allen

yes, png can compress nicely. But is the proposed reduction to 32 colors and going to indexed colors the final word? If not, we will have to re-download, and re-stitch everything we did before.

I would like to store the downloaded and stitched images without any loss of quality.

As a separate step we then can trial several ways to reduce file sizes.

Don't know if it qualifies as the final word, but as a lossless compression scheme, PNG is a much better choice than JPEG (which, importantly, is not very good for keeping text artifact free). Imagemagic docs really slam TIFF, so PNG seems a step in the right direction. I was going to run some trials, but stitching really hogs my CPU so I won't do anything until tonight.

Strictly speaking, the color reduction is altering the original image, but honestly I think the resulting images look clearer anyhow.

If you are using Imagemagick, and if size is not an issue, then I would weigh in for compressed MIFF as an intermediate file format.

[Surveyor]

I highly recommend the python script. The ability to download in batches is priceless. the command looks like this:

dezoomify.py -i c:\nga\list.txt -l -q 100 -o c:\nga\out.jpg

It outputs the files consecutively numbered out000.jpg out001.jpg out002 and so on, so there is some careful renaming to be done.

I think the code could probably be modified to output png files (or whatever) and possibly the necessary conversions in colors and palette. I am going to read up on PIL but I am certainly no programmer. I am just a guy that has the time, a machine and will follow the agreed upon conversion protocol.

I have 20 charts DLed and I just made a "list.txt" that has the next 100 in the 9xxxx series.

[dacust]

First priority should be to get the downloads in case they change the site or remove the content.

I would propose that the originals be downloaded at 100%, then converted directly to lossless .png at full size.

Preferably, the full size .png's should be made available for download by torrent. This is so any one wanting them for their own use can have them.

Then, they can be converted and torrent downloads made for them.

-dan

[dougwm10]

Hi

I consider the recommendation by Dan to download all the charts whilst they are still available to be well worth following.


Processing of them can then be done on a sectional basis spread around the group of volunteers.


The server location to take the full download should be the priority for now - any suggestions for a suitable location ????

Doug

[dacust]

If it is done as torrents, no one server is needed.

[brak]

Originally Posted by dacust

First priority should be to get the downloads in case they change the site or remove the content.

I would propose that the originals be downloaded at 100%, then converted directly to lossless .png at full size.

Preferably, the full size .png's should be made available for download by torrent. This is so any one wanting them for their own use can have them.

Then, they can be converted and torrent downloads made for them.

-dan

a set of 400 caribbean charts in full size png would take just about 100Gb (verified). a full set of these charts would be pushing a 1TB.

it seems unlikely that NOAA would "pull" them. these charts just became available at Oceangrafix (to purchase as paper), and also on Zoomify there (although they run a slightly different version of Zoomify that is quite a bit harder to use for downloading all tiles).

In fact, I would suggest contacting NOAA to ask if they may be willing to provide full-size complete images. Worst case scenario - they say "no"

also, a few charts seem to be missing tiles on NOAA site. if you find any like that - let them know.

[LeaseOnLife]

brak, how did you calculate 100GB for 400 charts?
That would be 256MB / chart. The largest png file I have made so far is 150MB, most are much smaller. I am also trying to estimate the needed space for all charts.

Instead of storing the lossless (and large) pngs, we could just store all the tiles in original quality. This would mean breaking up the python or ruby scripts to download only, and have a separate stitching session. I am back to playing with allenpeda's stitching script.

For downloading I am just using the following command line:

wget -LNr -nd -A jpg http://www.charts.noaa.gov/NGAViewer/61000/

This puts all the tiles/jpgs into the current directory as required by allenpeda's stitching script.

[cagney]

Originally Posted by brak

......
In fact, I would suggest contacting NOAA to ask if they may be willing to provide full-size complete images. Worst case scenario - they say "no"
........

I think this a good idea. Why not try before we put a lot of work into "restoring" downgraded chart files?

Brak, can you please contact Noaa on behalf of this "project"?
I'm not sure they will listen to an "alien" ....

Thomas

[cagney]

Originally Posted by Netsurfer

...

What does this mean?
For my understanding there is a OpenCPN community in addition to the pure program's project. And imho the community should get involved.

...
Gunther
...

Don't jump to conclusions. Why do you think I started this thread?
Not many users noticed "allanpedas" original thread. I thought it worth trying a new thread in the OpenCPN forum, to involve all the users that contributed to original "Chart" thread.
Looking at the users already involved here, and all the ideas put forward, I'm sure we together, will work something out.

Thomas

[dacust]

Originally Posted by brak

...
it seems unlikely that NOAA would "pull" them. these charts just became available at Oceangrafix (to purchase as paper), and also on Zoomify there (although they run a slightly different version of Zoomify that is quite a bit harder to use for downloading all tiles).
...

True, but:

There were 10's of thousands of historical charts on NOAA's site that I was downloading. After I got a little over 7,000 of them, they reorganized the site where you could no longer get to the directory directly. You used a cute interface where you requested documents by date range and location, then the selected ones were put in a .zip file for you to download.

So, having been locked out once, I am just on the cautious side.

-dan

[Netsurfer]

About the question of where to host the files:

I would recommend Dropbox.
The Basic (free) account offers 2GB (and could be enlarged up to 10GB by invitations).

So for hosting 30GB we would need 15 users who create a Dropbox account and host the files in their Public folders.

If you do not have a Dropbox account yet, you could use this link to go to their page and create one. As a side effect I will get more hosting space.

Of course I am willing to host some of the files ... .
So if you have any files ready for sharing, please upload them temporaryly and drop me a note with the download link. I'll then download them and upload them to my Dropbox. After that we'll need at least one place where to put the links for the downloads.

Gunther

[brak]

Originally Posted by cagney

I think this a good idea. Why not try before we put a lot of work into "restoring" downgraded chart files?

Brak, can you please contact Noaa on behalf of this "project"?
I'm not sure they will listen to an "alien" ....

Thomas

I found them to be usually quite responsive to anyone (beside the fact that unless your email address ends in something other than .com - they don't know where you are).

I.e. - they will always respond, though not always with the answer one would like

I already emailed them - if there is a positive answer, will let you know.

[allanpeda]

Originally Posted by LeaseOnLife

brak, how did you calculate 100GB for 400 charts?
That would be 256MB / chart. The largest png file I have made so far is 150MB, most are much smaller. I am also trying to estimate the needed space for all charts.

Instead of storing the lossless (and large) pngs, we could just store all the tiles in original quality. This would mean breaking up the python or ruby scripts to download only, and have a separate stitching session. I am back to playing with allenpeda's stitching script.

For downloading I am just using the following command line:

wget -LNr -nd -A jpg http://www.charts.noaa.gov/NGAViewer/61000/

This puts all the tiles/jpgs into the current directory as required by allenpeda's stitching script.

Gee, flattered anyone used that script I tossed together. I think Brak is right to point out that there is no crisis here. Though NOAA encourages sale of their charts by value added resellers, I don't think they mind fetching images directly, as the charts specify that they are not copyrighted.

It would be considerate to throttle any recursive dowloads, and wget does have options to do such that.

wget

-w, --wait=SECONDS wait SECONDS between retrievals.
--waitretry=secs wait 1..secs between retries.
--random-wait wait from 0...2*WAIT secs between retrievals.

I cleaned up the script I wrote and put it here. It works on OSX or windows with light editing of the path to convert.exe. In the present case the path to convert.exe is specified for windows machines. Note that Imagemagick's convert.exe conflicts with Microoft convert.exe, so one needs to take steps to avoid conflicts; either specify the path, symlink an alternate name (I used imconvert.exe) etc.
chartstitcher.pl - 4shared.com download

I realize I should roll wget into it as well, but the process is so slow as is anyhow. Also I would then want to delete intermediate files and everything would become very complex quickly. If anyone wants to add that, then have at it.

So even though the ENCs and RNCs are available for most of these charts, there is a demand for pure images?

[brak]

Originally Posted by allanpeda

So even though the ENCs and RNCs are available for most of these charts, there is a demand for pure images?

These charts are of non-US coverage and RNCs are not actually available (in fact, at this point for some of the areas there are no RNCs that I know of, at least not current). They are quite a bit out of date, unfortunately (15 - 30 years) - not even updated for their own NOTAMs. However, I suppose for a general area coverage and passage planning or "armchair sailing" at home - they would do just fine.


That said, reassembly of images isn't really the hardest project.

The real project would be to georeference all of these images and make them into actual BSB charts. That's a *big* project It also would require someone who's good at doing this sort of thing. These will be more difficult to deal with than NZ charts because like many US charts, most of these have "insets". Each chart would need to be separated into main image (with insets whited-out) and inset images, and those georeferenced individually.

Alternatively, if georeferencing software would understand that sort of thing, it could be done in one program (outline polygons, number charts, save) but I have a feeling that freely available referencing software won't do that (I bet NOAA has something)

Also, although this has not been discussed much, to make these into *decent* BSB charts (i.e. approximating in qualuty those available officially) these images would require some serious image processing. ImageMagick "blur" won't do it.

I have been playing with some custom filtering and results are pretty good, I would also like to color-change these to match official BSB charts - time permitting.