Cruisers Forum
 


Join CruisersForum Today

Reply
 
Thread Tools Rating: Thread Rating: 3 votes, 5.00 average. Display Modes
Old 03-07-2011, 19:19   #31
Registered User
 
LeaseOnLife's Avatar

Join Date: Apr 2008
Location: out cruising again
Boat: Sailboat
Posts: 1,047
Re: Charts II: NGA - 2700 Charts

Allen

yes, png can compress nicely. But is the proposed reduction to 32 colors and going to indexed colors the final word? If not, we will have to re-download, and re-stitch everything we did before.

I would like to store the downloaded and stitched images without any loss of quality.

As a separate step we then can trial several ways to reduce file sizes.
__________________

__________________
LeaseOnLife is offline   Reply With Quote
Old 03-07-2011, 19:30   #32
Registered User

Join Date: Dec 2010
Boat: Catalina 30
Posts: 211
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by LeaseOnLife View Post
Allen

yes, png can compress nicely. But is the proposed reduction to 32 colors and going to indexed colors the final word? If not, we will have to re-download, and re-stitch everything we did before.

I would like to store the downloaded and stitched images without any loss of quality.

As a separate step we then can trial several ways to reduce file sizes.
Don't know if it qualifies as the final word, but as a lossless compression scheme, PNG is a much better choice than JPEG (which, importantly, is not very good for keeping text artifact free). Imagemagic docs really slam TIFF, so PNG seems a step in the right direction. I was going to run some trials, but stitching really hogs my CPU so I won't do anything until tonight.

Strictly speaking, the color reduction is altering the original image, but honestly I think the resulting images look clearer anyhow.

If you are using Imagemagick, and if size is not an issue, then I would weigh in for compressed MIFF as an intermediate file format.
__________________

__________________
allanpeda is offline   Reply With Quote
Old 03-07-2011, 19:40   #33
Registered User
 
Surveyor's Avatar

Join Date: Mar 2010
Location: Ft. Pierce, FL - Bislig, Philippines - Gladstone QLD
Boat: 1968 Alberg 30 #329
Posts: 451
Re: Charts II: NGA - 2700 Charts

I highly recommend the python script. The ability to download in batches is priceless. the command looks like this:

Code:
dezoomify.py -i c:\nga\list.txt -l -q 100 -o c:\nga\out.jpg
It outputs the files consecutively numbered out000.jpg out001.jpg out002 and so on, so there is some careful renaming to be done.

I think the code could probably be modified to output png files (or whatever) and possibly the necessary conversions in colors and palette. I am going to read up on PIL but I am certainly no programmer. I am just a guy that has the time, a machine and will follow the agreed upon conversion protocol.

I have 20 charts DLed and I just made a "list.txt" that has the next 100 in the 9xxxx series.
__________________
Jim - Bahala na!
http://svcookie.blogspot.com/
KK4GGF
Surveyor is offline   Reply With Quote
Old 03-07-2011, 20:11   #34
Obsfucator, Second Class
 
dacust's Avatar

Cruisers Forum Supporter

Join Date: Feb 2008
Location: Southeast USA.
Boat: 1982 Sea Ray SRV360
Posts: 1,743
Re: Charts II: NGA - 2700 Charts

First priority should be to get the downloads in case they change the site or remove the content.

I would propose that the originals be downloaded at 100%, then converted directly to lossless .png at full size.

Preferably, the full size .png's should be made available for download by torrent. This is so any one wanting them for their own use can have them.

Then, they can be converted and torrent downloads made for them.

-dan
dacust is offline   Reply With Quote
Old 03-07-2011, 20:40   #35
Registered User

Join Date: Mar 2009
Posts: 83
Re: Charts II: NGA - 2700 Charts

Hi

I consider the recommendation by Dan to download all the charts whilst they are still available to be well worth following.


Processing of them can then be done on a sectional basis spread around the group of volunteers.


The server location to take the full download should be the priority for now - any suggestions for a suitable location ????

Doug
__________________
dougwm10 is offline   Reply With Quote
Old 03-07-2011, 20:44   #36
Obsfucator, Second Class
 
dacust's Avatar

Cruisers Forum Supporter

Join Date: Feb 2008
Location: Southeast USA.
Boat: 1982 Sea Ray SRV360
Posts: 1,743
Re: Charts II: NGA - 2700 Charts

If it is done as torrents, no one server is needed.
dacust is offline   Reply With Quote
Old 03-07-2011, 22:15   #37
Sponsoring Vendor
 
brak's Avatar

Community Sponsor
Cruisers Forum Supporter

Join Date: Jan 2009
Location: East of West
Posts: 252
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by dacust View Post
First priority should be to get the downloads in case they change the site or remove the content.

I would propose that the originals be downloaded at 100%, then converted directly to lossless .png at full size.

Preferably, the full size .png's should be made available for download by torrent. This is so any one wanting them for their own use can have them.

Then, they can be converted and torrent downloads made for them.

-dan
a set of 400 caribbean charts in full size png would take just about 100Gb (verified). a full set of these charts would be pushing a 1TB.

it seems unlikely that NOAA would "pull" them. these charts just became available at Oceangrafix (to purchase as paper), and also on Zoomify there (although they run a slightly different version of Zoomify that is quite a bit harder to use for downloading all tiles).

In fact, I would suggest contacting NOAA to ask if they may be willing to provide full-size complete images. Worst case scenario - they say "no"

also, a few charts seem to be missing tiles on NOAA site. if you find any like that - let them know.
__________________
Polar Navy - because life is too short to use ugly navigation software
brak is offline   Reply With Quote
Old 03-07-2011, 22:34   #38
Registered User
 
LeaseOnLife's Avatar

Join Date: Apr 2008
Location: out cruising again
Boat: Sailboat
Posts: 1,047
Re: Charts II: NGA - 2700 Charts

brak, how did you calculate 100GB for 400 charts?
That would be 256MB / chart. The largest png file I have made so far is 150MB, most are much smaller. I am also trying to estimate the needed space for all charts.

Instead of storing the lossless (and large) pngs, we could just store all the tiles in original quality. This would mean breaking up the python or ruby scripts to download only, and have a separate stitching session. I am back to playing with allenpeda's stitching script.

For downloading I am just using the following command line:
Code:
wget -LNr -nd -A jpg http://www.charts.noaa.gov/NGAViewer/61000/
This puts all the tiles/jpgs into the current directory as required by allenpeda's stitching script.
__________________
LeaseOnLife is offline   Reply With Quote
Old 04-07-2011, 02:52   #39
Registered User

Join Date: Dec 2005
Location: WNA
Boat: Dufour 35
Posts: 3,250
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by brak View Post
......
In fact, I would suggest contacting NOAA to ask if they may be willing to provide full-size complete images. Worst case scenario - they say "no"
........
I think this a good idea. Why not try before we put a lot of work into "restoring" downgraded chart files?

Brak, can you please contact Noaa on behalf of this "project"?
I'm not sure they will listen to an "alien" ....

Thomas
__________________
cagney is offline   Reply With Quote
Old 04-07-2011, 03:14   #40
Registered User

Join Date: Dec 2005
Location: WNA
Boat: Dufour 35
Posts: 3,250
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by Netsurfer View Post
...
Quote:
It is also questionable if OpenCPN, as a project should get involved in this.
What does this mean?
For my understanding there is a OpenCPN community in addition to the pure program's project. And imho the community should get involved.

...
Gunther
...
Don't jump to conclusions. Why do you think I started this thread?
Not many users noticed "allanpedas" original thread. I thought it worth trying a new thread in the OpenCPN forum, to involve all the users that contributed to original "Chart" thread.
Looking at the users already involved here, and all the ideas put forward, I'm sure we together, will work something out.

Thomas
__________________
cagney is offline   Reply With Quote
Old 04-07-2011, 06:18   #41
Obsfucator, Second Class
 
dacust's Avatar

Cruisers Forum Supporter

Join Date: Feb 2008
Location: Southeast USA.
Boat: 1982 Sea Ray SRV360
Posts: 1,743
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by brak View Post
...
it seems unlikely that NOAA would "pull" them. these charts just became available at Oceangrafix (to purchase as paper), and also on Zoomify there (although they run a slightly different version of Zoomify that is quite a bit harder to use for downloading all tiles).
...
True, but:

There were 10's of thousands of historical charts on NOAA's site that I was downloading. After I got a little over 7,000 of them, they reorganized the site where you could no longer get to the directory directly. You used a cute interface where you requested documents by date range and location, then the selected ones were put in a .zip file for you to download.

So, having been locked out once, I am just on the cautious side.

-dan
dacust is offline   Reply With Quote
Old 04-07-2011, 06:36   #42
Registered User
 
Netsurfer's Avatar

Join Date: Jan 2010
Location: Cologne, Germany
Boat: Beneteau Oceanis 331
Posts: 557
Re: Charts II: NGA - 2700 Charts

About the question of where to host the files:

I would recommend Dropbox.
The Basic (free) account offers 2GB (and could be enlarged up to 10GB by invitations).

So for hosting 30GB we would need 15 users who create a Dropbox account and host the files in their Public folders.

If you do not have a Dropbox account yet, you could use this link to go to their page and create one. As a side effect I will get more hosting space.

Of course I am willing to host some of the files ... .
So if you have any files ready for sharing, please upload them temporaryly and drop me a note with the download link. I'll then download them and upload them to my Dropbox. After that we'll need at least one place where to put the links for the downloads.

Gunther
__________________
Deutschsprachige Community- und Support-Website unter OpenCPN.de
Netsurfer is offline   Reply With Quote
Old 04-07-2011, 06:46   #43
Sponsoring Vendor
 
brak's Avatar

Community Sponsor
Cruisers Forum Supporter

Join Date: Jan 2009
Location: East of West
Posts: 252
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by cagney View Post
I think this a good idea. Why not try before we put a lot of work into "restoring" downgraded chart files?

Brak, can you please contact Noaa on behalf of this "project"?
I'm not sure they will listen to an "alien" ....

Thomas
I found them to be usually quite responsive to anyone (beside the fact that unless your email address ends in something other than .com - they don't know where you are).

I.e. - they will always respond, though not always with the answer one would like

I already emailed them - if there is a positive answer, will let you know.
__________________
Polar Navy - because life is too short to use ugly navigation software
brak is offline   Reply With Quote
Old 04-07-2011, 07:06   #44
Registered User

Join Date: Dec 2010
Boat: Catalina 30
Posts: 211
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by LeaseOnLife View Post
brak, how did you calculate 100GB for 400 charts?
That would be 256MB / chart. The largest png file I have made so far is 150MB, most are much smaller. I am also trying to estimate the needed space for all charts.

Instead of storing the lossless (and large) pngs, we could just store all the tiles in original quality. This would mean breaking up the python or ruby scripts to download only, and have a separate stitching session. I am back to playing with allenpeda's stitching script.

For downloading I am just using the following command line:
Code:
wget -LNr -nd -A jpg http://www.charts.noaa.gov/NGAViewer/61000/
This puts all the tiles/jpgs into the current directory as required by allenpeda's stitching script.
Gee, flattered anyone used that script I tossed together. I think Brak is right to point out that there is no crisis here. Though NOAA encourages sale of their charts by value added resellers, I don't think they mind fetching images directly, as the charts specify that they are not copyrighted.

It would be considerate to throttle any recursive dowloads, and wget does have options to do such that.

wget

-w, --wait=SECONDS wait SECONDS between retrievals.
--waitretry=secs wait 1..secs between retries.
--random-wait wait from 0...2*WAIT secs between retrievals.


I cleaned up the script I wrote and put it here. It works on OSX or windows with light editing of the path to convert.exe. In the present case the path to convert.exe is specified for windows machines. Note that Imagemagick's convert.exe conflicts with Microoft convert.exe, so one needs to take steps to avoid conflicts; either specify the path, symlink an alternate name (I used imconvert.exe) etc.
chartstitcher.pl - 4shared.com download

I realize I should roll wget into it as well, but the process is so slow as is anyhow. Also I would then want to delete intermediate files and everything would become very complex quickly. If anyone wants to add that, then have at it.

So even though the ENCs and RNCs are available for most of these charts, there is a demand for pure images?
__________________
allanpeda is offline   Reply With Quote
Old 04-07-2011, 08:08   #45
Sponsoring Vendor
 
brak's Avatar

Community Sponsor
Cruisers Forum Supporter

Join Date: Jan 2009
Location: East of West
Posts: 252
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by allanpeda View Post
So even though the ENCs and RNCs are available for most of these charts, there is a demand for pure images?
These charts are of non-US coverage and RNCs are not actually available (in fact, at this point for some of the areas there are no RNCs that I know of, at least not current). They are quite a bit out of date, unfortunately (15 - 30 years) - not even updated for their own NOTAMs. However, I suppose for a general area coverage and passage planning or "armchair sailing" at home - they would do just fine.


That said, reassembly of images isn't really the hardest project.

The real project would be to georeference all of these images and make them into actual BSB charts. That's a *big* project It also would require someone who's good at doing this sort of thing. These will be more difficult to deal with than NZ charts because like many US charts, most of these have "insets". Each chart would need to be separated into main image (with insets whited-out) and inset images, and those georeferenced individually.

Alternatively, if georeferencing software would understand that sort of thing, it could be done in one program (outline polygons, number charts, save) but I have a feeling that freely available referencing software won't do that (I bet NOAA has something)

Also, although this has not been discussed much, to make these into *decent* BSB charts (i.e. approximating in qualuty those available officially) these images would require some serious image processing. ImageMagick "blur" won't do it.

I have been playing with some custom filtering and results are pretty good, I would also like to color-change these to match official BSB charts - time permitting.
__________________
Polar Navy - because life is too short to use ugly navigation software
brak is offline   Reply With Quote
Reply

Tags
charts

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off




Copyright 2002- Social Knowledge, LLC All Rights Reserved.

All times are GMT -7. The time now is 13:49.


Google+
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Social Knowledge Networks
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.

ShowCase vBulletin Plugins by Drive Thru Online, Inc.