Cruisers Forum
 


Reply
 
Thread Tools Search this Thread Rating: Thread Rating: 3 votes, 5.00 average. Display Modes
Old 02-07-2011, 23:04   #16
Marine Service Provider

Join Date: Feb 2010
Posts: 279
Re: Charts II: NGA - 2700 Charts

I am happy to help with the conversion process to bsb format. I can tackle the med charts from NGA. As i live in the Med this is not as altruistic as it seems. I have tried the rubyy script but get a file not found error. What we need is process for all to follow to get consistent results. Broadly I assume the process may look like this

1.scrape the image tiles from NGA
2..stitch
3. process image
4.georef using mapcal
5.Convert to bsb using img2kap
6.upload to opencpn depository

my concern is that all of us hit NGA with dezoomify which may result in a denial of service situation leading NGA to remove the charts

If someone can supply the tools and process I can start on the conversion of the Med charts today. (I already have MapCal)

regards

Richard Fairman
r.fairman is offline   Reply With Quote
Old 02-07-2011, 23:08   #17
Registered User

Join Date: Dec 2010
Boat: Catalina 30
Posts: 213
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by r.fairman View Post
...
my concern is that all of us hit NGA with dezoomify which may result in a denial of service situation leading NGA to remove the charts

If someone can supply the tools and process I can start on the conversion of the Med charts today. (I already have MapCal)

regards

Richard Fairman
A few people should share these images using bittorrent. Then bandwith would be *increased* as more people download. Either pre-stitch the pieces into a lossless single image, or bundle the tiles into an archive.
allanpeda is offline   Reply With Quote
Old 03-07-2011, 03:26   #18
Registered User
 
LeaseOnLife's Avatar

Join Date: Apr 2008
Location: out cruising again, currently in Fiji
Boat: Sailboat
Posts: 1,466
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by r.fairman View Post
Broadly I assume the process may look like this

1.scrape the image tiles from NGA
2..stitch
3. process image
4.georef using mapcal
5.Convert to bsb using img2kap
6.upload to opencpn depository
100% agree to define a process early, but we also need to leave room for errors. I propose additional steps, which will require storage space and bandwidth:

Just in case it goes away, we should have a complete copy of all tiles on Index of /NGAViewer This would also help identifying changes on NGA side to trigger BSB chart updates. Is somebody able to define the size of all files in Index of /NGAViewer ?

There will be trials, errors, disagreements regarding the size reduction of the charts. I remember several revisions for the NZ BSB charts, similar revisions will happen.

After stitching, I think we need to keep the stitched charts in a loss-less format (is png ok?) on one of "our" servers.

We should also store the map-calibration data for each chart on the server, so anyone can easily reproduce a BSB chart after playing with i.e color spaces.

Not knowing ruby, I was able to make the ruby script work on a Gentoo box. One change needed was to add a memory limit to the montage:
Code:
`montage -limit memory 1024 -limit map 512 #{row_files.join(' ')} -geometry +0+0 -tile 1x#{tiles_high} #{filename}`
There seems to be a newer dezoomify.rb on
dezoomify.rb - maphew - A hewer of maps' small collection of tools I've built for myself, which may one day be useful for others - Google Project Hosting
but the newer version has some weird hard coded paths.

Could someone knowledgeable in Ruby, take the scripts and modify it to just take the chart number and output the stitched file into (chartnumber).png ? And please add a progress indicator while stitching, which does take a long time.

I am playing with the charts between Mauritius and Southern Africa, our next cruising grounds and have some stitched charts ready for upload and review.
LeaseOnLife is online now   Reply With Quote
Old 03-07-2011, 03:27   #19
Registered User

Join Date: Dec 2005
Location: Helsingborg
Boat: Dufour 35
Posts: 3,891
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by r.fairman View Post
......
6.upload to opencpn depository
.......


regards

Richard Fairman
Unfortunately there is no such thing.
The situation, presently, is pretty much the same as when the Pilot Charts were discussed.
OpenCPN.org has an informal bandwidth ceiling, which is the reason for the Pilot Charts being hosted, through "scubataz" generosity, on an external server.
It is also questionable if OpenCPN, as a project should get involved in this.
I think that we are better off doing this as a separate project.
So we need a "home" for ~2600 charts, say conservatively, 30-40 GB storage + bandwidth.

Thomas
cagney is offline   Reply With Quote
Old 03-07-2011, 04:49   #20
Registered User

Join Date: Feb 2010
Location: Tierra del Fuego
Boat: Phantom 19
Posts: 6,198
Re: Charts II: NGA - 2700 Charts

Thomas...
I can host this amount of data at least for some time until we find a nice permanent home for such data with no problem. Let me know if you decide to start this effort. Did not read much of this thread, but of course I can run the scripts directly on the server, so no need to repeatedly download and upload.

Pavel
nohal is offline   Reply With Quote
Old 03-07-2011, 05:43   #21
Registered User
 
Surveyor's Avatar

Join Date: Mar 2010
Location: Ft. Pierce, FL - Bislig, Philippines - Gladstone QLD
Boat: 1968 Alberg 30 #329
Posts: 451
Re: Charts II: NGA - 2700 Charts

I would like to participate in this effort. I downloaded and installed the programs to fetch the maps and all is working fine.

I am starting with 91004 and working down the list. For the time being I am only DLing raw files without stitching until the group decides on a standard. Once the standard is agreed to, I will stitch, convert or whatever and put the results up either in a torrent or or something like Depositfiles, whatever is best.

So DONT DOWNLOAD 91004 to 97564!!! When I am done for the day I will post where I stopped.

Speaking of that, it would be nice to somehow communicate to the group who has done what already. I think we need a Coordinator so that we are not wasting time duplicating efforts.

Please feel free to email or PM me.
__________________
Jim - Bahala na!
https://svcookie.blogspot.com/
KK4GGF
Surveyor is offline   Reply With Quote
Old 03-07-2011, 11:03   #22
Registered User
 
Surveyor's Avatar

Join Date: Mar 2010
Location: Ft. Pierce, FL - Bislig, Philippines - Gladstone QLD
Boat: 1968 Alberg 30 #329
Posts: 451
Re: Charts II: NGA - 2700 charts

Quote:
Originally Posted by allanpeda View Post
I installed Cygwin, Strawberry Perl, and ImageMagick. I used

wget -LNr Index of /NGAViewer/25609

to snag all the tile images before placing them in a single directory. Most interesting are the arguments passed to ImageMagick.

Stitcher I write is linked below:

http://goo.gl/4ZPX8
Allan, I am wondering if you wouldnt mind explaining how to use your stitcher. I have Cygwin, Strawberry Perl and Imagemagick installed but I am not having any luck with your script.

I have all of the zoom level 7 tiles in one directory and running the script from there.
__________________
Jim - Bahala na!
https://svcookie.blogspot.com/
KK4GGF
Surveyor is offline   Reply With Quote
Old 03-07-2011, 13:21   #23
Registered User

Join Date: Dec 2005
Location: Helsingborg
Boat: Dufour 35
Posts: 3,891
Re: Charts II: NGA - 2700 Charts

LeaseOnLife
Ok, a tweaked ruby script.
dezoom.rb.zip.doc
Hard coded for our purpose. Just give the chart number as argument.
Have a look at the python script. It is much more developed.

Thomas
cagney is offline   Reply With Quote
Old 03-07-2011, 14:17   #24
Marine Service Provider

Join Date: Feb 2010
Posts: 279
Re: Charts II: NGA - 2700 Charts

I get following error with your script

C:\Documents and Settings\Richard>dezoom.rb 54412
Visiting Chart 54412
Found image path Index of /NGAViewer/54412
Getting http://www.charts.noaa.gov/NGAViewer...oup4/6-0-0.jpg...
C:/Documents and Settings/Richard/dezoom.rb:86:in `initialize': No such file or
directory - /tmp/zoomify-6-0-0.jpg (Errno::ENOENT)
from C:/Documents and Settings/Richard/dezoom.rb:86:in `open'
from C:/Documents and Settings/Richard/dezoom.rb:86:in `block (3 levels)
in <main>'
from C:/Documents and Settings/Richard/dezoom.rb:76:in `times'
from C:/Documents and Settings/Richard/dezoom.rb:76:in `block (2 levels)
in <main>'
from C:/Documents and Settings/Richard/dezoom.rb:74:in `times'
from C:/Documents and Settings/Richard/dezoom.rb:74:in `block in <main>'

from C:/Documents and Settings/Richard/dezoom.rb:36:in `each'
from C:/Documents and Settings/Richard/dezoom.rb:36:in `each_with_index'

from C:/Documents and Settings/Richard/dezoom.rb:36:in `<main>'
r.fairman is offline   Reply With Quote
Old 03-07-2011, 14:45   #25
Registered User

Join Date: Dec 2005
Location: Helsingborg
Boat: Dufour 35
Posts: 3,891
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by r.fairman View Post
I get following error with your script

C:\Documents and Settings\Richard>dezoom.rb 54412
Visiting Chart 54412
Found image path Index of /NGAViewer/54412
Getting http://www.charts.noaa.gov/NGAViewer...oup4/6-0-0.jpg...
C:/Documents and Settings/Richard/dezoom.rb:86:in `initialize': No such file or
directory - /tmp/zoomify-6-0-0.jpg (Errno::ENOENT)
from C:/Documents and Settings/Richard/dezoom.rb:86:in `open'
from C:/Documents and Settings/Richard/dezoom.rb:86:in `block (3 levels)
in <main>'
from C:/Documents and Settings/Richard/dezoom.rb:76:in `times'
from C:/Documents and Settings/Richard/dezoom.rb:76:in `block (2 levels)
in <main>'
from C:/Documents and Settings/Richard/dezoom.rb:74:in `times'
from C:/Documents and Settings/Richard/dezoom.rb:74:in `block in <main>'

from C:/Documents and Settings/Richard/dezoom.rb:36:in `each'
from C:/Documents and Settings/Richard/dezoom.rb:36:in `each_with_index'

from C:/Documents and Settings/Richard/dezoom.rb:36:in `<main>'
It's done for "nix" as is. Try changing "/tmp" to a folder where the temporary download will take place. Perhaps just create "C:\tmp".
You need to change lines 78,100,105 and 112 in a text editor.
More things may pop up.

The Python script looks OK for win, but Dacust had some problems....
Thomas
cagney is offline   Reply With Quote
Old 03-07-2011, 14:50   #26
Marine Service Provider

Join Date: Feb 2010
Posts: 279
Re: Charts II: NGA - 2700 Charts

Thanks

I created a /tmp in C: and the script is scraping tiles as i am wrting this

regards

richard
r.fairman is offline   Reply With Quote
Old 03-07-2011, 15:48   #27
Registered User
 
Surveyor's Avatar

Join Date: Mar 2010
Location: Ft. Pierce, FL - Bislig, Philippines - Gladstone QLD
Boat: 1968 Alberg 30 #329
Posts: 451
Re: Charts II: NGA - 2700 Charts

I gave up on using Allan's wget scheme since I cannot get his stitch script to work.

Now after nearly 6 hours of problem solving I finally have the python script working on win. The only problem I can see with using this script is that you have to set a quality parameter and the author does not recommend anything over 95%. I am currently DLing 91004 at 90%. The default is 75%

EDIT - The python script is fairly fast. I redownloaded 91004 at 100% and the result was 10mb

I am going to continue with the Region 9 charts.
__________________
Jim - Bahala na!
https://svcookie.blogspot.com/
KK4GGF
Surveyor is offline   Reply With Quote
Old 03-07-2011, 16:45   #28
Registered User
 
Netsurfer's Avatar

Join Date: Jan 2010
Location: Cologne, Germany
Boat: Beneteau Oceanis 331
Posts: 557
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by cagney View Post
Unfortunately there is no such thing.
The situation, presently, is pretty much the same as when the Pilot Charts were discussed.
OpenCPN.org has an informal bandwidth ceiling, which is the reason for the Pilot Charts being hosted, through "scubataz" generosity, on an external server.
They are also available from opencpn.de for a long time ... .
The "problem/ question" is not where they are hosted (this doesn't matter in the end), but rather how and where the information about can be found.
There's still a big leck ...

Quote:
It is also questionable if OpenCPN, as a project should get involved in this.
What does this mean?
For my understanding there is a OpenCPN community in addition to the pure program's project. And imho the community should get involved.
Quote:
I think that we are better off doing this as a separate project.
So we need a "home" for ~2600 charts, say conservatively, 30-40 GB storage + bandwidth.
Why do you want to host them all on one (payed) server? There are so many free web-/ file hosting possibilities available that I really think that it would be a reasonable presumption to users to do 10 or 15 separate downloads (from different sources) instead of one (very large one).
Unless there are people like Nohal ...
Quote:
Originally Posted by nohal View Post
I can host this amount of data at least for some time until we find a nice permanent home for such data with no problem.
Gunther

BTW: I have several times offered to host OpenCPN related stuff at opencpn.info - no resonance so far ... .
But up to 40 GB that are downloaded a hundred times per month in average will cause too much traffic.
__________________
Deutschsprachige Community- und Support-Website unter OpenCPN.de
Netsurfer is offline   Reply With Quote
Old 03-07-2011, 17:30   #29
Registered User
 
LeaseOnLife's Avatar

Join Date: Apr 2008
Location: out cruising again, currently in Fiji
Boat: Sailboat
Posts: 1,466
Re: Charts II: NGA - 2700 Charts

Thomas: thanks for the updated ruby script!

Is jpg the format we want for the stitched image?

With jpg we already loose quality/information in the stitching step. And, as Jim/Surveyor asks, what setting for the jpg would be the right one?

I have been saving the stitched images as png, but that makes the files even larger.
LeaseOnLife is online now   Reply With Quote
Old 03-07-2011, 17:36   #30
Registered User

Join Date: Dec 2010
Boat: Catalina 30
Posts: 213
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by LeaseOnLife View Post
Thomas: thanks for the updated ruby script!

Is jpg the format we want for the stitched image?

With jpg we already loose quality/information in the stitching step. And, as Jim/Surveyor asks, what setting for the jpg would be the right one?

I have been saving the stitched images as png, but that makes the files even larger.
PNG is a very efficient compression scheme especially for large files, and you can achieve very good compression if you switch from RGB to a reduced set of indexed colors per brak's advice.

convert f1.miff f2.miff f3.miff (...etc) +append -quality 0 \
+dither -colors 32 -depth 8 -density 600x600 \
-units pixelsperinch outputfile.png

Compression and Filtering (PNG: The Definitive Guide)

I was able to reduce a 50MB file to under 6MB after taking brak's advice and dropping the number of colors to 32 in an (8 bit or less) indexed color space.
allanpeda is offline   Reply With Quote
Reply

Tags
charts

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Advertise Here


All times are GMT -7. The time now is 23:14.


Google+
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Social Knowledge Networks
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2024, vBulletin Solutions, Inc.

ShowCase vBulletin Plugins by Drive Thru Online, Inc.