Charts II: NGA - 2700 Charts

[r.fairman]

here is test6.kap for comapibility test I know i only used 2 ref points in Mapcal

https://docs.google.com/leaf?id=0B3S...4M2U4&hl=en_US

[dacust]

Originally Posted by cagney

...
A question, as I don't use mapcal. On the NGA charts, 4 positions are given very accurately, the 4 corners. That is why I, as a minimum uses these points as REFs. A simpple copy/paste then gives the PLYs. I use Gimp at 200% to get the pixel-count for these positions. Can you obtain the same accuracy in MapCal?

Thomas

Yes, you can zoom in and get to the pixel in MapCal.

-dan

[LeaseOnLife]

Originally Posted by r.fairman

here is test6.kap for comapibility test I know i only used 2 ref points in Mapcal

https://docs.google.com/leaf?id=0B3S...4M2U4&hl=en_US

Weirdest thing! I placed that chart in a separate directory and Fugawi doesn't even acknowledge a chart being in there. I have no idea... Could you do a rough 4 point cal, add some more values during the MapCal process and post it again?

[LeaseOnLife]

Originally Posted by cagney

Everyone using MapCal and mc2bsbh, in the end you have to check your header files manually.

Checking headers manually... Could we make the tools help us? Either mc2bsbh or imgkap could check for simple omissions / add some minimum defaults?

Mapcal / gimp&manual editing: Both are valid methods. For a square chart the gimp+edit is probably faster for someone used to it. Mapcal has the windows feel and some users might prefer that.

[svrevelations]

I've read this thread with great interest and thank you to all who is involved in this project. Although I do not as yet know or understand how the images are stitched or converted to Kap files, I wul like to somehow support this project.

Server Space
We have more than 900 Gigs available of webhosting on fast USA based servers. I am willing to provide offer all of this space for work-in-progress and for completed .kap files. We can also write a webpage where these files can be downloaded.

[bcn]

I've started with the Western Med - Region 5, 52000-52143 in the moment.
Testcase 52125 Approach Barcelona is running correctly under OpenCPN.

We have no other testing possibilities.

What size of .kap files is acceptable with the experiences gained up to now?
What influence has the DPI parameter in the header files?

Workflow:
- download dezoomify
- 50% resolution reduction Irfanview
- reduction of colors to 127, export PNG GIMP
- parameters for the headers with GIMP or Irfanview
- header files Notepad++ with OOcalc support
- imgkap for kap

Hubert

[Surveyor]

Just a note about my progress on the 9xxxx charts. I have 341 of the 484 downloaded and stitched in 100% resolution jpgs. When I have all 484 downed I will share them (raw) or convert and share. I don't want to do it twice so please decide where we want to store. Thanks.

[Netsurfer]

Hi ...

as the first tests have already shown the final kap file may vary depending on the used header file (e.g. number of REF's, REF's = PLY's or not) and the script for the conversion.

Surely we should find a general and unified way to create the kap files we'll going to host later on and asure that they will work at least in OCPN.

But with regards to the above my intention is to firstly share the downloaded (and stitched) JPG files.

So one will be able to (also) create kap files suiting one's personal needs.

Additionally other users may support the process by already creating appropriate header files (if we have defined how to create them).

Server Space
We have more than 900 Gigs available of webhosting on fast USA based servers. I am willing to provide offer all of this space for work-in-progress and for completed .kap files.

So the question of where to host them should be answered by this generous offer ...!

@MooseMoney:

We can also write a webpage where these files can be downloaded.

My proposal is to use the existing 'opencpn.xxx' websites for this (and any other site you like).
The more important question for now is how to upload all the files to your server?

@all:
It would be nice if we could define a general naming scheme for all possible parts like the tiles-, jpg-, header-, kap-files (for the upload to MooseMoney's server) ....!

Thanks
Gunther

[dacust]

Here are some quick thoughts on storage/download possibilities:

Free shared sites.

Free. Anyone can get to it and use with no special software.
Not permanent. Most of these sites require frequent visits or content is deleted. Many require waiting or hoops to jump through to download for free. No way to schedule downloads, so if you are downloading a large quantity, you have to set and watch your machine.

There ARE valid uses for these sites (see below).

Hosted site.

Someone pays for it. If they close their site, it may be gone. Bandwidth can be an issue since everyone is downloading from one spot.

Torrent

If enough people are interested, a torrent is free, load and storage are spread out, and if one person deletes theirs, the other copies are still there. But, if the interest is not wide enough, parts may disappear.

---

If you do not have an "always on" connection to the internet, put your downloads out on a free-share site. Then someone can download it to their own machine and create a torrent for it. It would work best for the torrents to not all be built by the same person, unless that person has a fast upload pipe. Someone with a server, however, would be a prime candidate for a single source.

Post the torrents on a site somewhere (I'll be glad to put that on my site). Post the torrents as many places as you like.

Then, additionally, anyone that wants to can post the downloads themselves on their site.

I would like to see downloads for the raw stitched files as well as the finished KAP charts.

I would prefer to see the downloads as torrents. I'd like to be able to schedule and bandwidth-prioritize my downloads with a tool (I use micro-torrent). I can pause anywhere, or choke the bandwidth while I watch a movie, then fire it back up when I leave for work or when I go to bed.

-dan

[cagney]

We need a uniform format for the raw files.
The python script, the way surveyor uses it, is a good candidate.
Download and stitch together a jpg just as it is.
To me it seems that running this script on MooseMoney's server, is a good way to get the files we still want to download.
Then opencpn.info can link to these files, and anyone can access the raw chart files for further processing.

I like Dan's suggestion about torrents. A well seeded torrent is a very fast download ....

I don't think that we should bother about all charts. Where there already is a good coverage of NZ charts, it would be a waste of time, for example.

Thomas

[LeaseOnLife]

Originally Posted by cagney

We need a uniform format for the raw files.
The python script, the way surveyor uses it, is a good candidate.
Download and stitch together a jpg just as it is.

Damn... and I used the ruby script. I guess I finish the download and stitch of Region 6, then switch?

Naming convention: anything is fine with me, easiest would be just the chart number. Do we need a prefix "NGA-"? Could be handled as a directory. There will be splits on the chartnumber for multiple charts on one page.

torrents: Great for distribution, not so great for syncing up on changes. I would prefer rsync or scp or even a revision control system

[Surveyor]

For what its worth, my raw stitched files are named 9xxxxRAW

Any script would do as long as it is scraping and stitching the origin files at a high resolution. The py batch script breaks often on low ram machines so I have switched gears and am now running py 1 at a time but on 3 machines :P

As a surveyor I was trained to be methodical and prudent. Well, the methodical part is coming in handy right now because I can't stop until I get these last few $%&*+ charts downloaded!

[svrevelations]

Originally Posted by Netsurfer

...........
The more important question for now is how to upload all the files to your server?

@all:
It would be nice if we could define a general naming scheme for all possible parts like the tiles-, jpg-, header-, kap-files (for the upload to MooseMoney's server) ....!

Thanks
Gunther

I have already contacted our hosting company and asked them to create a new ftp/shell user. Unfortunately, I am not clued about these things otherwise I would have done it myself. I hope to receive the ftp/shell username and password soon.

We pay our hosting company for 3 years in advance and we have just recently paid this account - unlimited web space and unlimited traffic. Of the unlimited available web space, I think 900 GB's allocation to this project will be helpful. Although our intention is to go sailing within the next 1 to 3 years - this server space will be available for at least the next 3 years. This will allow sufficient time and webspace to get things properly sorted out.

I also know that one can run PHP (and other) upload scripts allowing selected members to upload directly and individually to the server.

[Netsurfer]

Originally Posted by cagney

We need a uniform format for the raw files.
The python script, the way surveyor uses it, is a good candidate.
Download and stitch together a jpg just as it is.

I am doing this completely by PHP scripts. One for the download on my server and the other for stitching the tiles on my local system. Stitching 3.000 tiles together won't take a minute.
So imho there is no need to use a certain script for this task if you ensure to always use the highest quality level.

To me it seems that running this script on MooseMoney's server, is a good way to get the files we still want to download.

I thought this is the task we all are already on?

Then opencpn.info can link to these files, and anyone can access the raw chart files for further processing.

I like Dan's suggestion about torrents. A well seeded torrent is a very fast download ....

as addition to the other download source ...

I don't think that we should bother about all charts. Where there already is a good coverage of NZ charts, it would be a waste of time, for example.

Once we got a run don't stop half way ...

Originally Posted by LeaseOnLife

Naming convention: anything is fine with me, easiest would be just the chart number.

I agree.

Do we need a prefix "NGA-"?

No.

torrents: Great for distribution, not so great for syncing up on changes. I would prefer rsync or scp or even a revision control system

Are you talking about any future changes to the chart source images or what's the point for "a revision control system"?

Originally Posted by MooseMoney

I have already contacted our hosting company and asked them to create a new ftp/shell user.

Thank you very much. If I may help you in any way (as I am able to) feel free to contact me via PM or email.

Thanks
Gunther

[svrevelations]

The FTP/Shell user account is done.
I emailed the login details to "cagney".