OpenCPN.org and the Pilot Charts

[cagney]

As you all know, South Bay Network kindly and generously, donates opencpn.org web space and bandwidth.
The success of the new feature - Pilot Charts as OpenCPN charts - may just strain this, much appreciated, hospitality a bit.
The fact is that since the Pilot Charts were published, the bandwidth used by the OpenCPN project is more than three times greater than previously.
Another way to see it, is that we now use about 50% more than the limit that SBN is comfortable with.
This gives the the OpenCPN project some hard choices. It boils down to this.
* We can withdraw the download feature of the Pilot Charts from the opencpn.org site, and trust that users can find other download facilities.
The CNF.SeaSoft site already has the Pilot Charts available for download
* We can try to find other ways to host this resource hog, the files are huge. This is where we need ideas and suggestions from all OpenCPN users.
* A third, in my mind not very attractive option, is to somehow finance the OpenCPN.org with advertising.

All the web statistics for opencp.org is available here https://opencpn.org/plesk-stat/webstat/

So.... are the Pilot Charts worth keeping ......??

[nohal]

Thomas,
what about some kind of a mirror system where the expenses could be shared? The code needed to select a mirror will be couple of lines. I'm ok with providing some bandwidth from my servers...
Another option is hosting the files on sourceforge.net - I have seen bigger downloads there.

Pavel

[idpnd]

Is it possible to host on sourceforge? whats the pilot chart license vs sf.net license requirements?

[cagney]

I really had to do some quick changes to the download links, that will work for now.
Sourceforge is an option, but then really only if we start a new project, "Open source chart conversion tools", or something like that. I'm not to keen on this alternative.
Dave has good reasons for not wanting to use the OpenCPN project for these kind of files.
It will be interesting to see what kind of user reactions we will get, if any.... as a result of this change.

Thomas

[marshmat]

We can try to find other ways to host this resource hog, the files are huge. This is where we need ideas and suggestions from all OpenCPN users.

May I suggest offering the charts as a .torrent instead of a direct download? Host one seed copy on the server (throttling the bandwidth to whatever your ISP wants), and rely on the users to seed additional copies when they're on connections that can handle it.

The fact that media pirates like to use the Bittorrent protocol doesn't change the fact that it's a reliable, robust system designed for precisely the kind of situation OpenCPN is facing with these chart downloads (large data files, distributed to many users, by an organization with no money or bandwidth of its own).

I definitely want to see the charts kept... they can be quite hard to find and mixed with malware links at times. If OpenCPN can't host them itself, let all the users who want them share a bit of their own bandwidth to keep them available.

DepositFiles works, but is insanely annoying. I'd be happy to create the torrents for these charts if it's deemed desirable. It will take a few days to get them off DepositFiles before I can do so....

[marshmat]

OK, let's give this a try.

Here's the torrent file for the pilot charts:
http://www.marsh-design.com/sites/de...N_2010.torrent

The files included are:
PilotCharts_IO.zip
PilotCharts_NAC.zip
PilotCharts_NAN.zip
PilotCharts_NAW.zip
PilotCharts_PDF_IO.zip
PilotCharts_PDF_NA.zip
PilotCharts_PDF_SP.zip
PilotCharts_SP_Charts.zip
PilotCharts_SP_Info.zip
The complete set is 583 MB. The BSB charts alone are 344 MB if you omit the PDF sets (which can be obtained, one page at a time, from the NGA website).

I'm seeding the thing now. You'll need a Bittorrent client to open the torrent file and begin the download. It will be pretty slow for the first few people, until a "swarm" of complete seed copies can form.

If it works, you'll need to leave the torrent open for a few days after you finish the download, so that your computer can help seed it to everyone else.

For those unfamiliar with Bittorrent: More about how it works here on Wikipedia... or just download a BT client like Vuze or uTorrent, click the link above to open the .torrent file, and wait.

[dacust]

Originally Posted by marshmat

I'm seeding the thing now. You'll need a Bittorrent client to open the torrent file and begin the download. It will be pretty slow for the first few people, until a "swarm" of complete seed copies can form.

I am downloading it. Says 2 hours 13 minutes to go.

I can leave it active indefinitely, so it should help other in the future.

-dan

[dacust]

Seeding, now. But no takers, so far.

-dan

[scubataz]

Hi,

In the stats does that traffic amount include the downloading of the chart files, as 100GB a month doesn't seem that high

If it does include the chart downloads, I can host the download files until it goes over 200gb per month

[marshmat]

I think the August stats include the charts- hence the jump from ~20 to ~100 GB/month. That's not much by commercial standards, but a big deal to a FOSS project using donated server resources.
If you could host a seed of the torrent, scubataz, that'd be great

[scubataz]

VPS and torrents do not go well together, hits the io too much in the shared environment, best I can offer is a full file download spot for you with ftp access and http access for people to download

[cagney]

Thanks marshmat for setting up the torrents. I'm downloading now (ktorrent) and will seed later on.
scubataz, thank you very much for the offer, seems great to me.
I would like to link to the torrent and really to as many download sites as possible on the opencpn.org download side , that are not

insanely annoying.

to the users.
Is that Ok with your torrent Matt?
scubataz, send me the details, I'll add a link. Can you throttle the bandwidth yourself?

Ooops, very quick download, already seeding

Thomas

[scubataz]

I will send a pm with the ftp download details, do you want to setup a subdomain pointer to the download site eg: mirror-01.opencp.org etc?

I can throttle bandwidth if need be, will keep an eye on it and see how it goes

[barnakiel]

filehippo? brothersoft? softpedia?

private hosting?

Why cannot the Pilot Charts files be had right from the source (NOAA?)

barnie

[marshmat]

Re. torrents: Just put a copy of the .torrent file (about 24 kB) on whatever Web page you want. The actual data transfer is entirely peer-to-peer and they become self-sustaining with no server involvement once a handful of people complete the download and start seeding.

Some people may prefer a direct download (thanks scubataz for offering to host one)- perhaps best to link to this, but advise that it's a secondary choice so as not to overload scubataz's server too?

I hope we can offer more charts this way in the future.

Barnie: The charts can be had directly from the NGA as free downloads, but in formats that can't be integrated into OpenCPN. The ones in the torrent (see post 6) were repackaged by the OpenCPN team into a more useful format (BSB raster).