Chart Downloader/Updater Plugin

[rgleason]

In cleaning up a big chart mess I made while testing ChartDownloader, I found I could keep the downloads, rename and relocate them and then go into the opencpn.ini file and edit the two sections

First [ChartDirectories] at the bottom of the plugins section to get the Settings > Charts > Chart Files to be correct, after deleting all the Chart Groups because those can just get recreated. Then scan and rebuild the database. It took several iterations because I made errors. I also tried to manually sort the list so that it would come up in the order I wanted. This is much better, however several entries have "reverted" or changed order and I am unable to determine why and have given up on that.
-- It would be very nice to be able to just sort this Chart List by the paths listed!

Second [Settings/ChartDnldr] Toward the bottom of the opencpn.ini After moving, deleting and rearranging my charts I changed the relevant pathnames which was quite successful. I updated those catalogs again and made sure it was working well. Then I tried changing the order of the catalogs manually in the opencpn.ini. Because this is entered into the opencpn.ini [Settings/ChartDnldr] area as one continuous line with no carriage returns for each catalog record it is very hard to edit and manually sort. What I did was put carriage returns in the right place, sort it, and then take the carriage returns out. I was reasonably successful however some records "jumped" to unexpected places and it is difficult to figure out what is causing this. Does the program do any sorting of its own or are these tiny format problems I don't see?
--Of course having some sorting capability would really help us determine if we have our Chart List parallel with our Chart Download list...and having a little more user friendly format in the ini file would also help.

Of course Chartdownloader provides a very very simple way for the new user to start using charts, just set your path to your chart foleder and follow the plugin defaults!

Thanks for a great Plugin!!
The New Chart user's favorite plugin.

[rgleason]

This is the order in the opencpn.ini file that I would like, but a number of these drop to places lower in the order, even after checking my format.

[ChartDirectories]
ChartDir1=C:\\Data-Dart\\Chart\\CM93ed2_2010May31^
ChartDir2=C:\\Data-Dart\\Chart\\USrnc\\CG-01^896836381966
ChartDir3=C:\\Data-Dart\\Chart\\USrnc\\CG-07^733484480377
ChartDir4=C:\\Data-Dart\\Chart\\USrnc\\LISoverlap^145950289760
ChartDir5=C:\\Data-Dart\\Chart\\USrnc\\RE-02^446371929683
ChartDir6=C:\\Data-Dart\\Chart\\USrnc\\RE-03^523429192192
ChartDir7=C:\\Data-Dart\\Chart\\USrnc\\RE-04^334626323061
ChartDir8=C:\\Data-Dart\\Chart\\USrnc\\RE-06^305913237716
ChartDir9=C:\\Data-Dart\\Chart\\USrnc\\RE-07^231554536246
ChartDir10=C:\\Data-Dart\\Chart\\USrnc\\RE-08^259652593963
ChartDir11=C:\\Data-Dart\\Chart\\USrnc\\RE-10^95413092822
ChartDir12=C:\\Data-Dart\\Chart\\USrnc\\ST-IL^90363256929
ChartDir13=C:\\Data-Dart\\Chart\\USrnc\\ST-MA^22781835001
ChartDir14=C:\\Data-Dart\\Chart\\USrnc\\ST-RI^5688844041
ChartDir15=C:\\Data-Dart\\Chart\\USrnc\\USAcoast^75343490842
ChartDir16=C:\\Data-Dart\\Chart\\USenc\\ACE-Inland^1823681087946
ChartDir17=C:\\Data-Dart\\Chart\\USenc\\CG-01^2142663330576
ChartDir18=C:\\Data-Dart\\Chart\\USenc\\CG-05^1874383028482
ChartDir19=C:\\Data-Dart\\Chart\\USenc\\INLAND^1782264576967
ChartDir20=C:\\Data-Dart\\Chart\\USenc\\RE-02^1230341243185
ChartDir21=C:\\Data-Dart\\Chart\\USenc\\RE-03^1588559176888
ChartDir22=C:\\Data-Dart\\Chart\\USenc\\RE-04^1035778385601
ChartDir23=C:\\Data-Dart\\Chart\\USenc\\RE-06^1059588981884
ChartDir24=C:\\Data-Dart\\Chart\\USenc\\RE-07^700784721198
ChartDir25=C:\\Data-Dart\\Chart\\USenc\\RE-08^678371843222
ChartDir26=C:\\Data-Dart\\Chart\\USenc\\ST-IL^116931993287
ChartDir27=C:\\Data-Dart\\Chart\\Other\\Argentina^217573390589
ChartDir28=C:\\Data-Dart\\Chart\\Other\\Austria^43642367306
ChartDir29=C:\\Data-Dart\\Chart\\Other\\Brasil^1301862508195
ChartDir30=C:\\Data-Dart\\Chart\\Other\\Columbia^1422920826
ChartDir31=C:\\Data-Dart\\Chart\\Other\\NZ^730419012866
ChartDir32=C:\\Data-Dart\\Chart\\Other\\I-AT^348975469182
ChartDir33=C:\\Data-Dart\\Chart\\Other\\I-BG^38457244586
ChartDir34=C:\\Data-Dart\\Chart\\Other\\I-CH^25646047136
ChartDir35=C:\\Data-Dart\\Chart\\Other\\I-CZ^55269930378
ChartDir36=C:\\Data-Dart\\Chart\\Other\\I-FR^638048762243
ChartDir37=C:\\Data-Dart\\Chart\\Other\\I-HR^40236751582
ChartDir38=C:\\Data-Dart\\Chart\\Other\\I-HU^2866520710
ChartDir39=C:\\Data-Dart\\Chart\\Other\\I-NL^110485429399
ChartDir40=C:\\Data-Dart\\Chart\\Other\\I-PL^16860768549
ChartDir41=C:\\Data-Dart\\Chart\\Other\\I-RO^1137464956475
ChartDir42=C:\\Data-Dart\\Chart\\Other\\I-RS^113218033450
[Settings]

These are sprinkled throughout down below in order:
ChartDir2=C:\\Data-Dart\\Chart\\USrnc\\CG-01^896836381966
ChartDir3=C:\\Data-Dart\\Chart\\USrnc\\CG-07^733484480377
ChartDir4=C:\\Data-Dart\\Chart\\USrnc\\LISoverlap^145950289760

These end up at the very bottom:
ChartDir5=C:\\Data-Dart\\Chart\\USrnc\\RE-02^446371929683
ChartDir6=C:\\Data-Dart\\Chart\\USrnc\\RE-03^523429192192
ChartDir7=C:\\Data-Dart\\Chart\\USrnc\\RE-04^334626323061
ChartDir8=C:\\Data-Dart\\Chart\\USrnc\\RE-06^305913237716
ChartDir9=C:\\Data-Dart\\Chart\\USrnc\\RE-07^231554536246

[nohal]

Rick...
These have nothing to do with the plugin... If you want some advanced reordering functionality in the main application, please create a task in Flyspray.
But to be honnest, I have a feeling that there are lot of more useful features to implement - which the users may sure prove wrong by voting in the tracker.

Pavel

[transmitterdan]

Rick,

When 4.2 goes live this will be less of a user issue. When adding catalogs the plugin will add the chart folder and trigger the database update automatically. So I don't think the average plugin user will get out of sync easily because they don't have to remember to add the chart folder.

Dan

[rgleason]

Yes that is true to some extent. Mine were quite disorganized due to testing. However the catalog list is basically in order of the date selected and does not have any relationship to path which makes it inherently disorganize IMO

Sent from my SCH-I545 using Cruisers Sailing Forum mobile app

[rgleason]

Good plugin though and I thank you guys for making it!

Sent from my SCH-I545 using Cruisers Sailing Forum mobile app

[cagney]

An unreliable internet connection has revealed a bug in the plugin.
I start to download, say 110 chars, in a CG district, the connection goes bad and I stop the download. Trying again, but the connection still appears to be down, the plugin quickly flickers through 109 charts and a popup says something like "bad connection for 109 out of 109 charts". This is a bit silly, trying the same bad connection 109 times. If the first download fails due to a bad connection, break, and give the user some sensible choices.
/Thomas

[transmitterdan]

Cagney,

Thanks for the input. It's a tradeoff depending on what is wrong. During testing on lots of different internet connection types and speeds we found that sometimes a file download didn't make it. Most of the time the next file download did make it. If the connection is truly down it should blow through the rest of the files in the queue pretty quickly as you experienced. But we felt that it was better to try every file in the queue because most of the time only one or two files didn't make it through on intermittent connections. Running the update again should almost always get the few files that didn't make it. If we abort the entire queue when one file fails there are some slow/intermittent internet connections that will never be able to get all the files. The way it is now it usually takes no more than two manual retries to get all the files.

You can hit the Abort button any time and it should stop without trying all the chart files remaining in the queue.

The code is available on github if you want to look at it and suggest a patch.

[nohal]

Well... I actually don't like the pop-up download dialog at all. IMO the ideal solution is to integrate the download progress indication into the chart list, which would also allow us to intuitively enable multiple download threads at the same time - sure good on high(er) speed connections... But to be honest it is pretty low on the priorities list right now.

Pavel

[StargazerWA]

I have been using Chart Downloader for several weeks now and can report that it works great with the NOAA charts. I update often so that it doesn't take long.

Thank you for developing this!

[cagney]

Originally Posted by transmitterdan

Cagney,
...

You can hit the Abort button any time and it should stop without trying all the chart files remaining in the queue.
...

No I can't, as it flashes past so quickly that you don't see any buttons.
A hotkey would help....

Apart from this, mostly due to an unpredictable, faulty cable modem, the plugin is great and will save a lot of manual work.

Thanks!

/Thomas

[sbfreddie]

Dan:
I have been using this plug-in for a while now and I sure like it a lot!
However I have found a weird problem which I can't figure out. When I download all the charts available (into the default folders) from US sources, the IENC Inland Waterways get installed automatically into the ~/Charts/ENC/US_INLAND subdirectory along with the other NOAA charts which are here ~/Charts/ENC/US. It seems that OpenCPN cannot find the US_INLAND charts when they are buried so deep in subdirectories. When I remove them from ~/Charts/ENC/ and put them into their own folder ~/Charts/US_INLAND, OpenCPN finds and displays them with no problem. Is this a problem with O or the plugin?

Another small thing when downloading all the RNC and ENC charts with the Downloader Plugin, it seems that the maximum amount of charts that it can download without dying is 368. It always stops saying there is no internet connection, yet all my other computers are still working fine. If I quit OpenCPN and start it back up I can continue on my way downloading more charts.
I have a 80 megabit cable modem that is very reliable and can download massive amounts of data (more than 30 giga bytes) without so much as a single burp! Why can't this plugin download a few hundred mega bytes without hanging?

Thanks,
Freddie

[nohal]

Freddie...
The plugin itself has nothing to do with OpenCPN's ability to find the charts or not. What do you have configured in the chart directories?
I suppose you are testing it on a Mac, which is exactly the platform where I can't test it at all. I have seen no problems downloading any number of charts under neither Linux nor Windows. To understand better - you say that once the plugin stops downloading the charts (is it always the 368th?), it does not ever download any more unless you restart OpenCPN completely?
Finally, with which OpenCPN version are you testing? And are you testing your own build from source or the "official" package?

Thanks

Pavel

[sbfreddie]

Originally Posted by nohal

Freddie...
The plugin itself has nothing to do with OpenCPN's ability to find the charts or not. What do you have configured in the chart directories?

Pavel:
I apologize for not giving you all the needed information.
Currently I am using two Tegra-TK1's (ARM 7i quad core) running Ubuntu 14.04 and a Mac Mini (Late 2009 running OSX 10.10.4). OpenCPN 4.1.602 on one Tegra_TK1, OpenCPN 4.1.630 on the other Tegra-TK1, OpenCPN 4.0.0 on my Mac Mini.

Currently my chart directory looks like this:
~/Charts
~/Charts/ENC
~/Charts/ENC/US
~/Charts/RNC
~/Charts/RNC/NZ
~/Charts/RNC/US
~Charts/US_INLAND

Originally Posted by nohal

Freddie...
I suppose you are testing it on a Mac, which is exactly the platform where I can't test it at all. I have seen no problems downloading any number of charts under neither Linux nor Windows. To understand better - you say that once the plugin stops downloading the charts (is it always the 368th?), it does not ever download any more unless you restart OpenCPN completely?
Finally, with which OpenCPN version are you testing? And are you testing your own build from source or the "official" package?

Thanks

Pavel

I am testing on three different systems, Mac Mini, and 2 Tegra-TK1's. The Mac is using the official release from the OpenCPN downloads website, and the Tegra's are built from the latest master at the time of building.

The plugin does not always stop at the 368th file, but it has never gotten past that number on any of these computers. Once it stops with the "not available" message it never starts up again, I have to restart OpenCPN and then I can continue.

Thanks,
Freddie

[transmitterdan]

Freddie,

I don't think the depth of the chart folders should matter. Also, the number of charts downloaded at one go is unlimited. If it complains the Internet connection is down it most likely has something to do with the local IP stack. I had a wireless router that got upset once when downloading hundreds of files but power cycling it fixed the problem.

I am running O build 705 from latest git master branch and the latest chartdldr_pi from Nohal's git repository. It has none of the reported symptoms on Ubuntu Linux or Windows 7/8.1. I can't imagine what is happening to your downloads but if it is the same on different computers and OS's I would suspect something external that is common to all.