Chart Downloader/Updater Plugin

[nohal]

All...
With great help from SethDart, this code got to a presentable shape and seems to be ready for a public beta test (= it is not completely broken on my development machines).
After you install and enable the plugin, a new tab called Chart Downloader will appear in the Charts section of the Toolbox allowing you to download and update the charts.
There is no need to download all the charts again - if you point the downloader at your existing ENC_ROOT or BSB_ROOT directory, it should be able to start managing it.

Any comments, especially describing what you find unintuitive are welcome as the target audience are obviously users with little computer skills.

You can get the packages for all the major supported platforms from https://sourceforge.net/projects/ope.../chartdldr_pi/
Or build from source at https://github.com/nohal/chartdldr_pi/

Have fun

Pavel

[rgleason]

I like it akready. What magjiciam you are pavel. Can't wait to try tomorrow!

Sent from my SCH-I545 using Cruisers Sailing Forum mobile app

[transmitterdan]

Pavel,

The file chartdldr_pi\buildwin\curllib.lib is missing from the nohal git repository it seems. This file seems to be needed to build Windows target. I am not sure how to build this from curl and libcurl.

[nohal]

transmitterdan...
Read the README When you get to "Windows Specific Libraries" you will soon know where to get it... I will not include the binary build dependencies in the source tree, it is a waste of space and bandwidth.

Pavel

[transmitterdan]

An amazing amount of progress in a short time. I got it running tonight. First a bug report:

The "USACE IENC Charts" seems completely broken. Try it and you should see the problem straight away. This is probably because that web site doesn't use simple file names like the NOAA web site. It requires some data to get the desired files.

Now for the "observations and opinions" section.

We need something that is simple for newbies. This tool is nice in that it gives complete freedom to choose which charts you want and what folder you want them in. But newbies don't want to click a hundred check boxes to get some charts. They don't understand the existing system for installing charts and now we are making them do even more clicking around.

What I think we need should be more automated toward the newbie but allow freedom for the experienced O user.

First, the "Chart directory" should have a default path. I think that should be wherever the program data is kept in a folder called "Charts" unless someone disagrees. When the user "Adds" a predefined catalog it should fill in the "Chart directory" with the "Charts" folder. The user could override this if they want but normally they won't until they get more experience.

Second, if the specified "Chart directory" folder does not exist then can the plugin create the folder after asking the user for permission? This would save a lot of time.

Third, after update is selected for a particular NOAA group of charts the check box should be automatically selected for all new or updated charts. This would save a lot of clicking.

Fourth, after the charts have been downloaded how hard it is to automatically "install" them without having to go back to the "Chart Files" tab and go through that ritual. It would be more intuitive to just click "Apply" or "OK" on the Chart Downloader tab and let 'er rip.

Anyway, these are my comments and observations.

[seandepagnier]

I have so many raster charts for many countries in the world. Is there a way any user can upload a raster chart to a server somewhere to share it? Can this integrate with bittorrent somehow?

[SethDart]

Originally Posted by transmitterdan

This would save a lot of time.

There's a context menu (right rlick) on the chart list box. You'll find several options like "Select new" or "Select updated"

[rgleason]

Git cloned, cmake.. etc and installed. I took a number of screenshots and will work on the wiki.

What a huge improvement! This is a big step for the beginning user.
Downloading and using that first chart!

Then using it, I can't explain enough how having accomplished that once, the user is empowered and more confident and feels they can do it. They will start using the program and then gradually learn the more manual process (or perhaps not).

Dan's comments are good I think, but this does not take away at all what you have both accomplished! Its huge.

I do think starting with a nice simple folder defaulting to "charts" or "Charts" is very good idea.

I do think that after a user has selected a catalogue additional directories should be added such as "Charts/ENC/MA" or "Charts/BSB/RI" etc. Note this is adding several directories in one shot.

A prompt to create the directories would also be good. What happens if they don't want the directories created? Prompt for path?

This leads nicely to the creation of directories before download. In Windows when establishing the path to download to, I was able to walk down a path to my charts directory then BSB then I wanted a new directory MA for the destination charts. I was able to right click and create a new folder! --I wonder how many new users would do that? Of course this is a basic Windows feature, but should there be a Create new directory button? after definition of the destination path?
The more automatic features mentioned earlier might be acceptable, but this might allow greater flexibility.

I have not fully tested the plugin, but the basic features seem to be working fine.
The order may be a little off here.. but
1. Select a Catalog
2. Download the Catalog
3. Select a Chart from the Catalog
4. Select a Destination Directory
5. Download the Chart to the Directory
6. Then under Chart Files tab "Add a directory" to the new charts
7. Then "Scan and Rebuild" or "Force a full database rebuild"

Now this is where I have problems! And I do not know if it is due to the new Plugin or not. I have not noticed it until now with v4.0.

I am getting full database rebuilds all the time when I "close" or "apply" no matter what, and what is worse it is rebuilding absolutely everything I have within my "Charts" folder!!!!!

Please understand, I had lots of charts under that folder, so many O would totally croake if I were to want them all in the database! But now O is scanning the ENTIRE folder, not just the specific ones I have listed in the "Chart Directories"

Also is does not seem like the scan is remembering that db has been recently scanned, it will scan the whole thing over again (I mean EVERYTHING under Charts) if I try to close the Chart Directory folder. This takes a very long time and uses lots of CPU and fans.

Now I will tell you a mistake I made this morning. I selected NOAA RNC MA and NOAA RNC RI and I downloaded them to Charts/ENC/MA and Charts/ENC/RI

I could try deleting those charts from within O and the plugin and then re download, but I am going to try moving them over to BSB. My computer seems to be working overtime scrolling, zooming and is generally causing CPU and FAN activity to get anything done.

I don't know if I can copy those over to BSB now without confusing things because I find xml files like MA_RNCProdCat.xml

Should I just delete and reload?

I am very excited about this plugin. I think it is the best new thing for New Users.

[transmitterdan]

Originally Posted by SethDart

There's a context menu (right rlick) on the chart list box. You'll find several options like "Select new" or "Select updated"

Ok, that's helpful. We are talking about newbies. They don't read help or manuals. They never will until they have the program doing something interesting. It's just how they are. The more we can do for them the better so long as it doesn't inconvenience the experienced O user too much. Given that these options exist then I see no reason not to automagically preselect new+updated charts. If that's not what they want they can just "deselect all" then select what they want.

Also, after thinking about this overnight here's another suggestion. It would be helpful if each set of charts from the same .zip file (e.g. RNC charts from FL) by default went into their own directory beneath the root "Charts" folder. The folder should have a name that is meaningful. This would be better than having one folder at the root for each chart. The reason is that it would make creating logical chart groups much easier. One could group a few states together quite easily by just selecting each state folder. As it is now the user would have to add over 100 directories to a group just for Florida.

I notice that the coast guard districts and regions are missing from the drop down. If I were to hazard a guess I would say most people would prefer to see the regions appear in the drop down first. Here again, it would be helpful if each region went into its own directory. This would make it easier to create groups by region. One could for example create RNC and ENC groups for the Pacific Northwest by downloading two folders and adding them to a group called PNW. The All ENC and RNC zip files should be last in the drop down I think. Those files are huge and don't lend themselves to logical grouping. They could be eliminated as far as I am concerned. NOAA would thank us if we did I think.

Again, thank you for moving this so quickly. Some of the suggestions I have made I feel pretty confident I can make to the code. If you don't have time I can take a crack at it.

[rgleason]

Well, I just cut and pasted the MA and RI directories along with the appropriate XML files from ENC to BSB, then went in and deleted the incorrect Chart "Directories" and added new ones.

On closing the Chart Directory Opencpn went into scanning my entire Charts folder again. It seemed to complete faster but it is scanning chart folders that are not even in the Chart Directory still!

When this High CPU operation finished, a message came up, "Changing Chart Folders this one is empty! Switching to All Active Charts" --Very nice.

When I try to move around, Opencpn is totally hampered by this hugh chart database that it created. Lots of CPU and FANS running, can't pan or zoom easily, can't do much. Opencpn stopps responding. I am going to have to restart and try turning off OpenGL.

Sean, I wouldn't try this yet on low powered, unless you point it to small folder of charts. (don't store unused folders under your "Chart" directory until this can be fixed.

[rgleason]

I just saw Dan's last comment about Regions, that is how I do my charts too. It would be very nice if that were an option.

[nohal]

Dan, Rick...
I'm not American, you tell me if CG districts or states make more sense. CG districts have one huge advantage - there's much less of them, so I hope it will be your answer...

Pavel

[bcn]

Originally Posted by transmitterdan

Ok, that's helpful. We are talking about newbies. They don't read help or manuals. They never will until they have the program doing something interesting. It's just how they are. The more we can do for them the better so long as it doesn't inconvenience the experienced O user too much. Given that these options exist then I see no reason not to automagically preselect new+updated charts. If that's not what they want they can just "deselect all" then select what they want.

Also, after thinking about this overnight here's another suggestion. It would be helpful if each set of charts from the same .zip file (e.g. RNC charts from FL) by default went into their own directory beneath the root "Charts" folder.

Dan,

when tackling the issue of the charts folders, please take the following environment into account (this is from a discussion we had with Dave during the S-63 implementation):

On 23/8/14 1:27, dsr wrote:> On 08/22/2014 07:44 AM, info-o-charts wrote:
>> Dave,
>>
>> up to now in the installation of OpenCPN we are dealing with
>> individuals, not corporations or organizations. Here we have a
>> different case:
>>
>> The following scenario is from an Australian college which wants to
>> use S63 charts. They will get the charts for Tasmania free from AHS
>> and are using as well NZ raster charts.
>> In total they want to equip 25 workstations from their 1200 machines
>> park.
>> These are all MACs under OS-X 10.8 with an upgrade planned for next year.
>>
>> Software deployment is done via images. They know that they will need
>> to install the InstallPermits and S63 charts individually.
>>
>> When a user logs in or out the user data are getting cleared or
>> restored into/from a hidden directory under root, so that the user has
>> always his machine and does not know about the history of the machine.
>> The problem is the limited size of the user data as you can see below.
>>
>> Under OS-X we have the S63 data under the user directory
>> (/Users/"user"/Library/Preferences) and hence this might grow a lot.
>>
>> I'm wondering if it would make sense to have the S63 data in the
>> application folder and not in the user folder as we bind the
>> InstallPermit to a system and not to a user.
>> Same with cm93 etc.
>>
>> The log, the navobj.xml, the .ini, all this is user related.
>>
>> In Windows all the data are under ProgramData, not user related.
>> And Linux is like OS-X IIRC
>>
>> What is your opinion?
>>
>> Hubert
>>
>>

>>>
>>
> Hubert...
>
> Ok, it seems to me that the main issue is with the S63 directory, where
> the S63 SENC files are stored. There is no reason to have per-user
> storage of these files, if we do a per-system license. The other
> per-user files are small potatoes.
>
> We already have a config file entry to relocate the core OCPN
> unencrypted (e.g. NOAA) SENC files. I note that at the moment, this is
> reported not working in current Beta. But the concept is clear. This
> was created to avoid multiple SENC file copies when running multiple
> instances of OCPN using the "portable" option.
>
> So, we need a similar entry for S63 content directory. Then, each
> student's config file could point to a system-common location, of the
> admin's choice. Of course, the permissions would need to be correct...
>
> Would this work?
>
> Dave
>

[rgleason]

I just restarted O and turned off the new ChartLdr plugin.
The machine was still operating on high, so I turned of OpenGL and that fixed it.

Then I went into Options > Chart Group and highlighted MA-RI which no longer has any directory references because I moved the files and xml to BSB folder and it is empty now.

I added the BSB/MA and BSB/RI directories to the MA-RI Chart Group.
I hit OK and there was not a big change in CPU, so the problem may be more related to the Opencpn OpenGL caching.

I note that Display > Chart Quilting gets turned off after setting up new Chart Groups.

Pan and Zoom and Chart operations are better now with OpenGL off, however when In Options > Chart Files > I check "Scan charts and update database" with BSB/MA and BSB/RI directories highlighted, it scans absolutely all my chart directories still. If nothing is highlighted it also scans all my directories. Is this really necessary? If I push "OK" the window closes without causing my CPU & FAN on high which is nice.

Dare I turn OpenGL back on? It was obviously rebuilding the cached and rebuilding the cache and rebuilding the cache etc.

This problem was not happening before and in fact the program was very quick and efficient with Opengl on.

I no longer think this problem I am having is related to the Chartdld!

[transmitterdan]

Originally Posted by nohal

Dan, Rick...
I'm not American, you tell me if CG districts or states make more sense. CG districts have one huge advantage - there's much less of them, so I hope it will be your answer...

Pavel

Coast Guard districts are ok. I think people would be happen with these as the only defaults in the drop-down. So long as they can go state by state using the generic form which we can explain in the documentation that will be fine. It would make the plug-in much cleaner.

Dan