Chart Downloader/Updater Plugin

[sbfreddie]

Originally Posted by transmitterdan

Freddie,

I don't think the depth of the chart folders should matter. Also, the number of charts downloaded at one go is unlimited. If it complains the Internet connection is down it most likely has something to do with the local IP stack. I had a wireless router that got upset once when downloading hundreds of files but power cycling it fixed the problem.

Under different circumstances I might agree, however how do you explain the same problem with 2 different Mac OS's (OS X 10.9.5 & OS X 10.10.4) and 2 different ARM Linux's on two different boards (Cubietrucks running Ubuntu 12.04 & Ubuntu 13.04 and Tegra-TK1 running Ubuntu 14.04)

Originally Posted by transmitterdan

Freddie,
I am running O build 705 from latest git master branch and the latest chartdldr_pi from Nohal's git repository. It has none of the reported symptoms on Ubuntu Linux or Windows 7/8.1. I can't imagine what is happening to your downloads but if it is the same on different computers and OS's I would suspect something external that is common to all.

Then please explain why I can download massive amounts of data (more than 100 Gigabytes) and thousands of files on these computers without so much as a single problem?

Thanks,
Freddie

[nohal]

Freddie...
I'm just at chart 400+ out of the 1119 US ENCs and can't reproduce the problem, the plugin happily continues through the set - same as during all the tests I performed while developing this plugin.
Anything else suspicious? Can you try to have a look at the connections using iptraf or monitor the failure with tshark?
What about

sudo lsof -i -n | grep opencpn

when the download fails - any stalled connections?
What comes to mind as one of the possible explanations is that you are hammering the NOAA download servers too much over your super fast connection and get temporarily blocked, maybe... Can you try downloading some other charts when NOAA refuses to continue? Or just wait a while and try again, without the restart? I really can't test this hypothesis as my top download speed from these destinations is under 2mbps. Anyway, this really does not look like a systematic problem in the plugin itself.

Pavel

[transmitterdan]

Freddie,

I'm just telling you that I can't reproduce your symptoms with the current master builds of O and the plug-in. I don't have the ability to test your exact builds on your exact OS setup. As you are the only one reporting this issue so far and I cannot reproduce it I don't know how to explain it.

As a test I just deleted all my NOAA RNC chart files and downloaded anew 1026 charts in one go. It worked the first time. I did the same test on Windows 7 and Ubuntu.

What I know from testing the plug-in with 10's of thousands of chart downloads is that if the plug-in reports it cannot create a connection it usually has something to do with the underlying OS IP stack or internet connectivity such as a router. I have on a few occasions experienced a WiFi router that refused to make certain connections but inexplicably would make other connections just fine. Rebooting the router fixed that issue. I have never found a case where it was the plug-in itself at fault. I suggested rebooting your router because it is a common thread amongst all your computers.

Please post the exact message you are getting from the plugin when it fails. Maybe someone else will also have this problem and we can learn more.

Dan

[nohal]

Freddie...
If you want to test the ban theory, you can play with something like this (The units for CURLOPT_MAX_RECV_SPEED_LARGE and CURLOPT_MAX_SEND_SPEED_LARGE are bytes per second)

diff --git a/src/wxcurl/base.cpp b/src/wxcurl/base.cpp
index f28723d..1dcc8cf 100644
--- a/src/wxcurl/base.cpp
+++ b/src/wxcurl/base.cpp
@@ -831,6 +831,10 @@ void wxCurlBase::SetCurlHandleToDefaults(const wxString& relativeURL)
 #else
         SetOpt(CURLOPT_ENCODING, "gzip,deflate"); //Save bandwidth by using compression
 #endif
+        SetOpt(CURLOPT_LOW_SPEED_LIMIT, 0);
+        SetOpt(CURLOPT_LOW_SPEED_TIME, 0);
+        SetOpt(CURLOPT_MAX_SEND_SPEED_LARGE, (curl_off_t)100000);
+        SetOpt(CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)100000); 
 
         if(m_pEvtHandler && (m_nFlags & wxCURL_SEND_PROGRESS_EVENTS))
         {

Pavel

[sbfreddie]

Originally Posted by nohal

Freddie...
I'm just at chart 400+ out of the 1119 US ENCs and can't reproduce the problem, the plugin happily continues through the set - same as during all the tests I performed while developing this plugin.
Anything else suspicious? Can you try to have a look at the connections using iptraf or monitor the failure with tshark?
What about

sudo lsof -i -n | grep opencpn

when the download fails - any stalled connections?
What comes to mind as one of the possible explanations is that you are hammering the NOAA download servers too much over your super fast connection and get temporarily blocked, maybe... Can you try downloading some other charts when NOAA refuses to continue? Or just wait a while and try again, without the restart? I really can't test this hypothesis as my top download speed from these destinations is under 2mbps. Anyway, this really does not look like a systematic problem in the plugin itself.

Pavel

Pavel:
I will print out these instructions to try next time I download all the RNC charts. Could you please give me a little more detail for these commands, like what they are looking for? I have never used these tools before so I'm not sure what I am looking for.
I was also thinking the same thing as you were, about the high speed hammering of the NOAA download servers. I have only found one server farm on the entire Internet the can max out my speed capabilities and that is Apple's developer downloads server. I regularly get between 7 megabytes to 12 megabytes per second there (download the 2-3 GB Xcode in about 3-5 minutes).
I will try these test next time I download all the files.

Thanks,
Freddie

[sbfreddie]

Originally Posted by transmitterdan

Freddie,

I'm just telling you that I can't reproduce your symptoms with the current master builds of O and the plug-in. I don't have the ability to test your exact builds on your exact OS setup. As you are the only one reporting this issue so far and I cannot reproduce it I don't know how to explain it.

As a test I just deleted all my NOAA RNC chart files and downloaded anew 1026 charts in one go. It worked the first time. I did the same test on Windows 7 and Ubuntu.

What I know from testing the plug-in with 10's of thousands of chart downloads is that if the plug-in reports it cannot create a connection it usually has something to do with the underlying OS IP stack or internet connectivity such as a router. I have on a few occasions experienced a WiFi router that refused to make certain connections but inexplicably would make other connections just fine. Rebooting the router fixed that issue. I have never found a case where it was the plug-in itself at fault. I suggested rebooting your router because it is a common thread amongst all your computers.

Please post the exact message you are getting from the plugin when it fails. Maybe someone else will also have this problem and we can learn more.

Dan

Dan:
Its not my router, switch or cable modem. The reason I can say this is that all I have to do is quit OpenCPN (without rebooting anything) and then restart it and the chart downloads can begin right were it hung up waiting for a response from the NOAA servers. This tells me its got to be the interaction with the NOAA server farm. I think that Pavels idea is probably the best explanation for my troubles, but these are only minor difficulties compared with my other problems.

Thanks,
Freddie

[sbfreddie]

Originally Posted by nohal

Freddie...
If you want to test the ban theory, you can play with something like this (The units for CURLOPT_MAX_RECV_SPEED_LARGE and CURLOPT_MAX_SEND_SPEED_LARGE are bytes per second)

diff --git a/src/wxcurl/base.cpp b/src/wxcurl/base.cpp
index f28723d..1dcc8cf 100644
--- a/src/wxcurl/base.cpp
+++ b/src/wxcurl/base.cpp
@@ -831,6 +831,10 @@ void wxCurlBase::SetCurlHandleToDefaults(const wxString& relativeURL)
 #else
         SetOpt(CURLOPT_ENCODING, "gzip,deflate"); //Save bandwidth by using compression
 #endif
+        SetOpt(CURLOPT_LOW_SPEED_LIMIT, 0);
+        SetOpt(CURLOPT_LOW_SPEED_TIME, 0);
+        SetOpt(CURLOPT_MAX_SEND_SPEED_LARGE, (curl_off_t)100000);
+        SetOpt(CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)100000); 
 
         if(m_pEvtHandler && (m_nFlags & wxCURL_SEND_PROGRESS_EVENTS))
         {

Pavel

Pavel:
I'm sorry but you have lost me here, I can see this is a patch for wxcurl/base.cpp but what am I supposed to do with it, and how do I use it?
Sorry for being so ignorant, but I am not familiar with wxcurl at all.

Thanks,
Freddie

[nohal]

Freddie...
If I knew what you are looking for, I would certainly tell you, I don't... But let's have a look at the normal operation:
lsof showing a download in progress from one of the NOAA servers:

nohal@knut:~/source/opencpn/plugins/chartdldr_pi/build$ lsof -i -n |grep opencpn
opencpn     433 nohal   13u  IPv4 575769      0t0  TCP 192.168.1.66:35039->129.15.96.11:http (ESTABLISHED)

Iptraf is a "graphical" tool showing you all your active connections (BTW, it is kind of educative to have a look to where your machines are connecting all the time without you even knowing why...), their state and activity, what you should see during normal operation is on the attached screenshot (the highlighted row is one of the NOAA downloads). If something interesting, or what, there will be during the problem, I have no idea - generally it would be bad if there were too many open and unused network connections waiting for something (like the final ACK packet from the server). What is too many is hard to say.
tshark is a packet inspection tool (actually a CLI version of Wireshark, which you have probably seen already). As we don't know what we are looking for, hard to suggest how exactly you have to use it to capture something interesting and not every single packet with the chart data...
If you would like to have a look at all the packets you exchange with the NOAA cluster of servers, you would use it as

tshark -i wlan0 "host www.charts.noaa.gov and port 80"

The problem is that we actually would like to see just the output of

tshark -V -i wlan0 "host www.charts.noaa.gov and port 80"

from the very moment when your downloads stop, not the gazillion of lines produced before...
All this is actually a bit too hardcore to perform using a "remote monkey", I would definitely first try the patch limiting the bandwidth usage from my previous post.

Pavel

[nohal]

Originally Posted by sbfreddie

Pavel:
I'm sorry but you have lost me here, I can see this is a patch for wxcurl/base.cpp but what am I supposed to do with it, and how do I use it?
Sorry for being so ignorant, but I am not familiar with wxcurl at all.

Thanks,
Freddie

This one is trivial, so if don't know how to apply it as a patch, open wxcurl/base.cpp, scroll to line 831 where the code will start to look familiar and add those 4 lines marked with "+" (without the "+" at the beginning). Then build and install the plugin as you did before.

Pavel

[transmitterdan]

Quite a lot of government, ISP and private sector servers have anti-DOS features. Some of these systems are not as smart as one would like. But in today's world they are a necessary evil. Without them surely they would become inaccessible to the 20-30% of legitimate traffic the rest of us depend upon. I can imagine as more individual users gain access to massive bandwidth they will run into these "throttling" systems thus partially negating that bandwidth. I know one of the chart servers had rudimentary and nonsensical defensive measures that had to be dealt with to make the plug-in work.

[fogmachine]

Like others, I occasionally re-test the plugin (on OSX) by deleting my US charts and re-downloading from scratch. Occasionally a few charts fail but succeed upon later retry. If it fails partway through it's usually something at the far end (congestion, etc); if it doesn't even start it's usually DNS lookup. Most people get DNS through the ISP (assigned via DHCP when they hook up) and some ISPs have dodgy DNS policies and proxies. Configuring OSX for OpenDNS eliminated most of these but YMMV.

[sbfreddie]

Dan, Pavel et all:
In my case the problem always occurs while the plugin is trying to make the initial connection. The "Not available" message shows up in all the windows. If I let it, the plugin will just sit there waiting for a long time before it finally gives the usual message in a pop-up window saying it can't find the server and to check the connections. I can, every time simply quit OpenCPN and then run OpenCPN again and the plugin will find the server and the problematic file with no problem right away.

My ISP is Comcast and the Mac Mini is not using DHCP, its on manual settings as all of my Mac's are. The two Tegra-TK1's are connected with DHCP and don't have any problems connecting to my router (Cisco Etherfast Router). I often download zillions of files during the system building process while creating new Operating Systems for these ARM boards. Never so much as a burp connected to the usual Linux servers.

I'm guessing that Pavel's throttling idea may be the answer to solve this download problem.

I still did not get an answer as to why the IENC inland waterways do not show up in OpenCPN until I manually create a separate place outside the ENC folder and then re-download using the plugin.
Any ideas?

Thanks,
Freddie

[transmitterdan]

Is the IENC folder issue a problem on all OS and platforms? I have never seen it on Windows or Linux. I put the IENC charts in the folder "recommended" by the plug-in.

[fogmachine]

Originally Posted by sbfreddie

Dan, Pavel et all:
In my case the problem always occurs while the plugin is trying to make the initial connection. The "Not available" message shows up in all the windows. If I let it, the plugin will just sit there waiting for a long time before it finally gives the usual message in a pop-up window saying it can't find the server and to check the connections. I can, every time simply quit OpenCPN and then run OpenCPN again and the plugin will find the server and the problematic file with no problem right away.

This is consistent with DNS issues - you are probably using Comcast's DNS service through your cable modem. You can easily try changing the Mac Mini to use OpenDNS - https://support.opendns.com/entries/...4-OSX-Yosemite which has helped me in the past.

[sbfreddie]

Originally Posted by transmitterdan

Is the IENC folder issue a problem on all OS and platforms? I have never seen it on Windows or Linux. I put the IENC charts in the folder "recommended" by the plug-in.

Dan:
Its seems the problem is only in ARM Linux Ubuntu & LXDE.

I tried the plugin on the Mac Mini and it does not seem to matter, but while downloading charts the Mac Mini got through all the RNC charts with no problem, but crashed OpenCPN three times try to download the ENC charts.
OpenCPN on my Mac Mini (OS X 10.10.4) is completely hosed. It won't render RNC charts no matter what scale.
Its got a large amount of issues that are probably related to Wxwidgets.

Thanks,
Freddie