Cruisers Forum
 


Join CruisersForum Today

Reply
 
Thread Tools Rate Thread Display Modes
Old 15-07-2015, 17:02   #331
Registered User
 
sbfreddie's Avatar

Join Date: Mar 2012
Location: Southern Texas, Port Isabel
Boat: I Wish
Posts: 164
Images: 1
Send a message via Skype™ to sbfreddie
Re: Chart Downloader/Updater Plugin

Quote:
Originally Posted by transmitterdan View Post
Freddie,

I don't think the depth of the chart folders should matter. Also, the number of charts downloaded at one go is unlimited. If it complains the Internet connection is down it most likely has something to do with the local IP stack. I had a wireless router that got upset once when downloading hundreds of files but power cycling it fixed the problem.
Under different circumstances I might agree, however how do you explain the same problem with 2 different Mac OS's (OS X 10.9.5 & OS X 10.10.4) and 2 different ARM Linux's on two different boards (Cubietrucks running Ubuntu 12.04 & Ubuntu 13.04 and Tegra-TK1 running Ubuntu 14.04)

Quote:
Originally Posted by transmitterdan View Post
Freddie,
I am running O build 705 from latest git master branch and the latest chartdldr_pi from Nohal's git repository. It has none of the reported symptoms on Ubuntu Linux or Windows 7/8.1. I can't imagine what is happening to your downloads but if it is the same on different computers and OS's I would suspect something external that is common to all.
Then please explain why I can download massive amounts of data (more than 100 Gigabytes) and thousands of files on these computers without so much as a single problem?

Thanks,
Freddie
__________________

__________________
sbfreddie is offline   Reply With Quote
Old 15-07-2015, 18:41   #332
Registered User

Join Date: Feb 2010
Location: On the go. Not in Prague.
Posts: 3,945
Re: Chart Downloader/Updater Plugin

Freddie...
I'm just at chart 400+ out of the 1119 US ENCs and can't reproduce the problem, the plugin happily continues through the set - same as during all the tests I performed while developing this plugin.
Anything else suspicious? Can you try to have a look at the connections using iptraf or monitor the failure with tshark?
What about
Code:
sudo lsof -i -n | grep opencpn
when the download fails - any stalled connections?
What comes to mind as one of the possible explanations is that you are hammering the NOAA download servers too much over your super fast connection and get temporarily blocked, maybe... Can you try downloading some other charts when NOAA refuses to continue? Or just wait a while and try again, without the restart? I really can't test this hypothesis as my top download speed from these destinations is under 2mbps. Anyway, this really does not look like a systematic problem in the plugin itself.

Pavel
__________________

__________________
nohal is offline   Reply With Quote
Old 15-07-2015, 18:44   #333
Registered User
 
transmitterdan's Avatar

Join Date: Oct 2011
Boat: Valiant 42
Posts: 3,479
Re: Chart Downloader/Updater Plugin

Freddie,

I'm just telling you that I can't reproduce your symptoms with the current master builds of O and the plug-in. I don't have the ability to test your exact builds on your exact OS setup. As you are the only one reporting this issue so far and I cannot reproduce it I don't know how to explain it.

As a test I just deleted all my NOAA RNC chart files and downloaded anew 1026 charts in one go. It worked the first time. I did the same test on Windows 7 and Ubuntu.

What I know from testing the plug-in with 10's of thousands of chart downloads is that if the plug-in reports it cannot create a connection it usually has something to do with the underlying OS IP stack or internet connectivity such as a router. I have on a few occasions experienced a WiFi router that refused to make certain connections but inexplicably would make other connections just fine. Rebooting the router fixed that issue. I have never found a case where it was the plug-in itself at fault. I suggested rebooting your router because it is a common thread amongst all your computers.

Please post the exact message you are getting from the plugin when it fails. Maybe someone else will also have this problem and we can learn more.

Dan
__________________
transmitterdan is online now   Reply With Quote
Old 15-07-2015, 19:21   #334
Registered User

Join Date: Feb 2010
Location: On the go. Not in Prague.
Posts: 3,945
Re: Chart Downloader/Updater Plugin

Freddie...
If you want to test the ban theory, you can play with something like this (The units for CURLOPT_MAX_RECV_SPEED_LARGE and CURLOPT_MAX_SEND_SPEED_LARGE are bytes per second)
Code:
diff --git a/src/wxcurl/base.cpp b/src/wxcurl/base.cpp
index f28723d..1dcc8cf 100644
--- a/src/wxcurl/base.cpp
+++ b/src/wxcurl/base.cpp
@@ -831,6 +831,10 @@ void wxCurlBase::SetCurlHandleToDefaults(const wxString& relativeURL)
 #else
         SetOpt(CURLOPT_ENCODING, "gzip,deflate"); //Save bandwidth by using compression
 #endif
+        SetOpt(CURLOPT_LOW_SPEED_LIMIT, 0);
+        SetOpt(CURLOPT_LOW_SPEED_TIME, 0);
+        SetOpt(CURLOPT_MAX_SEND_SPEED_LARGE, (curl_off_t)100000);
+        SetOpt(CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)100000); 
 
         if(m_pEvtHandler && (m_nFlags & wxCURL_SEND_PROGRESS_EVENTS))
         {
Pavel
__________________
nohal is offline   Reply With Quote
Old 15-07-2015, 19:27   #335
Registered User
 
sbfreddie's Avatar

Join Date: Mar 2012
Location: Southern Texas, Port Isabel
Boat: I Wish
Posts: 164
Images: 1
Send a message via Skype™ to sbfreddie
Re: Chart Downloader/Updater Plugin

Quote:
Originally Posted by nohal View Post
Freddie...
I'm just at chart 400+ out of the 1119 US ENCs and can't reproduce the problem, the plugin happily continues through the set - same as during all the tests I performed while developing this plugin.
Anything else suspicious? Can you try to have a look at the connections using iptraf or monitor the failure with tshark?
What about
Code:
sudo lsof -i -n | grep opencpn
when the download fails - any stalled connections?
What comes to mind as one of the possible explanations is that you are hammering the NOAA download servers too much over your super fast connection and get temporarily blocked, maybe... Can you try downloading some other charts when NOAA refuses to continue? Or just wait a while and try again, without the restart? I really can't test this hypothesis as my top download speed from these destinations is under 2mbps. Anyway, this really does not look like a systematic problem in the plugin itself.

Pavel
Pavel:
I will print out these instructions to try next time I download all the RNC charts. Could you please give me a little more detail for these commands, like what they are looking for? I have never used these tools before so I'm not sure what I am looking for.
I was also thinking the same thing as you were, about the high speed hammering of the NOAA download servers. I have only found one server farm on the entire Internet the can max out my speed capabilities and that is Apple's developer downloads server. I regularly get between 7 megabytes to 12 megabytes per second there (download the 2-3 GB Xcode in about 3-5 minutes).
I will try these test next time I download all the files.

Thanks,
Freddie
__________________
sbfreddie is offline   Reply With Quote
Old 15-07-2015, 19:34   #336
Registered User
 
sbfreddie's Avatar

Join Date: Mar 2012
Location: Southern Texas, Port Isabel
Boat: I Wish
Posts: 164
Images: 1
Send a message via Skype™ to sbfreddie
Re: Chart Downloader/Updater Plugin

Quote:
Originally Posted by transmitterdan View Post
Freddie,

I'm just telling you that I can't reproduce your symptoms with the current master builds of O and the plug-in. I don't have the ability to test your exact builds on your exact OS setup. As you are the only one reporting this issue so far and I cannot reproduce it I don't know how to explain it.

As a test I just deleted all my NOAA RNC chart files and downloaded anew 1026 charts in one go. It worked the first time. I did the same test on Windows 7 and Ubuntu.

What I know from testing the plug-in with 10's of thousands of chart downloads is that if the plug-in reports it cannot create a connection it usually has something to do with the underlying OS IP stack or internet connectivity such as a router. I have on a few occasions experienced a WiFi router that refused to make certain connections but inexplicably would make other connections just fine. Rebooting the router fixed that issue. I have never found a case where it was the plug-in itself at fault. I suggested rebooting your router because it is a common thread amongst all your computers.

Please post the exact message you are getting from the plugin when it fails. Maybe someone else will also have this problem and we can learn more.

Dan
Dan:
Its not my router, switch or cable modem. The reason I can say this is that all I have to do is quit OpenCPN (without rebooting anything) and then restart it and the chart downloads can begin right were it hung up waiting for a response from the NOAA servers. This tells me its got to be the interaction with the NOAA server farm. I think that Pavels idea is probably the best explanation for my troubles, but these are only minor difficulties compared with my other problems.

Thanks,
Freddie
__________________
sbfreddie is offline   Reply With Quote
Old 15-07-2015, 19:38   #337
Registered User
 
sbfreddie's Avatar

Join Date: Mar 2012
Location: Southern Texas, Port Isabel
Boat: I Wish
Posts: 164
Images: 1
Send a message via Skype™ to sbfreddie
Re: Chart Downloader/Updater Plugin

Quote:
Originally Posted by nohal View Post
Freddie...
If you want to test the ban theory, you can play with something like this (The units for CURLOPT_MAX_RECV_SPEED_LARGE and CURLOPT_MAX_SEND_SPEED_LARGE are bytes per second)
Code:
diff --git a/src/wxcurl/base.cpp b/src/wxcurl/base.cpp
index f28723d..1dcc8cf 100644
--- a/src/wxcurl/base.cpp
+++ b/src/wxcurl/base.cpp
@@ -831,6 +831,10 @@ void wxCurlBase::SetCurlHandleToDefaults(const wxString& relativeURL)
 #else
         SetOpt(CURLOPT_ENCODING, "gzip,deflate"); //Save bandwidth by using compression
 #endif
+        SetOpt(CURLOPT_LOW_SPEED_LIMIT, 0);
+        SetOpt(CURLOPT_LOW_SPEED_TIME, 0);
+        SetOpt(CURLOPT_MAX_SEND_SPEED_LARGE, (curl_off_t)100000);
+        SetOpt(CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)100000); 
 
         if(m_pEvtHandler && (m_nFlags & wxCURL_SEND_PROGRESS_EVENTS))
         {
Pavel
Pavel:
I'm sorry but you have lost me here, I can see this is a patch for wxcurl/base.cpp but what am I supposed to do with it, and how do I use it?
Sorry for being so ignorant, but I am not familiar with wxcurl at all.

Thanks,
Freddie
__________________
sbfreddie is offline   Reply With Quote
Old 15-07-2015, 19:59   #338
Registered User

Join Date: Feb 2010
Location: On the go. Not in Prague.
Posts: 3,945
Re: Chart Downloader/Updater Plugin

Freddie...
If I knew what you are looking for, I would certainly tell you, I don't... But let's have a look at the normal operation:
lsof showing a download in progress from one of the NOAA servers:
Code:
nohal@knut:~/source/opencpn/plugins/chartdldr_pi/build$ lsof -i -n |grep opencpn
opencpn     433 nohal   13u  IPv4 575769      0t0  TCP 192.168.1.66:35039->129.15.96.11:http (ESTABLISHED)
Iptraf is a "graphical" tool showing you all your active connections (BTW, it is kind of educative to have a look to where your machines are connecting all the time without you even knowing why...), their state and activity, what you should see during normal operation is on the attached screenshot (the highlighted row is one of the NOAA downloads). If something interesting, or what, there will be during the problem, I have no idea - generally it would be bad if there were too many open and unused network connections waiting for something (like the final ACK packet from the server). What is too many is hard to say.
tshark is a packet inspection tool (actually a CLI version of Wireshark, which you have probably seen already). As we don't know what we are looking for, hard to suggest how exactly you have to use it to capture something interesting and not every single packet with the chart data...
If you would like to have a look at all the packets you exchange with the NOAA cluster of servers, you would use it as
Code:
tshark -i wlan0 "host www.charts.noaa.gov and port 80"
The problem is that we actually would like to see just the output of
Code:
tshark -V -i wlan0 "host www.charts.noaa.gov and port 80"
from the very moment when your downloads stop, not the gazillion of lines produced before...
All this is actually a bit too hardcore to perform using a "remote monkey", I would definitely first try the patch limiting the bandwidth usage from my previous post.

Pavel
Attached Thumbnails
Click image for larger version

Name:	Screenshot from 2015-07-15 21-33-10.png
Views:	50
Size:	166.5 KB
ID:	105454  
__________________
nohal is offline   Reply With Quote
Old 15-07-2015, 20:02   #339
Registered User

Join Date: Feb 2010
Location: On the go. Not in Prague.
Posts: 3,945
Re: Chart Downloader/Updater Plugin

Quote:
Originally Posted by sbfreddie View Post
Pavel:
I'm sorry but you have lost me here, I can see this is a patch for wxcurl/base.cpp but what am I supposed to do with it, and how do I use it?
Sorry for being so ignorant, but I am not familiar with wxcurl at all.

Thanks,
Freddie
This one is trivial, so if don't know how to apply it as a patch, open wxcurl/base.cpp, scroll to line 831 where the code will start to look familiar and add those 4 lines marked with "+" (without the "+" at the beginning). Then build and install the plugin as you did before.

Pavel
__________________
nohal is offline   Reply With Quote
Old 15-07-2015, 22:05   #340
Registered User
 
transmitterdan's Avatar

Join Date: Oct 2011
Boat: Valiant 42
Posts: 3,479
Re: Chart Downloader/Updater Plugin

Quite a lot of government, ISP and private sector servers have anti-DOS features. Some of these systems are not as smart as one would like. But in today's world they are a necessary evil. Without them surely they would become inaccessible to the 20-30% of legitimate traffic the rest of us depend upon. I can imagine as more individual users gain access to massive bandwidth they will run into these "throttling" systems thus partially negating that bandwidth. I know one of the chart servers had rudimentary and nonsensical defensive measures that had to be dealt with to make the plug-in work.
__________________
transmitterdan is online now   Reply With Quote
Old 16-07-2015, 09:32   #341
Registered User

Join Date: Apr 2015
Location: San Francisco
Posts: 61
Re: Chart Downloader/Updater Plugin

Like others, I occasionally re-test the plugin (on OSX) by deleting my US charts and re-downloading from scratch. Occasionally a few charts fail but succeed upon later retry. If it fails partway through it's usually something at the far end (congestion, etc); if it doesn't even start it's usually DNS lookup. Most people get DNS through the ISP (assigned via DHCP when they hook up) and some ISPs have dodgy DNS policies and proxies. Configuring OSX for OpenDNS eliminated most of these but YMMV.
__________________
fogmachine is offline   Reply With Quote
Old 16-07-2015, 10:27   #342
Registered User
 
sbfreddie's Avatar

Join Date: Mar 2012
Location: Southern Texas, Port Isabel
Boat: I Wish
Posts: 164
Images: 1
Send a message via Skype™ to sbfreddie
Re: Chart Downloader/Updater Plugin

Dan, Pavel et all:
In my case the problem always occurs while the plugin is trying to make the initial connection. The "Not available" message shows up in all the windows. If I let it, the plugin will just sit there waiting for a long time before it finally gives the usual message in a pop-up window saying it can't find the server and to check the connections. I can, every time simply quit OpenCPN and then run OpenCPN again and the plugin will find the server and the problematic file with no problem right away.

My ISP is Comcast and the Mac Mini is not using DHCP, its on manual settings as all of my Mac's are. The two Tegra-TK1's are connected with DHCP and don't have any problems connecting to my router (Cisco Etherfast Router). I often download zillions of files during the system building process while creating new Operating Systems for these ARM boards. Never so much as a burp connected to the usual Linux servers.

I'm guessing that Pavel's throttling idea may be the answer to solve this download problem.

I still did not get an answer as to why the IENC inland waterways do not show up in OpenCPN until I manually create a separate place outside the ENC folder and then re-download using the plugin.
Any ideas?

Thanks,
Freddie
__________________
sbfreddie is offline   Reply With Quote
Old 16-07-2015, 10:46   #343
Registered User
 
transmitterdan's Avatar

Join Date: Oct 2011
Boat: Valiant 42
Posts: 3,479
Re: Chart Downloader/Updater Plugin

Is the IENC folder issue a problem on all OS and platforms? I have never seen it on Windows or Linux. I put the IENC charts in the folder "recommended" by the plug-in.
__________________
transmitterdan is online now   Reply With Quote
Old 17-07-2015, 08:47   #344
Registered User

Join Date: Apr 2015
Location: San Francisco
Posts: 61
Re: Chart Downloader/Updater Plugin

Quote:
Originally Posted by sbfreddie View Post
Dan, Pavel et all:
In my case the problem always occurs while the plugin is trying to make the initial connection. The "Not available" message shows up in all the windows. If I let it, the plugin will just sit there waiting for a long time before it finally gives the usual message in a pop-up window saying it can't find the server and to check the connections. I can, every time simply quit OpenCPN and then run OpenCPN again and the plugin will find the server and the problematic file with no problem right away.
This is consistent with DNS issues - you are probably using Comcast's DNS service through your cable modem. You can easily try changing the Mac Mini to use OpenDNS - https://support.opendns.com/entries/...4-OSX-Yosemite which has helped me in the past.
__________________
fogmachine is offline   Reply With Quote
Old 17-07-2015, 16:47   #345
Registered User
 
sbfreddie's Avatar

Join Date: Mar 2012
Location: Southern Texas, Port Isabel
Boat: I Wish
Posts: 164
Images: 1
Send a message via Skype™ to sbfreddie
Re: Chart Downloader/Updater Plugin

Quote:
Originally Posted by transmitterdan View Post
Is the IENC folder issue a problem on all OS and platforms? I have never seen it on Windows or Linux. I put the IENC charts in the folder "recommended" by the plug-in.
Dan:
Its seems the problem is only in ARM Linux Ubuntu & LXDE.

I tried the plugin on the Mac Mini and it does not seem to matter, but while downloading charts the Mac Mini got through all the RNC charts with no problem, but crashed OpenCPN three times try to download the ENC charts.
OpenCPN on my Mac Mini (OS X 10.10.4) is completely hosed. It won't render RNC charts no matter what scale.
Its got a large amount of issues that are probably related to Wxwidgets.

Thanks,
Freddie
__________________

__________________
sbfreddie is offline   Reply With Quote
Reply

Tags
charts

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
OpenCPN PlugIn Development bdbcat OpenCPN 70 14-03-2017 04:01
Plugin: DashBoard SethDart OpenCPN 470 24-08-2016 20:34
BSB 4 PlugIn bdbcat OpenCPN 93 20-02-2016 03:41
Logbook Plug-in SethDart OpenCPN 98 10-01-2013 11:47
Chartwork Plugin mario f OpenCPN 10 29-03-2012 19:21


Our Communities

Our communities encompass many different hobbies and interests, but each one is built on friendly, intelligent membership.

» More about our Communities

Automotive Communities

Our Automotive communities encompass many different makes and models. From U.S. domestics to European Saloons.

» More about our Automotive Communities

Marine Communities

Our Marine websites focus on Cruising and Sailing Vessels, including forums and the largest cruising Wiki project on the web today.

» More about our Marine Communities


Copyright 2002- Social Knowledge, LLC All Rights Reserved.

All times are GMT -7. The time now is 13:51.


Google+
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Social Knowledge Networks
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.

ShowCase vBulletin Plugins by Drive Thru Online, Inc.