Sat2Chart

[Be Free]

It was meant as a compliment. Some developers think that their way is the only way to solve a problem. I admire your viewpoint that your tool is a very good way to build charts but not the only way and not even the best way for everyone.

It reminded me of the motto of the Pearl community, "There's more than one way to do it.". It's been a long time since I've coded in Pearl and even longer since I've used Rexx. Good memories.

[phiggins]

Sat2Chart.V16.0.10.0 is now available: fixes the tutorial links.
You can download it here: https://www.gdayii.ca/Downloads/Sat2Chart_Install.exe

[Miyalk]

Hi, I am struggling with the downloads from SASPlanet, I have everything set up and working as per Paul's instructions and my initial tests of very small area maps worked fine, but anything larger and the downloads from SASPlanet are so slow that eventually Sat2Chart will crash.

I have a fast fibre connection at home, speed testing even while it's downloading and I am getting nearly 100Mbps. So it appears its either a software issue or a server issue, the computer is new, i5 processor & 16GB RAM, not running anything other than the associated programs needed for Sat2Chart.

As an example the 20 splits I was trying to download for a chart were all around 1gb each and after about 36h they had downloaded less than 50% and then Sat2Chart crashed.

I have restarted now and the splits are saying roughly 24hrs to go to download the 1gb or so of each one. I concurrently started downloading a 1gb test file and it is saying less than 40 minutes to complete! So its pointing to an issue with either the software or the SASPlanet servers.

Does anyone have any suggestions I could try? (I'm an Apple user normally, I never use Windows and only set this up specifically for creating charts for OpenCPN.)

[phiggins]

Miyalk,

With the information you gave me I can't tell what the problem may be.
Are you using a polygon or path? If so send it to me and also tell me what map(s) you were downloading and what zoom levels. I'll try it.

My email is in the Help.

Paul

[Jon Hacking]

Hi Miyalk. First off, welcome to CF! We hope you find it a good experience.

As to your slow download issues, I'm Paul's performance tester. I would suspect your anti-virus program. You say it's a new computer (indeed, it's much like what I use for Sat2Chart work).

Did it, perhaps, come with McAfee or some other form(s) of crap-ware? McAfee & other anti-virus companies like to PAY computer manufacturers (or dealers) to put their products on new computers, with the hope that, when the free version wears off in 3-6 months, they can rope those users in to paying for it. (Yes, this is very different from Apple.)

The truth is that for many years, Windows has shipped with it's own extremely competent firewall & anti-virus program, called Defender. If you have McAfee or some other anti-virus program running, it could be examining EVERY file that SAS tries to download, to see if it's a potential problem. I've had a similar problem even with Defender when I tried to convert my SAS cache from the (default) file-system to SqLite, but normally, Defender doesn't get in the way of SAS downloads.

To start with, try the following:

1. Remove any anti-virus program except Defender using Add-Remove Programs (search for this with the magnifier in the lower left)
2. Most of these programs will have multiple components. Make sure that you remove them ALL.
3. Reboot your computer (in the lower left: Start, Power, Restart)
4. Use the magnifying glass to search for Defender (also apparently called Windows Security)
5. Start Defender & make sure that it's properly updated (it won't have been updating itself if some other anti-virus program is running)

See if that works. Also (& this is starting to get into expert territory):

1. Before SAS starts to download, right-click on an unused part of the task-bar at the bottom of your screen & select Task Manager. It might start as only a small window, but there's a way to get a bigger display (I'm sorry, I forget what it is, as I can't make my version go back there, but click around & you should find it. But don't make it full-screen, leave it as just a window).
2. Click on the Performance tab to see graphs of what the computer is doing.
3. Click on CPU to show the CPU utilization graph (if it's not already showing CPU utilization)
4. Right click on the graph itself, select Change graph to, & select Logical Processors to see how busy each of your CPU cores is.
5. Position the Task Manager window down near the bottom right of your screen, under where Sat2Chart is, so you can see it as Sat2Chart & SAS start working.
6. Start your download, & see how the graphs change.
7. If your CPU gets saturated (graph shows 100% or close to it) then switch to the Process tab
8. Click on the top of the CPU column, to sort by that column, & see which process is hogging the CPU (note that it should start out sorted by CPU utilization, in which case clicking on the top will invert the sort - just click on the top of the column again to sort from high-to-low)
9. If the CPU usage is OK but something else is saturating (like Networking) then click on the top of that column & see which program is using all of that resource

This should start narrowing in on the problem.

May I ask what area(s) you're trying to make charts for?

[Miyalk]

Originally Posted by phiggins

.
Are you using a polygon or path?

Paul

Polygon. I am back on the boat today, not at home with the computer. I will follow up when I get home again.

[Miyalk]

Originally Posted by Jon Hacking

As to your slow download issues, I'm Paul's performance tester. I would suspect your anti-virus program. You say it's a new computer (indeed, it's much like what I use for Sat2Chart work).

1. Before SAS starts to download, right-click on an unused part of the task-bar at the bottom of your screen & select Task Manager. It might start as only a small window, but there's a way to get a bigger display (I'm sorry, I forget what it is, as I can't make my version go back there, but click around & you should find it. But don't make it full-screen, leave it as just a window).
   .
   .

This should start narrowing in on the problem.

May I ask what area(s) you're trying to make charts for?

*Only Windows Defender, I am pedantic about removing anti-virus, viruses!

*I will have a play in Task Manager.

*I have charts for Queensland & NT that I was given by another sailor, I am adding charts for WA, Kimberleys. I have chartgroups for AUS, GIS, Google, Bing, C-Map & Navionics. At this stage I am just trying to create the mbtiles for GIS for the polygon I have created.

Like I said I have managed to create a smaller mbtile successfully so I know my process is working, its just the painfully slow downloads that are causing issues.

Thanks to Paul & you for responding promptly.

[Franziska]

AFAIK one can set the number of concurrent tile downloads in SASplanet. By default, it's 2 or 4 or some low number. If you allow it to download 20-30 in parallel, it's much much faster.
I'm not sure how to make this work with Sat2Charts, though.

[Miyalk]

Ok, came back home after a night on the boat to find Sat2Chart had crashed again, couple of obvious error messages, root problem still seems to be painfully slow upload rates from the SASPlanet servers.

As an experiment I changed from AIS to Navionics Sonar charts, using the same polygon for the Kimberleys and started with the lowest zoom I wanted only, (10), it had no problem as you would expect, only taking a very short time to download. Then I did Z11, worked fine, now doing z12 &13 concurrently, obviously slower, but looks like it will complete. I will continue from here up to Z18 individually and then I think I can merge the mbtiles, so at least it will be seamless in my chart group. Bit of PITA if I have to do that for all zooms of all chart types but at least its a work around at this stage!

Update - Trying to do z12 & 13 together is not working, throwing errors again.

Oh wel, I will keep trying one zoom level at a time, but its getting rather annoying.

[Miyalk]

Just had a look with Task Manager, really just confirms its SASPlanet servers, no real load on anything, downloading 20 splits at under 1Mbps according to the Network tab on Task Manager, if I start an OOKLA speedtest it jumps up to 95Mbps!

I am starting to think it's not worth bothering with, especially now with Starlink I don't really need the data stored locally anyway.

[Jon Hacking]

Originally Posted by Miyalk

Just had a look with Task Manager, really just confirms its SASPlanet servers, no real load on anything, downloading 20 splits at under 1Mbps according to the Network tab on Task Manager, if I start an OOKLA speedtest it jumps up to 95Mbps!

I am starting to think it's not worth bothering with, especially now with Starlink I don't really need the data stored locally anyway.

Yeah, Miyalk, 1Mb/s is pretty slow for multiple downloads, but ...

• What was your CPU utilization when doing your downloads?

• Were ALL of your cores behaving the same?

• Did you look at the Processes tab on Task Mangler to see which process was using all of your CPU cycles?

Something is getting in the way of your downloads. I regularly run 8-10Mb/sec for my downloads, even on a miserable tethered LTE connection. You'll want to sort this out, or it will always be slow. And it looks like that slowness is causing Sat2Chart to crash as well.

SAS itself doesn't have servers per se. The program on your computer is simply a conduit to the various servers that DO host the content (Bing, GE, etc), much like a web browser. That is, your SAS program is getting its content from the native sources, not from some SAS server (which would introduce all sorts of problems). If you're interested, look under the \SASPlanet\Maps\ folder to see all the descriptions of how it does this for each different map type.

The reason Paul put in the multiple SAS instances & 20 download threads per instance is because SAS itself is very slow when doing automated downloads. It seems to insert artificial delays in its download threads. If you simply fire up SAS & move it across your area of interest, you should see that the tiles fill in pretty quickly (& you should see your network utilization spiking up as well). But if you watch what happens with only a single download thread, you can see that it goes much slower than what it's capable of. We don't know why SAS does this (& all of the support is in Russian) but it is what it is, so we have to work around it.

Of the servers I commonly use (ArcGIS, Bing, GE, CMap, Navionics) Bing is blazingly fast. GE can be fast, but after a while it decides that it doesn't like all of these requests from a single source, & it simply refuses to supply tiles to that IP address for several hours (which seems like a violation of Google's "First, do no evil" mantra). Navionics is by far the slowest.

BTW, Is CMap_4D or 4D_Extra working for you? They've been offline for a while, & still don't seem to be working for me (here in Argentina). Or are you just using CMap_Basic?

Starlink is pretty cool! We've just bought a system ourselves (they're cheap here in South America), but I haven't installed it on Ocelot yet. If you want mounting suggestions or other help, send me a PM & we'll take it offline. Our son used to work there, & I've been following it for a while (I'm an EECS, & used to be a Microwave Tech).

But I wouldn't want to rely on Starlink for my navigation. It's a pretty complex system, with lots of opportunities for something to break down. While I'll be using ours for downloading charts, I'll do it ahead of time, & then back those charts up to 2 external drives, before I head out. So while I realize that you may have other priorities, I'm very willing to work with you to sort this out now.

As is Paul, I'm sure. He's the programmer, & will want to look at your logfile that Sat2Chart creates. It's in your Sat2Chart folder. Fire up Windows Explorer (looks like a manila folder on your task-bar, not Internet Explorer, which is an ancient web browser) & go to your Sat2Chart folder (should be C:\SAT2CHART\SAT2CHART.V16.0.7.0\). Change your view to Details & click the top of the Date Modified column, to sort by that column. The logfile should be one of the top entries (& yes, the Mac doesn't have anything like this, as Apple tries to hide the file system from its users, one reason I prefer Windows, even with its many other warts). Zip this logfile up (I use 7-zip, a free open-source download) & email it to Paul for the best service. His email is at the top of the Help file.

The reason I asked about where you're trying to get content for is that we publish our charts for other cruisers to download (see our Download Charts page). But while we cover over 30 countries, we don't have Oz (yet). But we're interested in expanding our coverage & working with other cruisers to host their charts.

[Jon Hacking]

Originally Posted by Franziska

AFAIK one can set the number of concurrent tile downloads in SASplanet. By default, it's 2 or 4 or some low number. If you allow it to download 20-30 in parallel, it's much much faster.
I'm not sure how to make this work with Sat2Chart, though.

You're essentially correct, Franziska. If you want to increase the number of download threads for Sat2Chart, you have to:

• Fire up SAS Planet
• Go to a Map that you're interested in (under the Maps tab)
• Go to Settings, Map settings
• Crank the "Max concurrent http requests number" up to about 50
• Repeat this for each map type you'll be using
• Close SAS Planet, so it saves your new settings

Then, when you feed your routes to Sat2Chart, it will ask you which map types you want, & how many "splits" you want for each of those. Sat2Chart defaults to 20 splits (download threads) for each map type, which is usually a pretty good number, but you have to tell SAS Planet to allow that many threads before you fire up Sat2Chart.

[Jon Hacking]

Originally Posted by Miyalk

Just had a look with Task Manager, really just confirms its SASPlanet servers, no real load on anything, downloading 20 splits at under 1Mbps according to the Network tab on Task Manager, if I start an OOKLA speedtest it jumps up to 95Mbps!...

Miyalk, it just occurred to me: Have you done what I just told Franziska to do?

If not, then that could be your problem. SAS Planet has to be told to allow at least [XX] download threads (splits in Sat2Chart). Sat2Chart can't set those limits by itself, as the communication between the 2 programs isn't robust enough (there's no real API for SAS, you wouldn't believe the hoops Sat2Chart has to jump through). You have to do it manually, first.

Then, when Sat2Chart directs SAS to download the content, you should see each of those threads starting (actually, you only see 9 in each instance of SAS, & then they start overlapping each other).

This would fit the problems you're seeing (& could be creating problems for Sat2Chart as well).

[Miyalk]

I will go back and check that, as I remember that's how I set it up, but will check.

CPU utilisation is 10-15% so no problem there.

EDIT - they were all set for 50.

[Blueblue]

Originally Posted by Miyalk

*Only Windows Defender, I am pedantic about removing anti-virus, viruses!

Hi Miyalk,

I recently started using Sat2Chart and SasPlanet on a completely fresh install of Windows 10 Pro, and experienced slow downloads and crashes. The thing which solved it for me, after installing the latest version of Sat2Chart version 16.0.10.0, was deactivating Windows Defender Real-time protection. I could reproduce the issue, and got crashes after a while every time I had it on. Now I've added exclusions in Defender excluding everything which involves the chart making process.

//Martin