VentureFarther Satellite Charts - vfkaps plugin

[rgleason]

Jon,
Date/Time has been added to the first 1O or so RMC
Also ZTA has been added.
All sentences that I have added or changed to add date information have been processed for a new checksum.
The freenet checker does not seem to recognize either of the main sentences and it does not process at venturefarther either.

See attached.

I have had to "doctor" & "filter" this file so much that I seriously doubt regular users will be able to use this feature with files saved from their system. As I said earlier, I think a plugin is needed for checking, filtering, removing unwanted and necessary sentences, reducing the density of the data and flaging issues,etc.

Originally Posted by jonturgeon

Hi Rick, as we have talked the file that you are trying to upload is lacking a date/time in your RMC sentences (might be because it is coming from an "Integrated Instrument"?). Not sure why this is happening but VF relies on this to calculate the current tide level so it is a bare minimum for processing.

As you also mentioned, since the RMC sentence should contain your position for a date/time (very important in a NMEA network), to insure the integrity there is also a checksum as the last attribute. So by manually changing the sentence it can not be processed since the checksum will not match. Since I was looking for the date/time from the RMC sentence I do not trap the ZDA (just date/time) sentence. I will add that but your file does not contain that either. Since this recording lasted 30 hours (what you told me earlier) the tide level will need to be calculated multiple times, so all of the RMC sentences need to be correct.

Now you mention that the file is only 1.3mb. You do realize that this file contains over 37,000 rows of data with half of them being depth recordings? I am not saying that this can not be handled, but these recordings happened over a much longer time frame than I had originally intended. I know that I should have planned for it, but my recordings generally last under 1 hour, pretty much to map out an anchorage (which was what I thought would be the most valuable to people and how it would be used). I insert into the DB each depth from the recording, so from this file there would be over 18,000 separate inserts into the db (fairly slow). I will change this to do some filtering before inserting and this will help performance. Less inserted rows, faster processing.

So given this, I am not sure if a plugin is actually needed at this point. What I need to do is generate a large nmea recording, and use that to do some performance testing and code modification. I will also add more feedback if the recording is lacking required info. Since VF is potentially where this info will be stored, it is where all the checks need to take place.

[jonturgeon]

Rick, I think you are really missing the point. You should not (an no user should) be trying to doctor the files, that never goes anywhere productive in my experience (20+ years of commercial software development). Since you are saying in a forum that you "seriously doubt a regular user" I'll respond. You are working way too hard trying to get an invalid file to process, without a date generated by your GPS it is invalid. The important thing to note for others, is this. The minimum spec for the RMC sentence includes date and time. Since the number of people who have tried this is very small (me, Mike Rossiter and you) I do not have a large dataset to see if this is the case. Mike and my recordings processed ok and contained the needed info. Here are the issues as I see them

1. Your RMC sentence lack the Date. Adding this manually to the first 10 rows incorrectly (no checksum match) does not fix it. Your recordings are too long (30 hours) for this to be useful at all, You have over 18,000 RMC sentences in your file. I absolutely need the date for calculating the tide.

2. I think you mean ZDA not ZTA (which would also cause a problem if you inserted ZTA), once again a true recording with will have this many times not once. This is important since over 30 hours the tide level changes.

3. If Freenmea.net (not freenet so others can find it) and VF does not process your file both the systems must be wrong?

4. So I go back to the question I have asked already, have you taken a look at your GPS to see if there is any settings for the date being passed along with the RMC sentence? Or maybe including the ZDA? This requires no doctoring at all and is the real root of the problem.

A very poor system design would require a third party package (a plugin) to do data quality checks. You have the idea that VF can not process large files which is simply untrue and misleading to those reading this. Yes, you have tried to load a much larger file over a longer time frame than I expected. I expected people to want to share depths for anchorages (about an hour recording) and not passages, my miss and I will fix it. What I take away from that is, I need to get my hands on a large NMEA recording that contains the needed info and work with the code to make sure VF can handle it. A simple fix would be something like "if the depths exceed 30 feet skip small changes in depths".

So, if anyone is interested in sending me their NMEA recordings taken straight from the VDR plugin I would appreciate it (long, short, big small). The more recordings I get from different systems the more I can tune the VF NMEA upload. To check and see if you have the issue that Rick's recording has is simple. Open up the recording in notpad (or any plain text editor). Look for the RMC sentence

Rick's looks like this (note at the end there is the ,,, this is where the date should be)
$IIRMC,050048,A,4125.69,N,07100.13,W,005.5,269.0,, ,*15

And this is from my recordings (note no ,,, at the end, the date is 07-15-2016)
$GPRMC,080348,A,3750.753,N,01517.253,E,000.0,011.8 ,150716,1.6,E*7C

So if your recording has the date in the RMC and DBT sentences like below
$IIDBT,30.7,f,9.5,M,,F*07
then you have met the minimum to share depths (depths, GPS location and date/time).

[Rasbats]

The wiki for vfkaps_pi has been updated:

https://opencpn.org/wiki/dokuwiki/do...plugins:vfkaps

Suggest the subject of nmea in VentureFarther be addressed in a new thread.

Mike

[cagney]

Originally Posted by Rasbats

v0.5 has now been pushed to GitHub.

https://github.com/Rasbats/vfkaps_pi/releases/tag/v0.5

Some major changes:

#1. Uses zip format for downloads to improve speed.
#2. Ability to download surrounding charts as well as the selected area.
#3. 64bit Windows problem MAY be resolved.
#4. Option to delete the last file(s) downloaded. Allows changing satellite source for the charts.
#5. Area for selection better highlighted.
#6. Surrounding charts cannot be downloaded if zoom level less than 12 i.e. very small scale.
#7. Direction arrows to move to adjoining area for download.
NOTE: Press 'O' to release the box.
#8. Selection box changes in size to reflect size of satchart that will be downloaded. TODO: This is a major upgrade and the wiki instructions will need to be edited.
TODO: Extensive testing on Linux.
TODO: Resolve directory saving problem on Linux. Preferences does not retain the correct directory.



Please test and comment. I would like to move this to production shortly.


Mike

The 64 bit linux problem is still not resolved. I still have to press "Scan Charts and update Database" manually.

Nothing happens when pressing "O". The Box remains as before.

Directory saving... The correct directory is saved in the .config file in the "kapdirectory" parameter. In the Plugin->Preferences->Directory for saving chart downloads, the last directory in the hierarchy ("kapdirectory") is always missing. This lends itself to an ugly hack. In my vfkaps directory I created a "dummy" directory. When "kapdirectory" points to this "dummy" directory downloads ends up in the "vfkaps" directory.......

Thanks!

Thomas

[rgleason]

RE Post: http://www.cruisersforum.com/forums/...ml#post2467535

1. I will take this to another thread. I thought this would be simple to do when I started recording.
2. I have not been to the boat to find out why there is no date. This is the first time I have had a chartplotter in 30 years, why wouldn't it include the date automatically!! I'll try to find out why.
3. If I am recording, all of my instrument data will be included, but some of it is not needed. Therefore we users will want to set up the recording in OpenCPN to "filter" out all that unnecessary data. I think I know how to do that now.
4. I think we need some easy "filter" technique to eliminate frequent position and depth recordings in Opencpn, similar to what is available Reduce Track Data.
5. Thank you. You are making exactly my point about making this user friendly.
6. Good point, I will make the files shorter until I know everything is working.

Question: Does every RMC require "date" or it is invalid?

Also does VentureFarther.com (VentureFurther is taken?) automatically remove/ignore bad sentences, or does it hang up?
Maybe a few clear error messages with some links to
Nmea Sentence format
Nmea Checksum checkers
would help us all?

[Rasbats]

Thomas,

Thanks. Appreciate your tests.

#3. 64bit Windows problem MAY be resolved.

Yes ... the task has been re-opened in FlySpray.

NOTE: Press 'O' to release the box.

Did you use a direction arrow? Once pressed the box travels with the chart. You need 'O' to re-centre on the new centre of the screen.

TODO: Resolve directory saving problem on Linux. Preferences does not retain the correct directory.

I normally work with Windows. "\" and "/" work the same. Not so on Linux. I have updated the code and the release. Has the problem been solved?

Mike

[rgleason]

#3. 64bit Windows problem MAY be resolved.
Yes ... the task has been re-opened in FlySpray.

Does not work in my 64bit Windows 10 Creator Dell Laptop either.

[jonturgeon]

Not sure what #3 issue was but I am using Windows 10 64 bit and I was able to download 3 kaps, without having to manually scan. Just click the "Generate" button and after a bit the kap is downloaded and the blue box (that represents the kap) shows up at the bottom of the screen. Click that and the chart shows. I am on 4.8.0.

[cagney]

OTE: Press 'O' to release the box.

Did you use a direction arrow? Once pressed the box travels with the chart. You need 'O' to re-centre on the new centre of the screen.

On my Linux the box always stays centered. It does not travel with the chart.
That is why I saw no action when pressing "O". No problem for the user in this case as far as I can see.

TODO: Resolve directory saving problem on Linux. Preferences does not retain the correct directory.

I normally work with Windows. "\" and "/" work the same. Not so on Linux. I have updated the code and the release. Has the problem been solved?

Thomas

Yes it works, but a little cosmetic thing in the opencpn.conf. The "kapdirectory" value ends with "//", this appears to come from two identical lines in your code.

m_sCopyUseDirectory += _T("/");

See lines 203 and 347 in vfkaps_pi.cpp

[Rasbats]

Thanks again. Sorry, I got carried away! TODO Remove lines.

Mike

[rgleason]

Not sure what #3 issue was but I am using Windows 10 64 bit and I was able to download 3 kaps, without having to manually scan. Just click the "Generate" button and after a bit the kap is downloaded and the blue box (that represents the kap) shows up at the bottom of the screen. Click that and the chart shows. I am on 4.8.0.

Can no longer test on Venture Farther. Out of credits.

[Rasbats]

Rick, I got this message several times as well. In the end I think I started with a new installation of OpenCPN. If the downloads direct from the VF website don't work the problem is in the plugin.

The message 'Are you out of credit?' is produced by a download fail not from a measure of the credits remaining with VF.

Mike

[jonturgeon]

Originally Posted by rgleason

Can no longer test on Venture Farther. Out of credits.

Hi Rick, good test , he was indeed out of credits. I have added 100 to your profile.

[rgleason]

Dear Mike & Jon,

I've been testing Vfkaps 0.5 and have simplified my chart setup, which had many chart directories and many chart groups & chartdownloads, by saving my opencpn.ini as opencpn-4.8.0-chartgroups.ini and then deleting...

Then I saved and reopened O and started testing. It seems like I had a crash after save kap, manually update db, save kap, crash. This happened multiple times with google, nokia and bing. I wondered if it was caused by being zoomed in too much, so I tried hitting the "O" in the middle each time. That did not seem to fix the problem. I also tried hitting "ok" at the prompt to scan db and not hitting "ok" at the prompt to scan db. Neither approach seemed to work.

I had thought this was working better earlier, but perhaps some unknown setting has changed. The results are quite nice and the user interface is quite good.

Any suggestions?

Also I had a couple of ideas about display of the Kaps...
1. I'd like to be able to force vfkaps to display a certain kap, or at least to step through all the kaps in a given area. I seem to have trouble getting the really zoomed up ones to display because the zoomed out one covers it or something.

Perhaps we need a way of identifying the zoomed out one and deleting it, too?

I don't know, I think some kind of display control might be helpful particularly when multiple adjacent kaps have been downloaded to get good resolution. Is there any quidance about that that would help?

[cagney]

The name of the plugin is currently "vfkaps_pi", this is not consistent with the established default naming which should be something like "opencpn-plugin-vfkaps".

See this post http://www.cruisersforum.com/forums/...ml#post2476242 for a good reason.

Thomas