VentureFurther - Nmea/Depth Sharing

[rgleason]

RE Post: VentureFarther Satellite Charts - vfkaps plugin

1. I will take this to another thread. I thought this would be simple to do when I started recording.
2. I have not been to the boat to find out why there is no date. This is the first time I have had a chartplotter in 30 years, why wouldn't it include the date automatically!! I'll try to find out why.
3. If I am recording, all of my instrument data will be included, but some of it is not needed. Therefore we users will want to set up the recording in OpenCPN to "filter" out all that unnecessary data. I think I know how to do that now.
4. I think we need some easy "filter" technique to eliminate frequent position and depth recordings in Opencpn, similar to what is available Reduce Track Data.
5. Thank you. You are making exactly my point about making this user friendly.
6. Good point, I will make the files shorter until I know everything is working.

Question: Does every RMC require "date" or it is invalid?

Also does VentureFarther.com (VentureFurther is taken?) automatically remove/ignore bad sentences, or does it hang up and stop processing?
It might be nice to have a progress bar.
Maybe a few clear error messages with some links to
Nmea Sentence format
Nmea Checksum checkers
would help us all?

[rgleason]

For those who would like to upload depth data, I would suggest checking your file first, at least the first few sentences, as Jon has suggested.

My recordings were not useful because something is misconfigured in my B&G Zeus12T or in the ST converter, such that the date is not included, there is too much instrument data that is unnecessary, and there are some spurious locations probably due to OpenCPN failing and then being restarted which result in SOG >100 mph. Even filtering with Connection settings, and hand entering the dates and then recalculating the checksum does not seem to fix it.

That thread is here:
http://www.cruisersforum.com/forums/...ng-189218.html

[transmitterdan]

Rick,

I would be willing to help. I think this feature request is best handled not by hacking O but by writing a filter program that just filters out all unnecessary or badly formed messages from the VDR file. Can you tell me exactly what NMEA messages VentureFarther actually needs? It's a pretty trivial matter to write a filtering program. It's a whole other thing to coerce O into doing it.

[conachair]

SLight thread drift, apologies, but relevent..

Has anyone played with node-red?
https://nodered.org/
Windows, linux, even (very shakily) android.
And online
https://fred.sensetecnic.com/


Amazingly powerful and fast for those of us with meagre programming skills.
So this morning I had a play around with NMEA.


Once you have node red up and running then ctrl I and control V the code below and you'll get this..



Two bits to it, top section - if you press the blue button to the left of the inject node then it shoots this text (with wrong checksum) down the flow "$GPGLL,5300.97914,N,00259.98174,E,125926,A*77 ", then some functions corrects the checksum and send it to opencpn via a TCP connection.
Bottom section does similar with a file limited to 1 message a second and sends it to opencpn & saves it to another file.

Hope this is of use, it really is such a powerful program, I use it a lot on the raspberry pi to save data to a sqlite database and mess around with some nmea so the opencpn logbook will save it.

You can easily create nmea sentences with random or fixed values for testing.

[{"id":"f6a5bd8d.3693","type":"inject","z":"1febfe0b.5f6be2","name":"","topic":"","payload":"$GPGLL,5300.97914,N,00259.98174,E,125926,A*77","payloadType":"str","repeat":"","crontab":"","once":false,"x":130,"y":160,"wires":[["3f92a47.b64215c","8c5f5b5f.a39c68"]]},{"id":"3f92a47.b64215c","type":"debug","z":"1febfe0b.5f6be2","name":"","active":true,"console":"false","complete":"false","x":460,"y":180,"wires":[]},{"id":"8c5f5b5f.a39c68","type":"function","z":"1febfe0b.5f6be2","name":"Strip  NMEA message","func":"var nmeaFull=msg.payload;      //full NMEA sentence\nvar messageType=nmeaFull.substring(1, nmeaFull.indexOf(\",\"));  //strips just the first message type\n\n//copies the message type to a flow variable\nflow.set('messageTypeFlow',messageType);\n\n\n//cut of the leading \"$\" and the bad checksum\nvar nmeaStripped=nmeaFull.substring(nmeaFull.indexOf(\",\") + 1 , nmeaFull.indexOf(\"*\"));\nmsg.payload = nmeaStripped;\n\nreturn msg;\n","outputs":1,"noerr":0,"x":220,"y":100,"wires":[["7c371d3c.014c94","a4d98a10.2be698"]]},{"id":"7c371d3c.014c94","type":"function","z":"1febfe0b.5f6be2","name":"Add checksum","func":"var nmea = msg.payload;    // stripped NMEA into a variable\nnmea = flow.get('messageTypeFlow') + \",\" + nmea;\nvar checksum = 0;         // initialise checksum variable\n\n// calculate checksum\nfor(var i = 0; i < nmea.length; i++) { \n  checksum = checksum ^ nmea.charCodeAt(i); \n}\nchecksum = checksum.toString(16);    //convert to hex\nnmea = \"$\" + nmea + '*' + checksum;  //make the full nmea sentence again\n\n\n\nmsg.payload = nmea +\"\\r\\n\";\n\nreturn msg;\n","outputs":1,"noerr":0,"x":400,"y":20,"wires":[["9a0375e7.69cd48","95f79a50.19a6f8"]]},{"id":"a4d98a10.2be698","type":"debug","z":"1febfe0b.5f6be2","name":"","active":true,"console":"false","complete":"false","x":450,"y":120,"wires":[]},{"id":"9a0375e7.69cd48","type":"debug","z":"1febfe0b.5f6be2","name":"","active":true,"console":"false","complete":"payload","x":530,"y":80,"wires":[]},{"id":"95f79a50.19a6f8","type":"tcp out","z":"1febfe0b.5f6be2","host":"localhost","port":"10109","beserver":"server","base64":false,"end":false,"name":"","x":710,"y":140,"wires":[]},{"id":"bbe6bb8a.8eb568","type":"function","z":"1febfe0b.5f6be2","name":"Add checksum","func":"var nmea = msg.payload;    // stripped NMEA into a variable\nnmea = flow.get('messageTypeFlow') + \",\" + nmea;\nvar checksum = 0;         // initialise checksum variable\n\n// calculate checksum\nfor(var i = 0; i < nmea.length; i++) { \n  checksum = checksum ^ nmea.charCodeAt(i); \n}\nchecksum = checksum.toString(16);    //convert to hex\nnmea = \"$\" + nmea + '*' + checksum;  //make the full nmea sentence again\n\n\n\nmsg.payload = nmea +\"\\r\\n\";\n\nreturn msg;\n","outputs":1,"noerr":0,"x":520,"y":380,"wires":[["ac74ce41.a6099","1dd0af23.b14cf1","95f79a50.19a6f8"]]},{"id":"a23ce034.31e45","type":"function","z":"1febfe0b.5f6be2","name":"Strip  NMEA message","func":"var nmeaFull=msg.payload;      //full NMEA sentence\nvar messageType=nmeaFull.substring(1, nmeaFull.indexOf(\",\"));  //strips just the first message type\n\n//copies the message type to a flow variable\nflow.set('messageTypeFlow',messageType);\n\n\n//cut of the leading \"$\" and the bad checksum\nvar nmeaStripped=nmeaFull.substring(nmeaFull.indexOf(\",\") + 1 , nmeaFull.indexOf(\"*\"));\nmsg.payload = nmeaStripped;\n\nreturn msg;\n","outputs":1,"noerr":0,"x":280,"y":300,"wires":[["bbe6bb8a.8eb568","e448786d.6d8a08"]]},{"id":"ac74ce41.a6099","type":"file","z":"1febfe0b.5f6be2","name":"","filename":"NMEA_OUT.txt","appendNewline":false,"createDir":false,"overwriteFile":"false","x":740,"y":340,"wires":[]},{"id":"1dd0af23.b14cf1","type":"debug","z":"1febfe0b.5f6be2","name":"","active":true,"console":"false","complete":"false","x":730,"y":420,"wires":[]},{"id":"e448786d.6d8a08","type":"debug","z":"1febfe0b.5f6be2","name":"","active":false,"console":"false","complete":"false","x":510,"y":500,"wires":[]},{"id":"3a12a443.cb951c","type":"delay","z":"1febfe0b.5f6be2","name":"","pauseType":"rate","timeout":"1","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":210,"y":500,"wires":[["a23ce034.31e45"]]},{"id":"2226c10e.11ce4e","type":"file in","z":"1febfe0b.5f6be2","name":"","filename":"test.txt","format":"lines","chunk":false,"sendError":false,"x":120,"y":380,"wires":[["3a12a443.cb951c"]]},{"id":"238b5508.6df92a","type":"inject","z":"1febfe0b.5f6be2","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":100,"y":440,"wires":[["2226c10e.11ce4e"]]}]

[rgleason]

These are awesome offers!

I will check with Jon about the exact Nmea sentences expected. He has confirmed that RMC with date and time is necessary, and that he needs depth. I was providing depth from transducer which so far is not accepted as presented.

I hope to get back to the boat in two weeks to figure out why I don't have RMC date coming from my B&G Zeus12T chartplotter with GPS, and to disconnect my VHF GPS puck from the Autopilot nmea connection....

Conachair makes a strong case that I should proceed with construction of the RPI3 Recording Instrument asap, because of node red!

[rgleason]

Email to Jon (Venture Farther)

Jon,

TDan and Conachair have offered to provide a plugin filter for VF bound data.
What are the requirements and filters needed to pass VF nmea depth upload?

User sets the maximum speed of the boat to help throw out bad locations.

1. Accept only sentences with correct checksums.
2. Accept only Required Sentences RMC,DBT
3. Filtered out all other Sentences GLL,MWV,DBT,VHW,MTW, etc.
4. Each RMC checked for Data & Time and checksum before passing. All other RMC removed.
5. Each DBT should be checked for checksum.

How often do you want RMC & DBT sentences? (define with time & distance?)
6. Remove excessive depth points and associated RMC from the file.
7. Provide user friendly messages about what is wrong with the file, line by line, as these lines get filtered out.
- Bad (RMC, DBT) sentence format.
- Bad (RMC, DBT) checksum.
- Speed excessive, location anomoly, RMC (s) removed.
8. Are there other sentences that are accepted? Which ones?

Thank you.
Rick

[rgleason]

Here is Jon's response. (I hope he does not mind my sharing this.)

Hi Rick, One thing I also want to point out about the NMEA process in VF. A lot of cruisers do not use OpenCPN (we do and really like it). One thing I have always kept in mind with VF is I want to provide services to OpenCPN and other charting software (hence the integration I already have with SEAiq). Because of the popularity of tablets (meaning iPad/Android) a lot of cruisers have wifi nmea repeaters onboard (we do). A side project that I am working on (really no time for it unfortunately) is a Android based NMEA recorder that connects to the onboard repeaters. Why this? My attempt at getting more people without OpenCPN able to share this kind of info to all VF users, including OpenCPN people.

• They could do this but how would the checksum get corrupted? No matter what I still need to check it on the server. I need to maintain the data integrity.
• The danger of this is sort of what we found with your recording. I am still not sure why your RMC did not contain the date and if this is widespread. I originally built the depth upload around the "Survey Plugin" with Mike. Working with it and the data started to cause concern. First, the Survey plugin used the computer time. Since we change time zones frequently I really do not trust computer time. The time/date need to come from the GPS. Also the survey plugin was a bit more complicated to use compared to the NMEA recorder.
• This could be of some help on the upload for file size but the filtering of these happens very quickly in VF. The problem I ran into with performance on your file was the amount of depths that needed to be inserted into the DB. This is my fault and is fairly easily corrected. I already "filter on view" the depths. I could use this same logic to filter on insert. With the current code even if the plugin filtered out all the excess, if there were 18,000 depths it would drag and take a bit to process. I need to fix that.
• This could be done but my guess is all GPS inserted data would pass.
• This could be done but my guess is all sounder inserted data would pass.
• This is a tricky one and sort of what the survey plugin attempted.
• I also need to improve the response to incorrect data to help the user figure bits out.
• I accept AIS sentences and will accept ZDA. I use the AIS sentences to create position reports for VF members that have entered their MMSI numbers in their profiles (if there is a match).

Now, here is what I think. I really need to implement most of these on the server side regardless of what a plugin does. I need to do this to allow other integrators as easy of a path to follow as possible and to insure the data is whole. If I had a large file with valid RMC sentences (which I will try to record the next time I am out), I can work on the performance/validation of the upload (trust me, Java can handle this easily if it is programmed to handle it, once again my mistake).

Now what I would think would be a cool feature of the VDR plugin is this. I am a firm believer in trying to make things as simple as possible for users (why I built the sat chart page in VF like it is and why I guided Mike towards that for the plugin). I also recognize that the VDR NMEA data is text and over a long period of time can build several mb's. Since Mike already has the idea of the api key in the new vfkaps plugin how about taking it a bit further in the VDR. Allow users to do a recording, when they are finished and save the file it is automatically zipped up (you see the size difference in your upload) and post it directly to VF using the api key. No more interaction is needed. Of course they would need an active internet connection to do this and the plugin may be able to made fancier to hold onto the file until that occurs but it could be a start.

Jon

p.s. consider the current NMEA upload as simply a proof of concept. I built it since I thought it was a good idea but before I spent heaps of time on it I wanted to gauge interest (since I am doing the site on my own I need to focus on what is being used). I don't mind spending more time getting the wrinkles worked out if people want to use this feature (for over a year Mike and I are the only contributors).

Jon's suggestion as I understand it, is that his server will need to do much of the processing and checking in any case, because the data will be coming from different sources and he'd like to keep the data quality high. However he does make some suggestions for improving VDR or perhaps some other plugin to enable compression and upload to Venture Further using the user API.

Regarding improvements to VDR_pi

Mike R [Rasbats] had started making some really nice improvements to the controls for VDRplayer which would be very helpful for testing etc. giving the user control over speed, fast replay back and forward, timeline, etc with a control similar to Grib_pi. I believe he was also going to build in filtering etc. VDR_pi is an important tool for testing and I'd like to see these improvements take hold. I think enabling file compression would also be useful.

Jon is suggesting that VDR become more specialized and be able to upload nmea depth files to VF using the users API.

There are many ways to skin the cat, and perhaps adding these features to VDR_pi is fine because it will be easy on the user. Are there some other ideas? Perhaps another dedicated plugin vfdepth_pi? That has some full set or or subset of features?

Note:
We also have VDRplayer from TDan which has the advantage of using a network TCP connection to create the nmea data stream, all without affecting OpenCPN in any way, because it is a separate program. Which is actually a purer environment for testing bugs etc.

[rgleason]

Conachair,
Does node-red simply work on a browser? Using java javascript and json? Does that mean I can use it now on windows?

[rgleason]

Might be ideal to make nmea files for testing. Perhaps should be listed in Supplimentary Software or in Dev?

[Moo]

What format does the VDR plugin record data in?

I do not use it for stability reasons and also the fact that I do not run OCPN or any other plotter all the time. I do however record all NMEA data in raw text format and can upload this.

This is what my data looks like. Any use?

$GPRMC,052224,A,1620.8441,S,16634.2770,W,6.4,243,260817,12.19,E,D*02
$WIVWR,109.9,L,19.8,N,10.2,M,36.7,K*79
$WIMWV,250.09,R,19.79,N,A*1B
$WIMWV,223.29,T,17.49,N,A*16
$GPGGA,052224,1620.8441,S,16634.2770,W,2,3,1.7,0.3,M,,M*6D
$GPGLL,1620.8441,S,16634.2770,W,052224,A,D*46
$IIVTG,243,T,,M,6.4,N,11.8,K,D*3E
$GPAPB,A,A,2.54,L,N,V,V,268.2,T,316,268.8,T,,T,D*59
$WIMWD,100.9,T,,M,,N,,M*7C
$IIVHW,,T,225.5,M,6.38,N,11.81,K*4F
$IIVLW,7756.04,N,5543.28,N,,N,,N*47
$GPZDA,052224,26,08,2017,00,00*43
$GPXTE,A,A,2.54,L,N,D*05
$HCHDG,225.5,0,E,12.2,E*6D
$GPDTM,W84,,0.0000,N,0.0000,E,0,W84*71
$TIROT,76.14,A*3F
$SDDPT,,0.8,*5D
$IIMDA,29.9,I,1.01,B,21,C,,C,,,,C,101.09,T,,M,22.88,N,11.77,M*0C
$WIVWR,112.8,L,19,N,9.8,M,35.2,K*50
$WIMWV,247.19,R,18.99,N,A*13
$WIMWV,225.4,T,22.88,N,A*24
$GPGGA,052225,1620.8441,S,16634.2791,W,2,4,2.8,0.3,M,,M*68
$GPAPB,A,A,2.54,L,N,V,V,268.2,T,316,268.8,T,,T,D*59
$GPDTM,W84,,0.0000,N,0.0000,E,0,W84*71
$TIROT,-11.7,A*21
$SDDPT,,0.8,*5D
$GPGSV,4,1,15,01,41,025,00,07,50,199,32,08,48,161,32,09,45,302,33*73
$GPGSV,4,2,15,11,70,041,39,16,09,115,30,17,06,313,00,22,06,024,27*7F
$GPGSV,4,3,15,23,37,342,37,27,16,145,00,28,10,251,00,30,17,215,00*72
$GPGSV,4,4,15,,,,*7D
$IIMDA,29.9,I,1.01,B,21.1,C,,C,,,,C,119.39,T,,M,26.69,N,13.73,M*14
$GPRMB,A,2.54,L,10000,316,1622.2900,S,17345.5100,W,414.6,268.79,4.86,V,D*54
$GPGSA,A,1,,,,,,,,,,,,,,3.4,0*07
$GPRMC,052226,A,1620.8448,S,16634.2808,W,6.34,241.04,260817,12.19,E,D*12
$WIVWR,115.2,L,19,N,9.8,M,35.2,K*5D
$WIMWV,244.79,R,18.99,N,A*16
$WIMWV,240.69,T,26.69,N,A*17
$GPGGA,052226,1620.8448,S,16634.2808,W,2,3,3.4,0.3,M,,M*67
$GPGLL,1620.8448,S,16634.2808,W,052226,A,D*4D
$IIVTG,241,T,,M,6.3,N,11.7,K,D*34
$GPAPB,A,A,2.54,L,N,V,V,268.2,T,316,268.8,T,,T,D*59
$WIMWD,119.39,T,,M,,N,,M*47
$IIVHW,,T,226.6,M,6.32,N,11.7,K*7B
$IIVLW,7756.04,N,5543.28,N,,N,,N*47
$GPZDA,052226,26,08,2017,00,00*41
$GPXTE,A,A,2.54,L,N,D*05
$HCHDG,226.9,0,E,12.2,E*62
$TIROT,-35.1,A*21
$SDDPT,,0.8,*5D
$IIMDA,29.9,I,1.01,B,21,C,,C,,,,C,119.39,T,,M,25.89,N,13.32,M*03
$WIVWR,114.9,L,19.6,N,10.1,M,36.3,K*7C
$WIMWV,245.09,R,19.59,N,A*1D
$WIMWV,238.49,T,25.89,N,A*17
$GPGGA,052227,1620.8461,S,16634.2830,W,1,4,2.5,0.3,M,,M*62
$GPAPB,A,A,2.54,L,N,V,V,268.2,T,316,268.8,T,,T,A*5C
$TIROT,134.76,A*0C
$SDDPT,,0.8,*5D
$IIMDA,29.9,I,1.01,B,20.9,C,,C,,,,C,116.69,T,,M,21.69,N,11.16,M*11
$GPRMB,A,2.53,L,10000,316,1622.2900,S,17345.5100,W,414.6,268.79,4.82,V,A*52
$GPRMC,052228,A,1620.8472,S,16634.2857,W,6.34,240.38,260817,12.19,E,A*14

[bcn]

VDR is using plain COM traffic like yours.

[Moo]

Originally Posted by bcn

VDR is using plain COM traffic like yours.

Cool bananas - I'll try uploading some.

[Moo]

Originally Posted by bcn

VDR is using plain COM traffic like yours.

Yep it works fine.

Perhaps the references to VDR plugin should be pushed a little more into the background and more emphasis put on the fact that plain NMEA logged from a COM port is fine as that should help with recruitment.

I use GPSGate Splitter to log all NMEA com traffic to file. I log hourly chunks which makes it really easy to go back and find a specific event.

[rgleason]

I think both are needed. We need new users to understand what plugin to use.

[Moo]

Originally Posted by rgleason

I think both are needed. We need new users to understand what plugin to use.

I think you are missing the point Rick.. or I didn't state it clearly which is always likely!.

You do not need a plugin to record NMEA data in fact I would argue that you better off recording the data outside of OCPN.

A plugin makes it easy for some perhaps but it may not be the best way.