OpenCPN 3.2 Crash / Invalid NMEA in Log

[kiltym]

Yesterday was the first trip we did with 3.2, and along the trip, about 8 hours, it crashed on average about every 20 minutes or so. Sometimes after only 2-3 minutes, sometimes after 30+ minutes.

It may be related to this thread: http://www.cruisersforum.com/forums/...ly-100102.html, but I wanted to start a new one in case this is different.

During the day, I did the following, with it still crashing:
1) Disabled all plug-ins (although, I did not remove them from the Plugin directory, and I do use Logbook Konni, so I will remove this plugin on our next trip)
2) Turned off all TCP and UDP servers leaving only 3 serial COM inputs (NMEA Nav Data, AIS, DSC).

I am running on Windows 7, 64 bit.

Not much installed on this computer except some Nav software like O, NavmonPC, PolarCom, etc....

Have run 3.02 before, and never had a crash, and nothing has changed on the computer except installing 3.2.

The thing I am noticing is the following in the log files. I do not know how it correlates to the crashing (and perhaps it does not and is a separate issue).

8:25:39 AM: Changing NMEA Datasource for ,,283 to Serial:COM6 (Priority: 1)
8:27:59 AM: Changing NMEA Datasource for S,089 to Serial:COM6 (Priority: 1)

And other nonsense NMEA sentences, however, these sentences do not ever seem to actually get sent to O from any source, so it would appear there is some issue in O in parsing the sentences (or memory not getting cleared correctly).

I have attached, 2 log files, and VDR files.

The two "Bad NMEA" are the log, and the VDR from when this log was created.
The two "Crash" are the log, and the VDR from when O crashed on me.

There are bad sentences present in both logs.

I will remove Konni from my Plugin and test the crashing issue, but something else seems off as well.

Please let me know what else I can do to provide help as it is pretty important to me to get O working without crashing as we rely on it heavily.

[bdbcat]

kiltym....

OK, first thing I would try is to filter away some of the traffic on COM6. The talker $II is quite wordy, and O ignores most of this (or should). It could be a simple overload on the com port handler...

Try to allow only $IIRMC, which will give you position/velocity, and be enough for normal navigation. Run this for a while, and see if it is stable. Then successively add more $IIxxx sentences until something breaks.

Good Luck
Dave

[nohal]

Mark...
Definitely try without the logbook - I just played both your files with VDR observing the behaviour of clean OpenCPN with valgrind and can't see any memory problems related to NMEA... Will try also with the logbook plugin when I get to it as from the other thread it appears that it's the cause of the problems under some circumstances.

Pavel

[kiltym]

$II is the Raymarine Seatalk to Nmea converter.

Prior to 3.2, the real COM port was sent through NavMonPC first, and then to a virtual port port that O read. I had no filtering on in that situation (all at 4800 baud) and never had an issue. O also wrote the autopilot commands back to the virtual port, and through NavMonPc back to the converter onto the Seatalk network. Now it writes directly to COM6.

The entire suite of instruments on our boat are on Seatalk (wind, log, temp, gps, autopilot, rudder, compass) and then convert to NMEA via this converter to COM6.

I can filter sentences, but don't really think that I should have to based on O dealing ok with it before through NavMonPc. And i want O to "repeat" all data on a UDP port for other apps to access, and in turn, remove NavMonPc from my software stack.

I will remove Logbook from the plugin folder and report back, but i do feel that something is causing O to receive/create invalid NMEA sentences, based on the log, that it should be able to disregard, or not create since they do not appear in the VDR, and I dont see how Logbook could create sentences into O.

[bdbcat]

Mark....

I'm suggesting incremental NMEA message filtering as a way for me to remotely debug a problem with O as first seen in the Galapagos.

We have lots of satisfied 3.2 users, and we have never received reports from other users receiving or generating phantom messages as you exhibit, so we need to break this down a bit and see what is going on with your setup.

Thanks for your help
Dave

[kiltym]

Dave,

All sounds good. I will remove the Konni plugin first, just to see the result, and will keep an eye on the log to see if I am still getting the bad NMEA displayed.

I would think (although I dont know the code at all), that with filtering, it would basically ignore all these invalid sentences anyway as I would create a list of "valid" sentences in the filter. But will go with your suggestion and start with minimal sentences and add them 1 at a time and see the results. Of course, we dont know if the invalid sentences are causing the crash either.

We are leaving here on Monday, so hopefully can post some results early next week and go from there. If i can play at anchor a bit I will do that also.

[kiltym]

Here are the results, I will try to not make this too long and boring as it has been at least 15+ hours trying to narrow down the issue.

My crash is not related to LogBook Konni.

I removed all plugins from the plugin directory, and still had the crash.

I started to filter sentences, as suggested, and was down to only GLL, and still had the crash.

I have lots of VDR files (1.5meg+) that I examined. I have found invalid sentences present in these files. However, I am not sure if it is truly invalid data coming in, or some artifact of O (probably the former though).

I have inserted NavMonPC back into the middle. So, all NMEA data (Nav, AIS, DSC) enters NavMonPC, and are merged into 1 Virtual COM out, that O is using (at 34k). I am not filtering any sentences, in either NavMonPC or O. O is outputting RMB (to seatalk network via NavMonPC), and RMC (only to NavMonPC).

I have logged the data from NavMonPC, and find no invalid sentences present, perhaps it is filtering before writing to the log....

It have been running about 1 hour, and no crash so far, which is longer than I have been able to do before.

So, perhaps there are some bad sentences coming through, and NavMonPC is doing a better job of disregarding, and in turn, only sending "clean" sentences to O.

That is my theory.

So, if you want any data from me, let me know, but perhaps there needs to be a more strict check in O for valid sentence before attempting to process anything? Again, just a theory as I don't know how things are coded.

For now, it seems OK again, but I had hoped to remove NavMonPC from my dependencies, but it has been stable for 2+ years so it should be OK to continue to use it.

[kiltym]

Unfortunately, I am still having crashes and using NavMonPC does not seem to help.

I have now uninstalled O, reinstalled it cleanly, added only CM93 2011 charts and one data source coming from NavMonPC. I created a 3 point route, and that is active. I am at anchor.

I will let this run and see what happens, but this is take 2. On the previous clean install I made a number of changes within O's settings before testing for a long period, and it crashed. So now I will try again with no changes to O, except adding the charts, adding the com port, and the active route.

If this doesn't work I will probably have to go back to 3.02, which is disappointing. If it does work OK, I will make small changes at a time and see if/when it crashes. Very time consuming as each run, assuming is does not crash, has to be an hour or more.

If someone can offer any other suggestions that would be great. If someone can give me a debug version with more logging information I am happy to run it and see if it sheds a light on the issue. I would really like O to work correctly before crossing the big ocean :-) .

[kiltym]

Also, 2 strange lines in my log file with the clean install.

Error: Cannot set locale to language Arabic (Uae).

and,

EnumerateSerialPorts() Found Garmin USB Driver.

I am pretty sure I do not know Arabic, and did not change the default from English US, and I do not believe I have ever installed a Garmin product on this computer.

I use a Keyspan Serial 4 port to USB to the computer, nothing else is attached to this computer. I don't own any Garmin equipment.

Why these 2 messages?

[bdbcat]

kiltym....

OK, the first thing we need to figure out is where O is crashing.

Can you configure your system for generating minidump crash reports, and forward a dump to me? This could be helpful.

Go to Start and type in "sysdm.cpl" (without the quotes) and press Enter
Click on the Advanced tab
Click on the Startup and Recovery Settings button
Ensure that "Automatically restart" is unchecked
Under the Write Debugging Information header select "Small memory dump (256 kB)" in the dropdown box
Ensure that the Small Dump Directory is listed as "%systemroot%\Minidump" (without the quotes)
Click OK twice to exit the dialogs, then reboot for the changes to take effect.

Run O until crash. You should find in c:\windows\Minidump a dump file.

I will admit that I have tried this, and cannot get a dump file to write on my Win7 box. Don't know why, but I continue to fiddle with it.

Thanks
Dave

[kiltym]

Dave,

Here is some more info as well. I am working with a clean install, and making small changes at a time to determine better what is causing the crash.

My setup is all NMEA info (GPS, Wind, AIS, DSC, Log, etc) is feeding into NavMonPC. I have NavMonPC merge the data into a single Virtual Port for O.

I am only sending AIS and the following Nav Data into O, filtered by NavMonPC. I assume this will make sure only "clean" data is making its way to O.

GGA, GLL, GSV, HHDG, HDM, HDT, VTG

I am sending APB, RMB, and RMC back into NavMonPC from O, and use NavMonPC to control/test what data goes back onto the Seatalk Network, and in turn the Autopilot and Chartplotter.

What I am finding, is the crash has something to do with sending the RMB and/or APB messages onto Seatalk. If these sentences only go to NavMonPC, O does not seem to crash. When the data is pushed to Seatalk, the crash happens.

When APB or RMB are sent to the Seatalk network, new sentences come in from there (APB, RMB, XTE), but now, NavMonPC is not sending any of them to O, yet O is still crashing (I am testing this again as I may be mistaken on this point, but will add more in a few hours of testing this). All very strange.....

I will try to get this dump you mention and will let you know once I have more to report.

[kiltym]

Dave,

Is there an advantage to sending RMB vs APB to control the autopilot, or is it just a matter of what works with a particular system?

I assume both should not be sent?

Also, what was the default in 3.02, as maybe that is different?

[bdbcat]

Kiltym....

Thanks for the info.
3.0.2 used RMB only.

Probably best to use the one output sentence that the autopilot recognizes, and only that one.

Dave

[kiltym]

An update.

O has been running now for a few hours, no crash. This is with sending RMB out onto Seatalk, but NavMonPC not allowing any of the additional NMEA sentences back into O.

So, it would lead me to believe the crash has something to do with a new sentence that is created on Seatalk, when a RMB sentence is sent to Seatalk.

I think 5 new sentences are sent, and I will add each one individually and see if I can find our which is causing the crash to O.

The good news however, is I seem to have a workaround for now. Will try to update later today.

[RhythmDoctor]

I realize that I am coming into this late, and I am just getting started with 3.2. Due to the cold spring, I have only done one 3 hour daysail, during which O ran fine, but I was not sending any data to the autopilot. But immediately upon reading your message, your problem looked familar to me.

I believe that the autopilot may have been causing your problem. The communications is totally different on 3.2, and all the data goes out to a common bus, which includes the autopilot. Under 3.0, since the output port was specifically limited to autopilot, the sentences were filtered by default. The new way on 3.2 is much more powerful, but more potential for problems, especially if you're using AIS.

The reason AIS is a problem is because it has so much more data so it's prone to overflowing the buffers. If you're taking in AIS at 38400 and open an output port at 4800 for the autopilot, you will definitely overflow the port unless you filter the AIS out. Whereas before the autopilot could simply ignore the irrelevant data, with AIS it gets deluged with more than it can handle.

I noticed this problem immediately when I upgraded from 3.0 to 3.2. I started getting crashes immediately in my testing (before I took my boat out). As soon as I turned off the autopilot output (which had been turned on because the settings came over from 3.0.2), everything worked fine.

Since I have not gone out in the boat for more than a few hours, I have not had the need or time to turn on the autopilot port. I have set up some preliminary filtering for the autopilot output port, but have not had time to test it out yet.

Much of this is described in this page of the user manual: Data Connections | Official OpenCPN Homepage

All this is different from the logic in earlier versions of OpenCPN. In 3.2, there is no defined "autopilot" port. The autopilot is simply connected to any available output- enabled datastream, and gets everything on the bus, subject to user specified output filtering. There is no specific "shared" AIS and GPS port, as all ports are shared.