Help! GPX File Import Not Functioning

[reflectionsv37]

Sorry for the delay in getting back with test results.

I tested this again this morning and learned a surprising thing. I verified that I did not have my GPX file in the layers folder, deleted all my waypoints in Route Manager and then attempted to import the file the I had updated and exported a few days ago.

Lo and behold, some of the waypoints imported this time. But not all of them. I had added a few new waypoints and changed the names of 35+- others. Those were the only waypoints that imported.

I then recalled that I have shared my waypoints with Terry on Valhalla and he had incorporated them in a GPX file that he had been distributing to others. That file was also in my layers folder. I removed his file from the layers folder, deleted all my waypoints and then tried the import again. All the waypoints were successfully imported this time. I verified this by exporting them to a different file and comparing the two files.

I'll email Terry to have him look at this thread, but I'm going to assume that he cut and pasted from my GPX file to add those waypoints to his GPX file which would have maintained the name, lat, long and GUID of the waypoints in my file. I'm going to assume that because the name, lat, long were the same, they did not import, but it brings up some interesting questions.

It's apparent that there can be two waypoints with the same name, lat, long as long as the waypoint already exists and you setup a layer file that contains that same waypoint.

1) How does OpenCPN differentiate between a regular waypoint and a layer waypoint? If the GUID identifies the waypoint in the navobj file does the GUID of the layer waypoint also exist in the navobj file? I would think that would create a duplicate situation in the navobj file, but it seems OpenCPN is aware that they are different since I can move the location of the regular waypoint to reveal the underlying layer waypoint.

2) When does the GUID get generated and updated? Does a new GUID get generated when a waypoint is imported?

3) What determines the value of the GUID? What makes it unique? Is it a hash of name, lat and long?

4) Would incorporating a creation timestamp of the waypoint allow duplicates to exist? Imported waypoints could have a new GUID incorporated at import time and no longer have an issue with duplicates.

5) Could you give me a brief explanation of where else the GUID is used? I assume it must play a role in routes.

In any event, I now understand what happened and why, but somehow I get the feeling this issue is going to bite me again!!

Thanks again for all the feedback!

[nohal]

Originally Posted by reflectionsv37

1) How does OpenCPN differentiate between a regular waypoint and a layer waypoint? If the GUID identifies the waypoint in the navobj file does the GUID of the layer waypoint also exist in the navobj file? I would think that would create a duplicate situation in the navobj file, but it seems OpenCPN is aware that they are different since I can move the location of the regular waypoint to reveal the underlying layer waypoint.

By knowing one is in a layer and the other isn't. We by design do not remove waypoints from layers and by same design never remove any "live" data without explicit user action. Which leads to the situation you describe when by the "correct" order of creation you manage to have two waypoints on the same spot.

2) When does the GUID get generated and updated? Does a new GUID get generated when a waypoint is imported?

Created when the waypoint is created, updated (as in changed) never, that's it's whole purpose. Objects imported from GPX are only assigned a new GUID when they do not have one yet. This allows you to have pretty consistent data even when you keep exchanging navobj.xml files, exporting from them, importing to others, synchronizing your data between several devices and do a lot of other weird stuff that you did not even imagine people might want to do.

3) What determines the value of the GUID? What makes it unique? Is it a hash of name, lat and long?

The algorithm used to create it makes it globally unique with reasonable certainty (Continue reading also the Collisions chapter on the page linked).

4) Would incorporating a creation timestamp of the waypoint allow duplicates to exist? Imported waypoints could have a new GUID incorporated at import time and no longer have an issue with duplicates.

We do not want duplicates, usually. That's why we try to avoid them pretty hard. Usually I really do not want 10 waypoints on the same spot just because they were by accident included in 10 GPX files I imported over time, do you? If so, why? And which one of them to use when we eg. create a route through that spot? We do not need to incorporate a timestamp or FWIW anything else to allow duplicates, but the design decision simply is we do not want duplicates.

5) Could you give me a brief explanation of where else the GUID is used? I assume it must play a role in routes.

Everywhere where OpenCPN needs to internally identify a unique object like waypoint, route point, route, track etc.

In your particular use case, you might be happy if we did not consider a waypoint duplicate between the "live" data and a layer a duplicity. I can imagine other users with slightly different use case or mindset thinking the exact opposite and being happy with the current implementation as easily.

[reflectionsv37]

Pavel,

Thank you so much for taking the time to explain this too me. I have now gained a much better understanding of not only layers, but how OpenCPN does things in general.

I had no idea that the GUID was universally unique and that the likelihood that you and I might generate the same one was so low as to be almost impossible. You learn something new everyday!!

Now that I understand what happened I fully agree that it is working as designed, and as desired. No, I certainly don't want 10 waypoints in the same location. The display indicating that imported waypoints were skipped would have given me a clue to look more closely at my layers file and will certainly do so in the future.

Since any piece of knowledge creates as many questions as it does answers, I have a couple more if you don't mind taking the time.

1) Is the random generation of the GUID unique to OpenCPN or do other chartplotters, etc also generate this unique identifier?

2) Can a layer be installed that does not contain GUID's? If so, does OpenCPN generate a GUID for those waypoints and is the layer file then updated? The help file says layer files are never updated, but it seems like waypoints in a layer would need a GUID for other uses.

3) I have noticed that when creating a route and you set a route waypoint close to an existing waypoint from a layer GPX it never asks the question "Do you want to use nearby waypoint?" I never answer yes to this question anyway because of what happens to a route if you delete that waypoint.

Thank you again for taking the time to explain this!!!

Gene

[transmitterdan]

Originally Posted by rgleason

Thanks Pavel for the clarification. "The online manual follows the merged master."
I'll fix it.

Rick,

Maybe I have misunderstood this comment. The online manual should not follow the merged master in my opinion. If you want to have a manual that follows the merged master that should be in git master.

Sometimes (including right now) the merged master is broken. (LLRegion has errors as does the patch that counts duplicate points). If the online documentation does not match the latest released installations there will be mostly confusion and more frustration as users attempt to build from master to get "new and neat features" as seen in the online documentation.

The documentation that should match master is that which is acquired via git IMO not the online web documentation. The web site should only document the latest release. Again, IMO.

[nohal]

Originally Posted by transmitterdan

Rick,



Maybe I have misunderstood this comment. The online manual should not follow the merged master in my opinion. If you want to have a manual that follows the merged master that should be in git master.



Sometimes (including right now) the merged master is broken. (LLRegion has errors as does the patch that counts duplicate points). If the online documentation does not match the latest released installations there will be mostly confusion and more frustration as users attempt to build from master to get "new and neat features" as seen in the online documentation.



The documentation that should match master is that which is acquired via git IMO not the online web documentation. The web site should only document the latest release. Again, IMO.

Then you just made OpenCPN not have documentation by creating a chicken and egg problem without an egg...

The static pages in git and packaged to the release are a snapshot of the online version of the wiki, with a lot of manual cleanup. Creating them is a very annoying manual process.

The documentation first has to be written, just then it can be exported and packaged... We write it in the wiki, wiki follows development as closely as possible, this has worked very well for us for years.

Pavel

[nohal]

Originally Posted by reflectionsv37

1) Is the random generation of the GUID unique to OpenCPN or do other chartplotters, etc also generate this unique identifier?

As far as I know other chartplotters do not use the same GPX extensions as OpenCPN. They usually address the same issue by using a completely proprietary format for navobject storage and if supporting GPX at all, they implement just the obligatory part of it. OpenCPN tries, by design, allow transfer of all the information it has and uses, in this open format.

2) Can a layer be installed that does not contain GUID's?

Yes, of course.

If so, does OpenCPN generate a GUID for those waypoints and is the layer file then updated? The help file says layer files are never updated, but it seems like waypoints in a layer would need a GUID for other uses.

In such a case GUIDs are temporary only for that session and generated as the layer objects are loaded. OpenCPN never changes the layer GPX file, by design.

3) I have noticed that when creating a route and you set a route waypoint close to an existing waypoint from a layer GPX it never asks the question "Do you want to use nearby waypoint?" I never answer yes to this question anyway because of what happens to a route if you delete that waypoint.

You could probably try and see for yourself, it would likely be more understandable than my explanation.
If you delete the waypoint, which at the same point is a route point from the chart canvas, it is deleted and removed from the route(s). If it should be a result of a batch operation (like Delete All from the Route & Mark Manager), OpenCPN asks you for confirmation (before actually deleting anything, one confirmation, regardless of the number of objects affected). At this point you can either cancel the whole batch operation, delete everything (while changing the routes and anchor alarms associated to the waypoints being deleted) or delete only the unused waypoints.
The other way around, if a route passes a waypoint and is deleted, the waypoint is not deleted with it, just the route point association is removed from it.

Pavel

[reflectionsv37]

Thanks again Pavel! Clear, concise and easy to understand explanations!

[rgleason]

Guys, I am just a yoke in the chicken and egg problem.
I think maybe we should have a reminder at the top of the Wiki. "We attempt to update this manual concurrently with the active merged master. The previous version of the manual has been included with your release download."

Just a suggestion..

Gene [Reflections37] you ask good questions that Pavel has provided. Wouldn't it be a good idea to share them in a more long term way? I don't have to the time to make a condensed statement of what you have learned. Do you think it would be possible to give us a good summary, and suggest where to put it in the manual? (That's the punishment for asking such good questions!)

If you could do so, I will make sure it gets done. Or I can make you and Editor Manager! Just PM me. Thanks.