OpenCPN Survey Plugin Depths and VentureFarther

[jonturgeon]

I am the developer of the site VentureFarther. One thing this site does is online satellite kap creation/download. One of the things I have always wanted with sat kaps are depths. So, I have started down the path of integrating crowd sourced depths into kaps. My first step was to get the depth info into my site. For the first release I accept files created by the OpenCPN survey plugin (I am hoping to build an Android recorder to help increase user participation). I then process the depths and offset for tide and store in a spatial MySQL database. For viewing online I overlay these recorded depths using leaflet with several different map layers. Here is an example. This is as far as I have gotten as I wanted to be able to upload surveys that I generate and also test the user participation before I spend a bunch more time with the kap creation.

For the kap creation I am hoping to create a custom MapBox depth layer that can be merged with the sat image at kap creation time. Will need to do some more looking on this as I have never done that before.

As always, I am open to any feedback.

[Rasbats]

Impressive.

Are those 10.2 soundings an aberration or are the depths significantly different from those shown by Navionics?

The work on the Linux side of survey_pi has just moved up the 'to-do' list! If anyone wants to pitch in with my fork of Nohal's survey_pi or make a better fork please shout. Happy to accept pull requests. Kudos Pavel for getting survey_pi started.

Mike

[jonturgeon]

Hi Mike, yup the 10.2m is correct. I thought Greece would be charted well but outside of the main ports I am finding the charting to be sketchy. I'll be uploading more surveys are the time goes.

If anyone wants guidance on how to contribute please just let me know. I'll probably update VentureFarther with more detailed "How To" but currently internet access is a challenge.

On a different note, if there are any bored OpenCPN plugin developers out there that would like to develop a kap download plugin from VentureFarther just let me know. I already have a web service in place to do this, the plugin would need to make a web service call with location, zoom and satellite provider. Then save the chart off to the charting folder.

[seandepagnier]

How are you converting from tiles to kap?

I intend to support mbtiles as a native chart format in opencpn soon which will allow transparent charts as well as png and jpg compression.

crowd sourced depths are a start, but right now few people have depth sounders with nmea output. Further, few are actually making survey covering entire areas. I think a small remote control boat could do this better.

[jonturgeon]

I have been in contact with the MapBox support team and am starting to look into vector tiles for the sounding info. If this works out I can use java to merge the depth image with the sat image. Will start working on that once I get a better internet connection, if depths are being uploaded.

I think you would be surprised with the number of people with NMEA to wifi onboard. With most cruisers now with an iPad and Navionics a lot have added this (we have). I think the bigger issue will be making it easy to record these depths with equipment people already have, my support for in OpenCPN was a start (we use it for all of our navigation but it seemed like not many others were) but I really think I need to support Android (and I might have a way for the iPad also). I will start work on an Android app once I figure how

These depths will never be an official government survey. But I know I have been in numerous areas that the official charts either had no soundings or very minimal which were not really accurate. I also have been in areas that the cruising guides sketch charts were so off that you wonder if the author had ever been there. These are the areas that I hope this will work best for. I also plan o n adding an option to display the recorders track so you know how the soundings were taken. I also plan on adding these soundings to the anchorage display page (VF also does marker sharing) if soundings are available.

[bcn]

Interesting terrain...

Originally Posted by jonturgeon

I have been in contact with the MapBox support team and am starting to look into vector tiles for the sounding info. If this works out I can use java to merge the depth image with the sat image. Will start working on that once I get a better internet connection, if depths are being uploaded.

Did MapBox give you access to some paper or documentation about vector maps and mbTiles? The information I found up today is sparse to non-existent.
Which vector format you want to apply for the data - or graphics?
I understand that you will render the resulting overlays then again as raster/.kap - at least OCPN does not understand vector data other than S-57 (and CM93) as of today.
And these formats do not allow transparent overlays neither.

I think you would be surprised with the number of people with NMEA to wifi onboard. With most cruisers now with an iPad and Navionics a lot have added this (we have). I think the bigger issue will be making it easy to record these depths with equipment people already have, my support for in OpenCPN was a start (we use it for all of our navigation but it seemed like not many others were) but I really think I need to support Android (and I might have a way for the iPad also). I will start work on an Android app once I figure how

SignalK would be handy in this context, as it is about communication, can forward and distribute data, knows about different data sources and formats, provides time stamps, identifies the originator and allows to filter.
Node.js based.
Again, there is no direct support for SK in OCPN yet, but as the result will be charts...

These depths will never be an official government survey. But I know I have been in numerous areas that the official charts either had no soundings or very minimal which were not really accurate. I also have been in areas that the cruising guides sketch charts were so off that you wonder if the author had ever been there. These are the areas that I hope this will work best for. I also plan o n adding an option to display the recorders track so you know how the soundings were taken. I also plan on adding these soundings to the anchorage display page (VF also does marker sharing) if soundings are available.

As you are underlining, these are ad-hoc data, not bathymetric material.
Nothing against it, but the user should be very aware about it.
It needs postprocessing - you are applying tides, but not heave nor meteo related offsets.
Perhaps the data source/date of survey should be retrievable.

How do want to handle different inputs for the same zone? If conflicting?
Kind of quality control?


More bits and pieces:
You know about Teamsurv?
TeamSurv - Home

What do you think about the Navionics crowd sourced bathymetry data?

Hubert

[Rasbats]

Hubert,

After many requests I failed to get any of my data exported back from TeamSurv, although several tracks were submitted. I wanted to use that data for development work. At one point I did a 'bar check' to validate my data and submitted my findings but the processed data visible on the TeamSurv website was not updated to reflect this check. A black hole.

I have stopped contributing to TeamSurv.

Mike

[bcn]

My impression from their website - tracks contributed by TeamSurv - gave an impression that fits well with your observations.

Mentioned TeamSurv because it is quite a time around, the basic idea about crowdsourced bathymetry and the difficulties are described, but yes, it looks a bit dead. For the leisure sailor.

Tim Thornton, the visible face of TeamSurv has been around here quite a times.
Perhaps we can shed some light on the actual situation of this project.

[TimThornton]

Hi,
I don't know if it is the same person as Raspbats, but we had a Mike who contributed a couple of tracks last year and wanted the output bathymetry, but as we explained to him at the time our data processing relies on getting a significant amount of data in an area, so we can have confidence in the results, and he stopped uploading before we had enough data to produce anything in the area. He came to see us at the Shallow Survey conference in Plymouth, and I quickly got the impression that his objective wasn't to contribute to TeamSurv, but to develop his own system for depth processing.

Yes, we're still going - we've been a bit quiet on recruiting boats etc recently as funding has been low and had to concentrate on paying work to meet the bills, but we've quietly been working with the likes of the IHO and GEBCO who are now supporting the use of crowd sourced bathymetry. The IHO have a working group set up on crowd sourced bathymetry, and we're contributing to that, and we're just finalising things to send our data to GEBCO for their global ocean maps.

Things are looking better now, though. We're part of a project looking at both satellite derived bathymetry and crowd sourced bathymetry, see BASE Platform - Home. We're just starting recruiting boats in the Channel Islands, Wadden See, Azores, Balearics and Mauritius for this project, plus we're redoing the web site and also working on finishing automating the processing of data into charts so instead of downloadable PDFs updated whenever I have time (i.e. not often) this will be dne automatically, viewable online, and also downloadable in a umber of formats like BSB, S-57, AT5 etc.

We're also doing some trials on sending data ashore via AIS monitored by satellite - we're looking for a couple of boats to participate, and in so doing get a Class A AIS transponder to keep - follow the link on our home page to the form is you are interested.

Finally, we're about to recruit someone to spread the word about TeamSurv via the yachting press, social media, and into other groups like fishermen, shipping etc, so this should raise our profile and get more boats signed up.

Tim

[bcn]

Originally Posted by TimThornton

We're also doing some trials on sending data ashore via AIS monitored by satellite - we're looking for a couple of boats to participate, and in so doing get a Class A AIS transponder to keep - follow the link on our home page to the form is you are interested.

Tim

Tim,

can you fill this a bit more:
AIS with bathymetry data as payload? Which message type?
AIS-Sat is usually not realtime - there aren't that much LowOrbiting sats for coverage (the NORSAT-1 that should have provided two reception packages being launched with Sentinel-1B is delayed). Data buffering? OK, bathymetry in real time is not common neither. And bandwidth available for this? Who captures this data?

Curious...

Hubert

[TimThornton]

Hubert,
It is going into message 8. We're working with ExactEarth, who don't have full coverage at the moment, but have a bunch more satellites being launched soon that will give full coverage in a year or so. We've been running trials from the office roof for a while now and the AIS transponder sends a message every 5 seconds, and the trials at sea will give us a better handle on how many of these are picked up by the satellite etc, but obviously not enough bandwidth for buffering. We get the data with about an hour's delay, but haven't done anything to try and improve this.
As I understand it, none of the non-government AIS coastal monitoring networks do anything with message 8 at present.
We're interested in this as it means data can be collected with no user input - particularly important to get onto ships - and minimal comms costs, much less equipment and comms costs than ARGOS or Iridium or Sat C.
If the trials work out, we hope to get funding for a much wider deployment, and also extending from depth to other parameters, e.g. wind, pressure, current, and possibly also water quality.
And looking ahead to VDES we should have much higher bandwidth...

[Skipper Web]

Originally Posted by jonturgeon

... I then process the depths and offset for tide and store in a spatial MySQL database. For viewing online I overlay these recorded depths using leaflet with several different map layers. Here is an example.

Cool website. Leaflet is a great library and can do amazing things. You might want to disable that blue polygon around marker clusters. I can't recall the right setting, but will look it up.

That said, why did you settle on MySQL? PostGIS is a much more powerful platform for doing spatial queries and calculations.

I did a similar project for a now defunct site. We used to collect anchorage data. The system was built around Drupal 7 on MySQL, but the mapping / spatial stuff was built with PostGIS and Leaflet. Although the venture failed I've retained all rights to the source code. Let me know if you want to pillage it.

[Rasbats]

Tim,

All well and good but this does not help the cruising yacht who frequents isolated anchorages or rivers. By the nature of these transits the data will always be sparse.

Mike

[TimThornton]

Mike,
In our experience crowd sourced data is always quite noisy - variable quality depth sounders, the effects of heel and heave, differing accuracies in calibration etc. In the limit, if you just have one boat going into an anchorage, depths may be as much as a meter out either way. If you have a number of tracks, and preferably a number of boats, then you can treat things statistically to check that there is some degree of confidence in the data, rather than just publishing it on the assumption that it is correct - as with just a single track there is nothing to check against.
Our approach ensures good quality data but doesn't output results where the data is too sparse or inconsistent; other approaches publish more data, and so cover the chart more quickly, but some of it may be of unknown quality. Neither is right or wrong, just different approaches.

[rgleason]

With survey_pi and this website plus the programming horsepower and developer interest, I have no doubt this could become a great tool. Skipper Web has developed code for Good Harbor_pi, so that could become part of it too.

Another note: Tried to register, no success. Anyone else succeed? Will try again later.