Cruisers Forum
 


Join CruisersForum Today

Reply
 
Thread Tools Rate Thread Display Modes
Old 13-05-2017, 13:17   #1
Marine Service Provider

Join Date: Jun 2010
Posts: 27
Preferred file format for depth data?

Hi,
At TeamSurv we're just automating the output of our crowd sourced depth data into a number of digital formats.
We have a couple of options open to us, and I'd like some feedback on what people would prefer.

The obvious route is to use electronic chart formats such as BSB or S-57, but the drawback is that we are just producing depth data, not complete charts.

Alternatively we could do an overlay, e.g. as a GRIB file This is slightly abusing the GRIB format (we'd probably have to output it as temperature or something, as GRIB doesn't have a standard ID for water depth, and even if it did, it wouldn't be supported without modifying OpenCPN).

Some time later this year, when all is in place, charts will be updated daily automatically as new tracks are uploaded. If you want to participate, you just need to upload NMEA0183 data, which you can log using the VDR plugin, for example.

Thanks for any feedback,
Tim
__________________

__________________
TimThornton is offline   Reply With Quote
Old 13-05-2017, 14:51   #2
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,167
Re: Preferred file format for depth data?

There is a survey-pi plugin which was started by Pavel and Mike Rossiter took it a number of steps forward.
I did some testing for Mike, but at the moment cannot recall the format used. I am wondering if that format would be appropriate? Either Mike can advise or I can look up the format.
__________________

__________________
rgleason is offline   Reply With Quote
Old 13-05-2017, 14:52   #3
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,167
Re: Preferred file format for depth data?

PS Thanks Tim. Let me know if you'd like me to look it up.
__________________
rgleason is offline   Reply With Quote
Old 14-05-2017, 01:48   #4
Registered User

Join Date: May 2012
Posts: 391
Re: Preferred file format for depth data?

Tim,

Are you now saying that a newly submitted track will appear on the daily update?

Previously you were concerned that a number of submissions would have to be made from an area, to validate the quality of the data. At the time I argued against this because of the infrequent use of areas such as anchorages and rivers.

Survey_pi tried to display the soundings on the chart, but there was always the complication of tide and atmo corrections. VDR_pi seems a much simpler route for basic data gathering.

Survey_pi was able to export/import Hydromagic and 'xyz' text files. It would be nice to have the option of working with these formats from TeamSurv but I think a more useful option would be Grib or some other simple overlay.

Mike
__________________
Rasbats is offline   Reply With Quote
Old 14-05-2017, 03:37   #5
Marine Service Provider

Join Date: Jun 2010
Posts: 27
Re: Preferred file format for depth data?

Mike,
I'm saying that the updates to the output data set will be on the daily update.
I still hold that a number of tracks are required to have any confidence in the data - we've been working with a guy in Plymouth Uni on accuracies of crowd sourced data. Where we have a high data density errors are about +/-0.16m in about 20m dpth, whereas at lower densities (and still higher than a single track) this increases to 06m or more when compared to a multibeam survey carried out at the same time. And if it is just a single vessel that has not given us the correct value for transducer depth, it could be way out.
We have the existing processes for looking at the data quality before we output it, and removing cells from the output data set where the quality falls below a given threshold, in which case nothing in that cell is output.
Also, to avoid "update spam", if there are no changes to the output data set, or they are not significant (we're still playing with this, but a few areas changing by under 0.1m wouldn't be significant), then again we wouldn't issue an update.
For tides, on the predictions side we now have a composite model that merges coastal tidal stations for best inshore use with a gridded model that gives global offshore use, and also is used in areas where there are no coastal stations. Then we ingest tide gauge data (where available), compare against the predictions and apply a correction based on the difference - this takes into account atmospheric and other factors that cannot be predicted in the long term.
We can ingest xyz or other CSV formats (using our uploader software rather than via the web site), but prefer NMEA as then we know it hasn't been messed about with - some nav software is quite vague in how it processes depth data, obviously seeing the keel as the depth sounder of last resort!
__________________
TimThornton is offline   Reply With Quote
Old 17-05-2017, 13:55   #6
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,167
Re: Preferred file format for depth data?

Mike, how difficult would it be to export nmea 0183 depth/ position, files adjusted for tranducer depth and perhaps tide such that the data could be added to Team survey?
__________________
rgleason is offline   Reply With Quote
Old 17-05-2017, 15:22   #7
Registered User

Join Date: May 2012
Posts: 391
Re: Preferred file format for depth data?

Rick,
TeamSurv will have all the transducer installation info. They just use the VDR nmea file.
Mike
__________________
Rasbats is offline   Reply With Quote
Old 17-05-2017, 23:13   #8
Marine Service Provider

Join Date: Jun 2010
Posts: 27
Re: Preferred file format for depth data?

Rasbats,
That's right. We prefer to do our own correction for transducer depth, as we've found some software packages don't do this rigorously enough for our purposes. Also, our tide corrections are better than what can be achieved in on board systems as, for example, we ingest tide gauge data and apply a correction between predicted and actual sea levels.

It would be good if we could work towards better integration with OpenCPN, though. There could be a TeamSurv plug-in that would work through the initial setup/calibration, send data up to our server, and also download the latest depth data. However we're fully committed with our current projects until the end of this year, so that will have to wait (unless someone wlse would like to work with us to produce something?).

Tim
__________________
TimThornton is offline   Reply With Quote
Old 18-05-2017, 00:56   #9
bcn
Registered User

Join Date: May 2011
Location: underway whenever possible
Boat: Rangeboat 39
Posts: 2,746
Re: Preferred file format for depth data?

Tim,

a format that might suite OpenCPN and you:
S-57 overlay
We had a short discussion here:
OpenCPN Beta Version 4.5.213

You would need to produce your output as a S-57 chart. Perhaps some programming required, but the result would be ready not just for OpenCPN.

Hubert
__________________
bcn is offline   Reply With Quote
Old 18-05-2017, 01:44   #10
Marine Service Provider

Join Date: Jun 2010
Posts: 27
Re: Preferred file format for depth data?

Thanks Hubert,
We have the tools to generate S-57, and if OpenCPN can display it as an overlay then that is obviously better than repurposing the GRIB format.
__________________
TimThornton is offline   Reply With Quote
Old 18-05-2017, 01:46   #11
bcn
Registered User

Join Date: May 2011
Location: underway whenever possible
Boat: Rangeboat 39
Posts: 2,746
Re: Preferred file format for depth data?

Tim,

then I would suggest that you generate some S-57 overlays with the bathymetric data so that we can test the outputs.

Hubert
__________________
bcn is offline   Reply With Quote
Old 18-05-2017, 03:06   #12
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,167
Re: Preferred file format for depth data?

Ok, so teamserv corrects for transducer depth and tides.

So the process would be to record VDR files and upload them to teamserv?
Is it worthwhile to strip unwanted data from the file and perhaps compress it for upload?
What is initialization?
Perhaps a plugin that does this strips and compresses and uploads to teamserv so more users would contribute would help?
__________________
rgleason is offline   Reply With Quote
Old 18-05-2017, 04:05   #13
Marine Service Provider

Join Date: Jun 2010
Posts: 27
Re: Preferred file format for depth data?

Yes, that would be good.
To get a boat up and running, we set up an account on the server for them to be able to upload data. We also ask them to supply data such as make and model of GPS and depth sounder, transducer depth, and separation between GPS antenna and depth transducer.
As a minimum we need date/time, position and depth, and heading (pref compass or otherwise COG) is good. However in the long term we're also looking at processing more environmental data, so wind, temperature, and also with log, compas, COG and SOG we can derive the surface current.
We have a REST service for data upload, and sample code in C#/Xamarin.

Producing S-57 charts is next in the queue on the R&D side. But the format is fairly complex so we need to sort out what is actually required, and then how to set up FME and the Seven Cs plug-in to produce it from our data sets. But we'll upload something as soon as we think we're there.
__________________
TimThornton is offline   Reply With Quote
Old 22-05-2017, 11:56   #14
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,167
Re: Preferred file format for depth data?

Thank you Tim, this is very encouraging. Rasbats & BCN any thoughts about this?

This seems pretty ambitious with respect to data, but there is some relationship to both wind and tide in this information.
"COG and SOG we can derive the surface current" Separation into gribs would be great, however it is dependent on each ship's characteristics & activities at the time, really, so I don't understand why this particular information will be helpful in a general sense.
__________________
rgleason is offline   Reply With Quote
Old 23-05-2017, 00:42   #15
Marine Service Provider

Join Date: Jun 2010
Posts: 27
Re: Preferred file format for depth data?

Surface current (or tidal streams in tidal waters) is obviously important when coastal sailing - how far in or out do you go to stay in or out of the tide?
Going back a good few years to when I was doing software for race boats, we got quite a good handle on deriving this from log, compass, COG and SOG. Yes, boat characteristics have some effect on this, but we found the results better than the tidal diamonds on most charts. And even with HOs like SHOM producing tidal streams from hydrodynamic models, the on board sensor approach is likely to be better when very close in shore, as the topography is probably not modelled in fine enough detail in the models.
Although we haven't done work on it yet, I anticipate that by combining the results from many vessels the outcome will be more accurate than using data from a single boat, as in the past.
__________________

__________________
TimThornton is offline   Reply With Quote
Reply

Tags
depth

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Can Garmin format be converted into a format to use in OpenCPN..... Syserenity OpenCPN 1 16-07-2013 11:41
Preferred Chart Format ? PDA1 OpenCPN 18 06-07-2013 07:24
3.0.0 : make package : bad file right after installing the deb file manuprenlair OpenCPN 1 12-06-2012 05:17
How to Convert "*.route" Route File into ".*wpt" File To Be Used in Maxsea V.10 doiroda Navigation 6 10-07-2011 00:53
Garmin Format BlueChart Data Card - Purchasing Problem Darryl Navigation 15 08-10-2010 14:49



Copyright 2002- Social Knowledge, LLC All Rights Reserved.

All times are GMT -7. The time now is 06:52.


Google+
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Social Knowledge Networks
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.

ShowCase vBulletin Plugins by Drive Thru Online, Inc.