Cruisers Forum
 


Join CruisersForum Today

Reply
 
Thread Tools Rate Thread Display Modes
Old 19-05-2016, 05:57   #1
Registered User

Join Date: Sep 2013
Location: Sydney
Boat: Chincogan 40
Posts: 18
Shipping plugin suggestion

Hi,
Not sure if this is an appropriate place, so please point me in the right direction.

I stumbled upon www.shipmap.org, if you click "Show" in the top right, & chose "Routes" on the menu, you can see there are some places on the oceans you're much more likely to find shipping than others.
So my suggested plugin is to use this or more recent data, to shade or colour our chart by the probably of encountering shipping.

Having recently sailed from Brisbane to Sydney with AIS, we were quite sure it wasn't working. Until we got to Newcastle. Where we found heaps of ships. If we had of known in advance where the highest densities of them were, we may have chosen to go further East, or keep a tighter watch, or other action.

The Tasman sea, & many other areas, are crossed by distinct shipping lanes & it would to fantastic to know when you're approaching one.

Please add your thoughts.

Cheers
Attached Thumbnails
Click image for larger version

Name:	Shipping.jpg
Views:	97
Size:	409.0 KB
ID:	124563  
__________________

__________________
Larfalitl is offline   Reply With Quote
Old 20-05-2016, 19:34   #2
Registered User
 
boat_alexandra's Avatar

Join Date: Aug 2009
Location: Masachusetts
Boat: bristol 27
Posts: 2,802
Re: Shipping plugin suggestion

Do they have data on what vessel type? Can you see the differences in different seasons?
__________________

__________________
boat_alexandra is offline   Reply With Quote
Old 20-05-2016, 20:38   #3
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,314
Re: Shipping plugin suggestion

Sean,
They do have data on different shipping types, also by shipping month.
The website seems slow so it took awhile to download a month, loading "tiles". I was not able to note the differences between months. It would be nice if this data were open source, or available. Perhaps they would be willing to give us a consolidated year's worth, by month with ship types for the Climatology Plugin?
__________________
rgleason is offline   Reply With Quote
Old 21-05-2016, 01:14   #4
Registered User

Join Date: Sep 2013
Location: Sydney
Boat: Chincogan 40
Posts: 18
Re: Shipping plugin suggestion

The US coastal data is freely available by the month for 2009, 10, 11, 12, 13 & 14 here.
MarineCadastre.gov | Vessel Traffic Data
The data files are geodatabases by Universal Transverse Mercator (UTM) zone & contain location, time, ship type, speed, length, beam, and draught.
The Ship name and call sign fields have been removed, and the MMSI (Maritime Mobile Service Identity) field has been encrypted.

This may be a good start...
__________________
Larfalitl is offline   Reply With Quote
Old 21-05-2016, 01:25   #5
Registered User

Join Date: Sep 2013
Location: Sydney
Boat: Chincogan 40
Posts: 18
Re: Shipping plugin suggestion

The Australian coastal data for the past 18 years by the month is freely available here
https://www.operations.amsa.gov.au/S...es/DigitalData
There is no info on what the files contain other than it's AIS data.
__________________
Larfalitl is offline   Reply With Quote
Old 21-05-2016, 04:29   #6
Registered User

Join Date: Sep 2013
Location: Sydney
Boat: Chincogan 40
Posts: 18
Re: Shipping plugin suggestion

Couldn't find an other freely available data. We can buy the data later if need be, but let's see what we can do with what's available.

12 months worth of data for the whole of Australian waters in 2015 is only 500MB.
Jan data for the whole of the US for 2014 is 5100MB. So a years worth is roughly 60Gig.

http://marinecadastre.gov/ais/AIS%20...oneMap2014.png
Zones 1, 2, 4, 6, 7, 8 & 9 are all tiny, <50Meg per month
Zone 5 is 55Meg
Zone 3 is 75Meg
Zone 11, 14 & 19 are ~125Meg
16, 17 & 18 are 500Meg a month
10 is 800Meg
15 is 2Gig a month

Maybe do one of the mid sized US zones & the Australian data for a 12 month period & see how it looks?

I've got all Jan 2014 data for the US & all Aust data for 2015 if you'd like me to push it somewhere or roll a torrent. May be easier than messing with the website again.
__________________
Larfalitl is offline   Reply With Quote
Old 21-05-2016, 04:38   #7
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,314
Re: Shipping plugin suggestion

Larfalitl
That's great. Sean has techniques for compressing and consolidating this data. He was download internet limited, so I downloaded all the data and was the repository. He compressed 130gb (35 years of Noaa weather data into 7mb!). The way we did this was, I would send him a small piece of the data so he could analyze it, then he'd send me an executable for linux which I would run on the data with resultant files, then we would test them in the Climatology plugin he was developing.

Perhaps if Sean is willing to do this again for Shipping data, and if you run Linux, it would make most sense for you to work with him on this.

Best
Rick
__________________
rgleason is offline   Reply With Quote
Old 21-05-2016, 15:08   #8
Registered User

Join Date: Sep 2013
Location: Sydney
Boat: Chincogan 40
Posts: 18
Re: Shipping plugin suggestion

Yes, run Tanglu Linux.
That's a good strategy, clean & compress before downloading.
Happy to help, either way is good for me.

Cheers
__________________
Larfalitl is offline   Reply With Quote
Old 26-05-2016, 08:00   #9
Marine Service Provider

Join Date: Dec 2014
Posts: 46
Re: Shipping plugin suggestion

The reason for the gigantic data files is a sampling frequency of AIS reports. Don't know about Australian data source but NOAA's data files show reports at 1 minute interval. For the purposes of establishing shipping lanes this is simply wasteful.

There are other data source available, including global ones, but as you said, they are for-pay and the fee is typically based on the number of records requested. The tricks to reduce costs are to:
- restrict the area on interest;
- ask for data sampled at something like once-per-hour rate;
- limit types of ships.

I wrote some scripts for parsing NOAA's data some time ago. If you are interested ping me.
__________________
GoodAnchorageLawsuit.com
Skipper Web is offline   Reply With Quote
Old 27-05-2016, 09:06   #10
Registered User
 
boat_alexandra's Avatar

Join Date: Aug 2009
Location: Masachusetts
Boat: bristol 27
Posts: 2,802
Re: Shipping plugin suggestion

If we can get all of the data for the entire world, even for say only 1 year, it would be possible to make a plugin for opencpn using it.
__________________
boat_alexandra is offline   Reply With Quote
Old 27-05-2016, 09:40   #11
Marine Service Provider

Join Date: Dec 2014
Posts: 46
Re: Shipping plugin suggestion

Quote:
Originally Posted by boat_alexandra View Post
If we can get all of the data for the entire world, even for say only 1 year, it would be possible to make a plugin for opencpn using it.
The costs associated are non-trivial. Even excluding North America, since that's NOAA-covered, and with once-per-hour sampling we'd be talking about tens of thousands USD. AIS data is a hot commodity right now.

Approaching one of the AIS data aggregators with a request to sponsor the project in exchange for advertising within OCPN might be an option.
__________________
GoodAnchorageLawsuit.com
Skipper Web is offline   Reply With Quote
Old 27-05-2016, 10:02   #12
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,314
Re: Shipping plugin suggestion

Excellent Idea.

"Approaching one of the AIS data aggregators with a request to sponsor the project in exchange for advertising within OCPN might be an option. "

How about MarineTraffic? We could offer to identify MarineTraffic as the source of the data, with a link to their website in the About or Information area of the plugin.

I really think this would be very useful. Every 2-5 years perhaps they would agree to update the data.
__________________
rgleason is offline   Reply With Quote
Old 30-05-2016, 12:40   #13
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,314
Re: Shipping plugin suggestion

Larfalitl
Dimitry has provided us with his python script for extraction of the NOAA AIS data!
Perhaps it will help with the data you've downloaded too? Thankyou Dimitry Skipper Web

Please see this post
OpenCPN Beta Windows Plugins

Sean hope you can help once the data is available?
__________________
rgleason is offline   Reply With Quote
Old 30-05-2016, 22:23   #14
Registered User
 
boat_alexandra's Avatar

Join Date: Aug 2009
Location: Masachusetts
Boat: bristol 27
Posts: 2,802
Re: Shipping plugin suggestion

Quote:
Originally Posted by Skipper Web View Post
The reason for the gigantic data files is a sampling frequency of AIS reports. Don't know about Australian data source but NOAA's data files show reports at 1 minute interval. For the purposes of establishing shipping lanes this is simply wasteful.
Not the only reason. They could sure invent a specialized binary format which uses a fraction of the space, not a raw ais stream.
Quote:
There are other data source available, including global ones, but as you said, they are for-pay and the fee is typically based on the number of records requested. The tricks to reduce costs are to:
- restrict the area on interest;
- ask for data sampled at something like once-per-hour rate;
- limit types of ships.
Once per hour, and a few days a month may be sufficient. Areas of high shipping could get less requests, but it complicates the programming has to compensate for variable request density to correctly display the shipping.


Quote:
Originally Posted by Skipper Web View Post
The costs associated are non-trivial. Even excluding North America, since that's NOAA-covered, and with once-per-hour sampling we'd be talking about tens of thousands USD. AIS data is a hot commodity right now.

Approaching one of the AIS data aggregators with a request to sponsor the project in exchange for advertising within OCPN might be an option.
Ideally we would have the complete dataset. In any case... how do they get the data? Is it from satellites?
__________________
boat_alexandra is offline   Reply With Quote
Old 31-05-2016, 06:49   #15
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,314
Re: Shipping plugin suggestion

I have written MarineTraffic with the attached proposal. I hope it is ok with everyone.

There is a section Apply for Historic Data
https://help.marinetraffic.com/hc/en...istorical-Data

but it is not appropriate for our proposal, so I just used their general form.

One of these images is a pretty good example. Maybe they have it already available?
Satellite AIS - Global AIS Coverage | AIS Marine Traffic
Attached Files
File Type: doc OpenCPN-Climatology-MarineTraffic-Shipping-Monthly.doc (2.1 KB, 19 views)
__________________

__________________
rgleason is offline   Reply With Quote
Reply

Tags
plug, shipping

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Checking Shipping / Forwarding Company for Boat Shipping pred02 Product or Service Reviews & Evaluations 0 05-05-2011 08:28
Woman on board - new thread suggestion poll seagypsywoman General Sailing Forum 20 30-09-2008 09:31
forum suggestion seagypsywoman Forum Tech Support & Site Help 2 26-06-2005 14:59
suggestion: sailing schools cbare Forum Tech Support & Site Help 0 04-06-2003 10:41



Copyright 2002- Social Knowledge, LLC All Rights Reserved.

All times are GMT -7. The time now is 00:47.


Google+
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Social Knowledge Networks
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.

ShowCase vBulletin Plugins by Drive Thru Online, Inc.