Shipping plugin suggestion

[Larfalitl]

Hi,
Not sure if this is an appropriate place, so please point me in the right direction.

I stumbled upon www.shipmap.org, if you click "Show" in the top right, & chose "Routes" on the menu, you can see there are some places on the oceans you're much more likely to find shipping than others.
So my suggested plugin is to use this or more recent data, to shade or colour our chart by the probably of encountering shipping.

Having recently sailed from Brisbane to Sydney with AIS, we were quite sure it wasn't working. Until we got to Newcastle. Where we found heaps of ships. If we had of known in advance where the highest densities of them were, we may have chosen to go further East, or keep a tighter watch, or other action.

The Tasman sea, & many other areas, are crossed by distinct shipping lanes & it would to fantastic to know when you're approaching one.

Please add your thoughts.

Cheers

[seandepagnier]

Do they have data on what vessel type? Can you see the differences in different seasons?

[rgleason]

Sean,
They do have data on different shipping types, also by shipping month.
The website seems slow so it took awhile to download a month, loading "tiles". I was not able to note the differences between months. It would be nice if this data were open source, or available. Perhaps they would be willing to give us a consolidated year's worth, by month with ship types for the Climatology Plugin?

[Larfalitl]

The US coastal data is freely available by the month for 2009, 10, 11, 12, 13 & 14 here.
MarineCadastre.gov | Vessel Traffic Data
The data files are geodatabases by Universal Transverse Mercator (UTM) zone & contain location, time, ship type, speed, length, beam, and draught.
The Ship name and call sign fields have been removed, and the MMSI (Maritime Mobile Service Identity) field has been encrypted.

This may be a good start...

[Larfalitl]

The Australian coastal data for the past 18 years by the month is freely available here
https://www.operations.amsa.gov.au/S...es/DigitalData
There is no info on what the files contain other than it's AIS data.

[Larfalitl]

Couldn't find an other freely available data. We can buy the data later if need be, but let's see what we can do with what's available.

12 months worth of data for the whole of Australian waters in 2015 is only 500MB.
Jan data for the whole of the US for 2014 is 5100MB. So a years worth is roughly 60Gig.

http://marinecadastre.gov/ais/AIS%20...oneMap2014.png
Zones 1, 2, 4, 6, 7, 8 & 9 are all tiny, <50Meg per month
Zone 5 is 55Meg
Zone 3 is 75Meg
Zone 11, 14 & 19 are ~125Meg
16, 17 & 18 are 500Meg a month
10 is 800Meg
15 is 2Gig a month

Maybe do one of the mid sized US zones & the Australian data for a 12 month period & see how it looks?

I've got all Jan 2014 data for the US & all Aust data for 2015 if you'd like me to push it somewhere or roll a torrent. May be easier than messing with the website again.

[rgleason]

Larfalitl
That's great. Sean has techniques for compressing and consolidating this data. He was download internet limited, so I downloaded all the data and was the repository. He compressed 130gb (35 years of Noaa weather data into 7mb!). The way we did this was, I would send him a small piece of the data so he could analyze it, then he'd send me an executable for linux which I would run on the data with resultant files, then we would test them in the Climatology plugin he was developing.

Perhaps if Sean is willing to do this again for Shipping data, and if you run Linux, it would make most sense for you to work with him on this.

Best
Rick

[Larfalitl]

Yes, run Tanglu Linux.
That's a good strategy, clean & compress before downloading.
Happy to help, either way is good for me.

Cheers

[Skipper Web]

The reason for the gigantic data files is a sampling frequency of AIS reports. Don't know about Australian data source but NOAA's data files show reports at 1 minute interval. For the purposes of establishing shipping lanes this is simply wasteful.

There are other data source available, including global ones, but as you said, they are for-pay and the fee is typically based on the number of records requested. The tricks to reduce costs are to:
- restrict the area on interest;
- ask for data sampled at something like once-per-hour rate;
- limit types of ships.

I wrote some scripts for parsing NOAA's data some time ago. If you are interested ping me.

[seandepagnier]

If we can get all of the data for the entire world, even for say only 1 year, it would be possible to make a plugin for opencpn using it.

[Skipper Web]

Originally Posted by boat_alexandra

If we can get all of the data for the entire world, even for say only 1 year, it would be possible to make a plugin for opencpn using it.

The costs associated are non-trivial. Even excluding North America, since that's NOAA-covered, and with once-per-hour sampling we'd be talking about tens of thousands USD. AIS data is a hot commodity right now.

Approaching one of the AIS data aggregators with a request to sponsor the project in exchange for advertising within OCPN might be an option.

[rgleason]

Excellent Idea.

"Approaching one of the AIS data aggregators with a request to sponsor the project in exchange for advertising within OCPN might be an option. "

How about MarineTraffic? We could offer to identify MarineTraffic as the source of the data, with a link to their website in the About or Information area of the plugin.

I really think this would be very useful. Every 2-5 years perhaps they would agree to update the data.

[rgleason]

Larfalitl
Dimitry has provided us with his python script for extraction of the NOAA AIS data!
Perhaps it will help with the data you've downloaded too? Thankyou Dimitry Skipper Web

Please see this post
http://www.cruisersforum.com/forums/...ml#post2132497

Sean hope you can help once the data is available?

[seandepagnier]

Originally Posted by Skipper Web

The reason for the gigantic data files is a sampling frequency of AIS reports. Don't know about Australian data source but NOAA's data files show reports at 1 minute interval. For the purposes of establishing shipping lanes this is simply wasteful.

Not the only reason. They could sure invent a specialized binary format which uses a fraction of the space, not a raw ais stream.

There are other data source available, including global ones, but as you said, they are for-pay and the fee is typically based on the number of records requested. The tricks to reduce costs are to:
- restrict the area on interest;
- ask for data sampled at something like once-per-hour rate;
- limit types of ships.

Once per hour, and a few days a month may be sufficient. Areas of high shipping could get less requests, but it complicates the programming has to compensate for variable request density to correctly display the shipping.

Originally Posted by Skipper Web

The costs associated are non-trivial. Even excluding North America, since that's NOAA-covered, and with once-per-hour sampling we'd be talking about tens of thousands USD. AIS data is a hot commodity right now.

Approaching one of the AIS data aggregators with a request to sponsor the project in exchange for advertising within OCPN might be an option.

Ideally we would have the complete dataset. In any case... how do they get the data? Is it from satellites?

[rgleason]

I have written MarineTraffic with the attached proposal. I hope it is ok with everyone.

There is a section Apply for Historic Data
https://help.marinetraffic.com/hc/en...istorical-Data

but it is not appropriate for our proposal, so I just used their general form.

One of these images is a pretty good example. Maybe they have it already available?
Satellite AIS - Global AIS Coverage | AIS Marine Traffic