Cruisers Forum
 


Join CruisersForum Today

Reply
 
Thread Tools Rate Thread Display Modes
Old 23-02-2016, 22:15   #1
Marine Service Provider
 
bdbcat's Avatar

Join Date: Mar 2008
Posts: 4,878
GRIB2 PlugIn

Folks...

Its time to focus our efforts on bringing up GRIB2 file support for OCPN.

I have created a branch in the OpenCPN github account (called "grib2") to coordinate our efforts.
This branch was made as follows:

1. Started with current master (4.2 ++)
2. Cherry picked a few important commits from did-g's P.O.C branch. These commits implement the basic GRIB2 support. Thanks did-g! https://github.com/did-g/OpenCPN.github
3. Added a local source, stripped down build of jasper library elements to support JPEG2000 packing found in some GRIB2 files. I did this instead of simply linking to a common jasper library so that we can build easily on Windows and Mac.
Source: https://github.com/vmarkovtsev/jasper.git

4. did-g may want to amplify on the UI features new from his P.O.C. For one thing, it supports multi-file selection on the GRIB input file selector dialog. This is one way to study timed forecasts without manual concatenation. There is also some time-scale interpolation implemented, though untested by me.

The branch currently builds on linux only. Almost builds on Windows, needs some slight config tweaks. Next on my list. Probably builds on Mac, but not tested yet.

Current status:

1. Tested very minimally on simple single-time Wind GRIBs from GFS. Works OK.
2. JPEG2000 decode untested. I don't know what GRIB provider/model/parameters are typically so encoded. Anybody?
3. I have initiated a Flyspray task for this project.
4. And of course Pull Requests against this branch are accepted.

So, what we need to do now is test, test, and more test. Providers, models, and parameters of all flavors should be looked at. Regression testing against GRIB1 files is critical. UI issues must be ironed out, etc. etc. The plan would be to introduce this PlugIn as a standalone upgrade to O42, and simultaneously announce a 4.2.1 version of base OCPN comprising all known bug fixes at that time.

This is our plan forward for GRIB2 support.
I look forward to your comments and help.

Thanks
Dave
__________________

__________________
bdbcat is offline   Reply With Quote
Old 24-02-2016, 01:14   #2
bcn
Registered User

Join Date: May 2011
Location: underway whenever possible
Boat: Rangeboat 39
Posts: 2,752
Re: GRIB2 PlugIn

Dave...

what nice news for breakfast!

Quote:
Originally Posted by bdbcat View Post
Folks...

Its time to focus our efforts on bringing up GRIB2 file support for OCPN.

I have created a branch in the OpenCPN github account (called "grib2") to coordinate our efforts.
This branch was made as follows:

1. Started with current master (4.2 ++)
2. Cherry picked a few important commits from did-g's P.O.C branch. These commits implement the basic GRIB2 support. Thanks did-g! https://github.com/did-g/OpenCPN.github
A small typo:
https://github.com/did-g/OpenCPN
Quote:
4. did-g may want to amplify on the UI features new from his P.O.C. For one thing, it supports multi-file selection on the GRIB input file selector dialog. This is one way to study timed forecasts without manual concatenation. There is also some time-scale interpolation implemented, though untested by me.
Folllowing Rasbats comments from here
oGrib2 Plugin using the ECMWF grib_api

it would be very interesting to support the Copernicus/Sentinel route.
Quote:
Weather4D sources this data from the EU CMEMS project (Copernicus), where it is freely available as NetCDF (nc) format files. These cover a large area of Western Europe and are potentially more useful than the BSH Grib 2 files.
Copernicus with Sentinel1A and 2A in the space is covering the whole planet, Sentinel 3 has just been launched and will complement with additonal environmental data data.
Sentinel 1B and 2B are planned to follow in 2016 and 2017.
All data from Copernicus are free.

More about NetCDF here:
OGC network Common Data Form (netCDF) standards suite | OGC

Expect METOCEAN data to be available from multiple sources in the future in NetCDF.

Hubert
__________________

__________________
bcn is online now   Reply With Quote
Old 24-02-2016, 04:02   #3
Registered User
 
Gilletarom's Avatar

Join Date: Mar 2010
Location: France
Boat: 10.50 mètres
Posts: 2,294
Re: GRIB2 PlugIn

Rasbats is already capable of displaying files in copernicus with OpenCPN. But this is very basic. Moreover, downloading files at Copernicus is quite complex.

By example, this morning, no download possible.

When to Weather4D, it seems to have stopped putting free of charge, available to the general public after Copernicus their Grib file. This is quite logical since Weather4D is a commercial enterprise.
__________________
Gilletarom is offline   Reply With Quote
Old 24-02-2016, 05:12   #4
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,320
Re: GRIB2 PlugIn

Mike R. has been helping me build a plugin for grib2 from his github and I have been picking my way through the process several times. Right now I am working on building his most recent version. I now have an account at NetCDF Unidata | Participation Policy and Copernicus - Marine environment monitoring service

Can someone give us a summary of how these various efforts dovetail and what the overall goal is, please?
1. There is a report that work on grib1 is ended and grib2 started. -What does that mean? Are we going to loose the interface Jean Pierre & Sean worked on?
2. There is a report that some data is no longer freely available from Weather4D, I don't understand how that affects us, as most data is from NOAA (imho) even though Weather4D is excellent. How would this affect our use of grib?
3. What are the goals for the grib2 effort? This may be readily apparent to many, but not to me as there appear to be multiple efforts, repositories, and data sources.
4. How can we best help?
__________________
rgleason is offline   Reply With Quote
Old 24-02-2016, 08:06   #5
Marine Service Provider
 
bdbcat's Avatar

Join Date: Mar 2008
Posts: 4,878
Re: GRIB2 PlugIn

Rick...

There are many proposed alternatives to simple GRIB1/2 for distribution of meteo data. Some are very interesting and workable now, some are unimplemented proposals.

As I mentioned above, the project described here is an effort to bring simple GRIB2 to the existing OCPN grib_pi framework. We will do this while maintaining compatibility with the existing GRIB1 formats, and retain the well understood UI for OCPN.

We will limit our scope so that this goal is reachable in a reasonable time frame, with minimal dependencies on undefined or moving interface specifications. And so we can actually ship something in the next month or so.

This effort will not involve NetCDF file decoding, nor the ECMWF grib_api. That would be a separate project, with goals and timing TBD. Not discussed here.


How to help:
1. linux users, build the branch right now, and test some GRIB2 files as downloaded from whatever source suits you.

2. Windows users, stand by a day or so until the MSW build works. Rick can then build and place the OCPN installer on the file thingie. And then Windows users can test GRIB2 as above.

Focus, guys

Thanks
Dave
__________________
bdbcat is offline   Reply With Quote
Old 24-02-2016, 08:42   #6
Registered User
 
Gilletarom's Avatar

Join Date: Mar 2010
Location: France
Boat: 10.50 mètres
Posts: 2,294
Re: GRIB2 PlugIn

hello All, Hello RGLeason
Quote:
Originally Posted by rgleason View Post
...
2. There is a report that some data is no longer freely available from Weather4D, I don't understand how that affects us, as most data is from NOAA (imho) even though Weather4D is excellent. How would this affect our use of grib?
...
Copernicus is a structure set up by the EC (European Community). This structure appears to be aimed to bring together all existing data and studies in meteorology and the study of the sea. And all these data are currently being free in "Open data".

They are free of charge made available to all who need them. Whether scientific, industrial, or particularly like us.

Theoretically, if I read the information in the Copernicus Charter, any user of this data must make them free too, available to all other users.

This is clearly what the developers of software Weather4D made early.

But the developers of other software fetched data in Weather4D. This is even how it started in OpenCPN. Obviously, it is understandable that Weather4D do not agree to that. server costs, and downloading work remained "in charge" of Weather4D.

I think that explains why Weather4D has stopped putting the data online through its website.
__________________
Gilletarom is offline   Reply With Quote
Old 24-02-2016, 11:09   #7
Marine Service Provider
 
bdbcat's Avatar

Join Date: Mar 2008
Posts: 4,878
Re: GRIB2 PlugIn

Testers...

Here is a good place to get GRIB2 GFS files:

NOMADS-NOAA Operational Model Archive and Distribution System

Use the "grib filter" interface to select values and geographical regions.

Dave
__________________
bdbcat is offline   Reply With Quote
Old 24-02-2016, 12:23   #8
Registered User
 
Gilletarom's Avatar

Join Date: Mar 2010
Location: France
Boat: 10.50 mètres
Posts: 2,294
Re: GRIB2 PlugIn

Quote:
Originally Posted by bdbcat View Post
Testers...
Here is a good place to get GRIB2 GFS files:
NOMADS-NOAA Operational Model Archive and Distribution System
Use the "grib filter" interface to select values and geographical regions.
Dave
And here some other Grib file for Currents. With two folder : Grib 1 and Grib 2.

ftp://ftp.bsh.de/Stroemungsvorhersagen/

In the Grib1 folder, there is Grib files, for a day at a time, for reasons of 3 days, which are extracted from GRIB2 files from the other folder.

There is in each folder, a "txt" file with some explanations.

Note that these Grib1 are, actually, well displayed by the plugin Grib, V 3.0 of OpenCPN. So, it is possible to make test for comparison of display.

These Grib cover, in Europe sea : Channel, North Sea, Baltic area.
Attached Thumbnails
Click image for larger version

Name:	2016-02-24-OpenCPN 4.2.0_Grib_Baltic_sea.jpg
Views:	96
Size:	401.9 KB
ID:	119544  
__________________
Gilletarom is offline   Reply With Quote
Old 24-02-2016, 15:29   #9
Registered User

Join Date: May 2012
Posts: 405
Re: GRIB2 PlugIn

Great to see the progress for Grib2 data.

Using CDO (Climate Data Objects) it is possible to translate NetCDF to Grib1/2. However, I have not been able to read the Grib 1 files made by this program, unless I use my own code (grib_api). u and v variable names for current values in netcdf translate to var1 and var2 in the Grib, which are not recognised by the present Grib plugin.

Could the new Grib2 plugin accomodate translations made by CDO? If so, it would be possible to make a small converter program to prepare files for OpenCPN.

An example of one translation is attached.

Mike
Attached Files
File Type: pdf ibi.grb.pdf (567.3 KB, 25 views)
__________________
Rasbats is offline   Reply With Quote
Old 24-02-2016, 15:57   #10
Registered User
 
Gilletarom's Avatar

Join Date: Mar 2010
Location: France
Boat: 10.50 mètres
Posts: 2,294
Re: GRIB2 PlugIn

Hello rasbats,
Quote:
Originally Posted by Rasbats View Post
Great to see the progress for Grib2 data.
...
An example of one translation is attached.
Mike
After put out ".pdf", It is impossible to display it with Grib (V3) and of course whith "Grib2 0.1". it is empty !
__________________
Gilletarom is offline   Reply With Quote
Old 24-02-2016, 15:59   #11
Registered User

Join Date: Jun 2015
Posts: 198
Re: GRIB2 PlugIn

Quote:
Originally Posted by Rasbats View Post
Great to see the progress for Grib2 data.

Using CDO (Climate Data Objects) it is possible to translate NetCDF to Grib1/2. However, I have not been able to read the Grib 1 files made by this program, unless I use my own code (grib_api). u and v variable names for current values in netcdf translate to var1 and var2 in the Grib, which are not recognised by the present Grib plugin.

Could the new Grib2 plugin accomodate translations made by CDO? If so, it would be possible to make a small converter program to prepare files for OpenCPN.

An example of one translation is attached.

Mike
In my understanding the bug is in cdo but you can convert them, I'm using:
Code:
# convert netcf
cdo -f grb2 copy IBI.nc IBI.grb
# split by type
grib_copy IBI.grb 'IBI_[shortName].grib'

# fixup
grib_set -s discipline=10,parameterCategory=1,dataTime=30,typeOfProcessedData=1 IBI_u.grib IBI_ucurr.grb

grib_set -s discipline=10,parameterCategory=1,dataTime=30,typeOfProcessedData=1 IBI_v.grib IBI_vcurr.grb
I think you can convert them in grib1 too with grib_set -s editionNumber=1
may or may not work...

URLs in my previous post are still active, IBI current, SHOM current and wave, Meteo France Arome and Arpege.
__________________
did-g is offline   Reply With Quote
Old 24-02-2016, 19:37   #12
Registered User
 
rgleason's Avatar

Join Date: Mar 2012
Location: Boston, MA
Boat: 1981 Bristol 32 Sloop
Posts: 9,320
Re: GRIB2 PlugIn

Thanks to bdbcat, mike, bcn and gille and Did-g. This promises to be a rewarding area for improvements. I look forward to helping.
__________________
rgleason is offline   Reply With Quote
Old 24-02-2016, 21:44   #13
Marine Service Provider
 
bdbcat's Avatar

Join Date: Mar 2008
Posts: 4,878
Re: GRIB2 PlugIn

Testers...

As mentioned, I have uploaded a Windows version of OCPN 4.2.1 with the integrated GRIB2 compatible PlugIn.

Here it is:

http://opencpn.navnux.org/beta/opencpn_4.2.1_setup.exe

Enjoy!
Dave
__________________
bdbcat is offline   Reply With Quote
Old 25-02-2016, 03:47   #14
Registered User
 
boat_alexandra's Avatar

Join Date: Aug 2009
Location: Masachusetts
Boat: bristol 27
Posts: 2,803
Re: GRIB2 PlugIn

Quote:
Originally Posted by bdbcat View Post
Rick...

There are many proposed alternatives to simple GRIB1/2 for distribution of meteo data. Some are very interesting and workable now, some are unimplemented proposals.
I would like to propose a format which stores the wind data in polar form instead of vector, and uses a logarithmic scale for wind speed. This means using 2 bytes (one for speed one for direction) for a very accurate measurement, or possibly fewer bits. The data could then be deinterlaced to give better compression results.

I think this file would be a lot smaller than grib with identical useful accuracy.
__________________
boat_alexandra is offline   Reply With Quote
Old 25-02-2016, 04:25   #15
bcn
Registered User

Join Date: May 2011
Location: underway whenever possible
Boat: Rangeboat 39
Posts: 2,752
Re: GRIB2 PlugIn

Sean...

are you aware of the CDO tool set mentioned by did-g ?

https://code.zmaw.de/projects/cdo/wiki

And have a safe trip!

Hubert
__________________

__________________
bcn is online now   Reply With Quote
Reply

Tags
plug, rib

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
OpenCPN PlugIn Development bdbcat OpenCPN 78 29-07-2017 03:15
Plugin: DashBoard SethDart OpenCPN 470 24-08-2016 21:34
BSB 4 PlugIn bdbcat OpenCPN 93 20-02-2016 04:41
QTVLM Route Optimizer and GRIB2 sinbad7 OpenCPN 1 26-07-2013 01:13
Writing Plugin for nv.digital Map Format ? Netsurfer OpenCPN 7 17-08-2010 17:09



Copyright 2002- Social Knowledge, LLC All Rights Reserved.

All times are GMT -7. The time now is 04:58.


Google+
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Social Knowledge Networks
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.

ShowCase vBulletin Plugins by Drive Thru Online, Inc.