GRIB2 PlugIn

[bdbcat]

Folks...

Its time to focus our efforts on bringing up GRIB2 file support for OCPN.

I have created a branch in the OpenCPN github account (called "grib2") to coordinate our efforts.
This branch was made as follows:

1. Started with current master (4.2 ++)
2. Cherry picked a few important commits from did-g's P.O.C branch. These commits implement the basic GRIB2 support. Thanks did-g! https://github.com/did-g/OpenCPN.github
3. Added a local source, stripped down build of jasper library elements to support JPEG2000 packing found in some GRIB2 files. I did this instead of simply linking to a common jasper library so that we can build easily on Windows and Mac.
Source: https://github.com/vmarkovtsev/jasper.git

4. did-g may want to amplify on the UI features new from his P.O.C. For one thing, it supports multi-file selection on the GRIB input file selector dialog. This is one way to study timed forecasts without manual concatenation. There is also some time-scale interpolation implemented, though untested by me.

The branch currently builds on linux only. Almost builds on Windows, needs some slight config tweaks. Next on my list. Probably builds on Mac, but not tested yet.

Current status:

1. Tested very minimally on simple single-time Wind GRIBs from GFS. Works OK.
2. JPEG2000 decode untested. I don't know what GRIB provider/model/parameters are typically so encoded. Anybody?
3. I have initiated a Flyspray task for this project.
4. And of course Pull Requests against this branch are accepted.

So, what we need to do now is test, test, and more test. Providers, models, and parameters of all flavors should be looked at. Regression testing against GRIB1 files is critical. UI issues must be ironed out, etc. etc. The plan would be to introduce this PlugIn as a standalone upgrade to O42, and simultaneously announce a 4.2.1 version of base OCPN comprising all known bug fixes at that time.

This is our plan forward for GRIB2 support.
I look forward to your comments and help.

Thanks
Dave

[bcn]

Dave...

what nice news for breakfast!

Originally Posted by bdbcat

Folks...

Its time to focus our efforts on bringing up GRIB2 file support for OCPN.

I have created a branch in the OpenCPN github account (called "grib2") to coordinate our efforts.
This branch was made as follows:

1. Started with current master (4.2 ++)
2. Cherry picked a few important commits from did-g's P.O.C branch. These commits implement the basic GRIB2 support. Thanks did-g! https://github.com/did-g/OpenCPN.github

A small typo:
https://github.com/did-g/OpenCPN

4. did-g may want to amplify on the UI features new from his P.O.C. For one thing, it supports multi-file selection on the GRIB input file selector dialog. This is one way to study timed forecasts without manual concatenation. There is also some time-scale interpolation implemented, though untested by me.

Folllowing Rasbats comments from here
http://www.cruisersforum.com/forums/...ml#post2053011

it would be very interesting to support the Copernicus/Sentinel route.

Weather4D sources this data from the EU CMEMS project (Copernicus), where it is freely available as NetCDF (nc) format files. These cover a large area of Western Europe and are potentially more useful than the BSH Grib 2 files.

Copernicus with Sentinel1A and 2A in the space is covering the whole planet, Sentinel 3 has just been launched and will complement with additonal environmental data data.
Sentinel 1B and 2B are planned to follow in 2016 and 2017.
All data from Copernicus are free.

More about NetCDF here:
OGC network Common Data Form (netCDF) standards suite | OGC

Expect METOCEAN data to be available from multiple sources in the future in NetCDF.

Hubert

[Gilletarom]

Rasbats is already capable of displaying files in copernicus with OpenCPN. But this is very basic. Moreover, downloading files at Copernicus is quite complex.

By example, this morning, no download possible.

When to Weather4D, it seems to have stopped putting free of charge, available to the general public after Copernicus their Grib file. This is quite logical since Weather4D is a commercial enterprise.

[rgleason]

Mike R. has been helping me build a plugin for grib2 from his github and I have been picking my way through the process several times. Right now I am working on building his most recent version. I now have an account at NetCDF Unidata | Participation Policy and Copernicus - Marine environment monitoring service

Can someone give us a summary of how these various efforts dovetail and what the overall goal is, please?
1. There is a report that work on grib1 is ended and grib2 started. -What does that mean? Are we going to loose the interface Jean Pierre & Sean worked on?
2. There is a report that some data is no longer freely available from Weather4D, I don't understand how that affects us, as most data is from NOAA (imho) even though Weather4D is excellent. How would this affect our use of grib?
3. What are the goals for the grib2 effort? This may be readily apparent to many, but not to me as there appear to be multiple efforts, repositories, and data sources.
4. How can we best help?

[bdbcat]

Rick...

There are many proposed alternatives to simple GRIB1/2 for distribution of meteo data. Some are very interesting and workable now, some are unimplemented proposals.

As I mentioned above, the project described here is an effort to bring simple GRIB2 to the existing OCPN grib_pi framework. We will do this while maintaining compatibility with the existing GRIB1 formats, and retain the well understood UI for OCPN.

We will limit our scope so that this goal is reachable in a reasonable time frame, with minimal dependencies on undefined or moving interface specifications. And so we can actually ship something in the next month or so.

This effort will not involve NetCDF file decoding, nor the ECMWF grib_api. That would be a separate project, with goals and timing TBD. Not discussed here.


How to help:
1. linux users, build the branch right now, and test some GRIB2 files as downloaded from whatever source suits you.

2. Windows users, stand by a day or so until the MSW build works. Rick can then build and place the OCPN installer on the file thingie. And then Windows users can test GRIB2 as above.

Focus, guys

Thanks
Dave

[Gilletarom]

hello All, Hello RGLeason

Originally Posted by rgleason

...
2. There is a report that some data is no longer freely available from Weather4D, I don't understand how that affects us, as most data is from NOAA (imho) even though Weather4D is excellent. How would this affect our use of grib?
...

Copernicus is a structure set up by the EC (European Community). This structure appears to be aimed to bring together all existing data and studies in meteorology and the study of the sea. And all these data are currently being free in "Open data".

They are free of charge made available to all who need them. Whether scientific, industrial, or particularly like us.

Theoretically, if I read the information in the Copernicus Charter, any user of this data must make them free too, available to all other users.

This is clearly what the developers of software Weather4D made early.

But the developers of other software fetched data in Weather4D. This is even how it started in OpenCPN. Obviously, it is understandable that Weather4D do not agree to that. server costs, and downloading work remained "in charge" of Weather4D.

I think that explains why Weather4D has stopped putting the data online through its website.

[bdbcat]

Testers...

Here is a good place to get GRIB2 GFS files:

NOMADS-NOAA Operational Model Archive and Distribution System

Use the "grib filter" interface to select values and geographical regions.

Dave

[Gilletarom]

Originally Posted by bdbcat

Testers...
Here is a good place to get GRIB2 GFS files:
NOMADS-NOAA Operational Model Archive and Distribution System
Use the "grib filter" interface to select values and geographical regions.
Dave

And here some other Grib file for Currents. With two folder : Grib 1 and Grib 2.

ftp://ftp.bsh.de/Stroemungsvorhersagen/

In the Grib1 folder, there is Grib files, for a day at a time, for reasons of 3 days, which are extracted from GRIB2 files from the other folder.

There is in each folder, a "txt" file with some explanations.

Note that these Grib1 are, actually, well displayed by the plugin Grib, V 3.0 of OpenCPN. So, it is possible to make test for comparison of display.

These Grib cover, in Europe sea : Channel, North Sea, Baltic area.

[Rasbats]

Great to see the progress for Grib2 data.

Using CDO (Climate Data Objects) it is possible to translate NetCDF to Grib1/2. However, I have not been able to read the Grib 1 files made by this program, unless I use my own code (grib_api). u and v variable names for current values in netcdf translate to var1 and var2 in the Grib, which are not recognised by the present Grib plugin.

Could the new Grib2 plugin accomodate translations made by CDO? If so, it would be possible to make a small converter program to prepare files for OpenCPN.

An example of one translation is attached.

Mike

[Gilletarom]

Hello rasbats,

Originally Posted by Rasbats

Great to see the progress for Grib2 data.
...
An example of one translation is attached.
Mike

After put out ".pdf", It is impossible to display it with Grib (V3) and of course whith "Grib2 0.1". it is empty !

[did-g]

Originally Posted by Rasbats

Great to see the progress for Grib2 data.

Using CDO (Climate Data Objects) it is possible to translate NetCDF to Grib1/2. However, I have not been able to read the Grib 1 files made by this program, unless I use my own code (grib_api). u and v variable names for current values in netcdf translate to var1 and var2 in the Grib, which are not recognised by the present Grib plugin.

Could the new Grib2 plugin accomodate translations made by CDO? If so, it would be possible to make a small converter program to prepare files for OpenCPN.

An example of one translation is attached.

Mike

In my understanding the bug is in cdo but you can convert them, I'm using:

# convert netcf
cdo -f grb2 copy IBI.nc IBI.grb
# split by type
grib_copy IBI.grb 'IBI_[shortName].grib'

# fixup
grib_set -s discipline=10,parameterCategory=1,dataTime=30,typeOfProcessedData=1 IBI_u.grib IBI_ucurr.grb

grib_set -s discipline=10,parameterCategory=1,dataTime=30,typeOfProcessedData=1 IBI_v.grib IBI_vcurr.grb

I think you can convert them in grib1 too with grib_set -s editionNumber=1
may or may not work...

URLs in my previous post are still active, IBI current, SHOM current and wave, Meteo France Arome and Arpege.

[rgleason]

Thanks to bdbcat, mike, bcn and gille and Did-g. This promises to be a rewarding area for improvements. I look forward to helping.

[bdbcat]

Testers...

As mentioned, I have uploaded a Windows version of OCPN 4.2.1 with the integrated GRIB2 compatible PlugIn.

Here it is:

http://opencpn.navnux.org/beta/opencpn_4.2.1_setup.exe

Enjoy!
Dave

[seandepagnier]

Originally Posted by bdbcat

Rick...

There are many proposed alternatives to simple GRIB1/2 for distribution of meteo data. Some are very interesting and workable now, some are unimplemented proposals.

I would like to propose a format which stores the wind data in polar form instead of vector, and uses a logarithmic scale for wind speed. This means using 2 bytes (one for speed one for direction) for a very accurate measurement, or possibly fewer bits. The data could then be deinterlaced to give better compression results.

I think this file would be a lot smaller than grib with identical useful accuracy.

[bcn]

Sean...

are you aware of the CDO tool set mentioned by did-g ?

https://code.zmaw.de/projects/cdo/wiki

And have a safe trip!

Hubert