Use of RTOFS for Current.Gribs - Gulf of Maine

[estarzinger]

^^ you do know that you can get multi-day rtofs file downloads by using Jim Corenman's very easy to use graphic interface called ViewFax (despite the name it does grib downloads not faxes)? You dont need any scripts. It comes bundled in with the airmail program if you already have that, or you can download seperately (get the beta version) free.

[.Paul.]

Originally Posted by rgleason

Actually I think XXX defines the max hours wanted so the file name would be
rtofs_glo_2ds_f144_3hrly_prog.nc

Just need surface currents and a lat/long box next.

Now going to
ftp://ftpprd.ncep.noaa.gov/pub/data/...tofs.20170524/

I see that these files are huge. More later.

Rick,

I think the "rtofs_glo_2ds_fxxx_3hrly_prog.nc" files have the surface current data you want, but that FTP server (in the quote box above) only has these files with forecasts out to 96 hours (rtofs_glo_2ds_f096_3hrly_prog.nc). If you use the NOMADS page to access the "RTOFS Global" data by clicking on the "HTTP" box, you can get the forecasts out to 192 hours (rtofs_glo_2ds_f192_3hrly_prog.nc).

If you click on the "HTTP" column heading (at the top of the matrix) you are taken to instructions for getting partial downloads of the GRIB data, but there are no instructions for downloading a regional subset (latitude and longitude limits). I think you found this, but I mention it to help someone else. I have not tried this.

If you click on the "gds-alt" column heading (top of matrix) you get instructions for downloading by data name and for a region bounded by latitudes and longitudes. If you go this route, you access the files by clicking on the "OpenDAP-alt" box to the right of "RTOFS Global". I have not tried this.

Paul

[rgleason]

Evans, Thank you. I did not know this. Earlier I was going to try that and got sidetracked.

you do know that you can get multi-day rtofs file downloads by using Jim Corenman's very easy to use graphic interface called ViewFax (despite the name it does grib downloads not faxes)? You dont need any scripts. It comes bundled in with the airmail program if you already have that, or you can download seperately (get the beta version) free

Paul.
I will give that location a try. Also perhaps use of Viewfax will shed some light on the sources.

I've also been looking into PredictWind 3month Standard and Professional for the purpose of receiving 3hr wind and current data via Iridium Go, and it appears it would work fine and give the Skipper's Mac a routing using one polar file which would be great, but they do not yet cover the Gulf of Maine for "currrents". Tech advises it will be 1-2 months. Tech Support has been very helpful though.

Any suggestions are welcome.

I note here again that:
1. Opencpn Grib_pi Downloads using zygrib username and password is no longer available, so this download feature appears to be broken and should be fixed.
2. Opencpn Grib_pi Downloads using Saildocs with Rtofs request results in 24 hr gribs even when 12 hr is requested. Additionally 3hr Rrtof current gribs are available so that should work too, but Saildocs grib server has been affected by the changes at NOAA to using the RTOFS Global Model, thus breaking the Saildocs Rtofs download.
3. Since Grib_pi has been updated to grib2 and since we can load two grib files now, it would be truely wonderful to update this handy and essential plugin for improved downloading.
4. I have found the "walking or moving grib" feature invaluable for reducing the size data downloads and hope that at some point download of Rtofs currents would work in the same way with 3hrr resolution.

-PS I will post on Flyspray and Github.

On to try ViewFax!

[rgleason]

Dear Evans,

Things have changed. I've tested Viewfax and it is downloading only 24hr RTOFS data now, even though the interface allows 12hr selection.

This is the same situation with the OpenCPN Grib_pi Download because Grib_pi is also using Saildocs. The change that has caused this is NOAA's change in servers.

In short Saildocs needs to revise the RTOFS download source to the new servers, and while doing so, perhaps they can provide 3hr interval because these are available now.

So the fix is becoming clearer now. Can someone advise Saildocs? Or is there someone who is involved who can get this change made?

[rgleason]

This is a bit like groping around in the dark, but Paul's and Evans help is much appreciated.

I think I found the necessary data on the NOAA Global RTOFS Product>Data Access server:
ftp://ftpprd.ncep.noaa.gov/pub/data/...tofs.20170525/

down towards the bottom:

rtofs_glo_2ds_f192_3hrly_prog.nc
405842 KB 5/25/17 4:04:00 PM
This file needs to have a lat/long area and only surface current data extracted. There must be a method to do this.
---

Using Paul's suggestions:
HTTP
1. Use Nomads: NOMADS-NOAA Operational Model Archive and Distribution System
2. Click on "http" leads to instructions for partial downloads.
"Using this procedure, you can download just the data you need from a remote GRIB2 file." Fast Downloading of GRIB2 Files The "Details" paragraph indicates that several linux programs are needed. "The wgrib inventory is used for the index file, and cURL is used for the random-access http program. " and two perl scripts "Two perl scripts are also required, get_inf.pl and get_grib.pl, which are downloadable from the NCEP CPC website (see "Requirements" below)." There are some examples as well. It appears this would be a linux server function.

1. perl
2. grep
3. cURL
4. grib files and their wgrib inventory on an http server
5. get_inv.pl
6. get_grib.pl

There are several examples of Windows users who have configured Perl, curl and wgrib so they can download partial files. I don't know if I will try this yet I am using Perl scripts now.

At the bottom there is a note about Regional Subsetting
"Right now, I am happy with the g2subset software that is running on the nomads servers. "

The other alternative GDS-ALT

1. Use Nomads: NOMADS-NOAA Operational Model Archive and Distribution System
2. Paul suggests click on the "gds-alt" column heading (top of matrix) you get instructions for downloading by data name and for a region bounded by latitudes and longitudes.

I think I'll try this first. __I guess not. Configuring a Grads Data Server -DODS is well beyond my capability.

DODS
"

The GrADS-DODS server is a stable, secure, easy to install, and easy to configure data server that provides subsetting and analysis services across a network.

The subsetting capability allows a user to request a specified range of time and space from a large dataset, eliminating the need to download everything simply to access a small relevant portion of a dataset."

I think this is the GrADS data format needed...
http://nomads.ncep.noaa.gov:9090/dod...aily_prog.info

u_velocity: Grid_FillValue: 1.26765E30
missing_value: 1.26765E30
long_name: "eastward_sea_water_velocity (m/s) "

v_velocity: Grid_FillValue: 1.26765E30
missing_value: 1.26765E30
long_name: "northward_sea_water_velocity (m/s) "

But I don't see where the data is located and am not connecting the dots....

[rgleason]

I have to say, that I've reviewed this for windows and looked at wgrib and the notes, and I could try to set this up sometime but there must be an easier way!

I wish svsarana website were reconfigured for the new NOAA setup. That would be very useful, but that website also does only 24hr RTOFS current now too.

[rgleason]

Check out https://www.gribfiles.com/#/Weather-models
RTOFS TAB They are using HYCOM & offer what we need:

RTOFS Type of model Regional Ocean Forecast System, HYCOM Geographic area Inside the red box at the frontpage map Model grid (horizontal, km)
Grib file grid (horizontal, deg) 0,083 x 0,083 Timespan (hours) 144 Time step (hours) 3 Available parameters Surface current
Initialization time (UTC) 00:00 Available from server (app, UTC) 17:00 Source of data NCEP - NOAA Remarks
Use for Give a brief overview of the ocean system
Do not use for Tidal streams and ocean currents for coastal areas Comments

[rgleason]

NOAA notice: Rtofs Atlantic ended, superceded by Global Real Time Forecasting RTOFS Global.

NOTICE: Effective on or about March 14, 2017, the National Centers for Environmental Prediction (NCEP) will discontinue the North Atlantic Basin Real Time Ocean Forecasting System (RTOFS Atlantic). This system has been superseded by the Global Real Time Ocean Forecasting System (RTOFS Global), which became operational October 2011. More information is available here.

[rgleason]

I should have gone to the notice of termination earlier.
http://www.nws.noaa.gov/os/notificat...fs_removal.htm
which suggests this page for replacement of Atlantic Rtofs
WAVE Products

"RTOFS HYCOM - Global 2 Dimensional Surface Prognostic- 1/12 Degree Resolution Forecast - 3 hrly
rtofs_glo_2ds_fxxx_3hrly_prog.nc FH00-192

For example, todays version found in this directory:
http://www.ftp.ncep.noaa.gov/data/nc...tofs.20170525/

Now how to extract the lat-lon area and just the surface currents?

[rgleason]

Now that I've downloaded 410mb, I see that this file is a netcd file which ends with .nc and it is probably not compatible with Grib_pi.

However Mike Rossiter was working on a Netcdf current reader earlier, maybe that will work, but it won't work with WRouting....
NCDF_pi works on .nc files but chokes on this 410mb file.

This isn't going to solve my problem.

[rgleason]

Has anyone tried partial HTTP downloads from NECP Nomads with an OS = Windows using:

Requirements

1. get_gfs.pl.
2. perl
3. cURL

https://curl.haxx.se/download.html

Refer to

Fast Downloading of GRIB2 Files
Fast Downloading of GRIB Files, Part 2

NOMADS-NOAA Operational Model Archive and Distribution System
These operations would be run on files similar to:
Index of /pub/data/nccf/com/rtofs/prod/rtofs.20170525

[rgleason]

In a nutshell, here is what happened at Noaa.
Atlantic Real-Time Ocean Forecast System (Discontinued) Notice effectiv March 14.

Follow the link to Global RTOFS Notification http://www.nws.noaa.gov/os/notificat...fs_removal.htm

More details about the suite of products from the Global RTOFS
that replaced the legacy RTOFS Atlantic are available at
WAVE Products

See rtofs_glo_2ds_nxxx_3hrly_prog.nc

See this note:
The Inventory links contain detailed model information
Use as a guide when selecting specific parameters from NOMADS through the grib filter option

Which leads to the NOMADS Grib Filter page
NOMADS-NOAA Operational Model Archive and Distribution System

Down the list under "Ocean Models"

RTOFS Atlantic daily grib filter http OpenDAP-alt

RTOFS Atlantic HIRES daily grib filter http OpenDAP-alt

RTOFS Global daily - http OpenDAP-alt

Rtofs Global has no "Grib Filter" and the RTOFS Atlantic are now cancelled.
I hope this changes soon!

Using the Grib Filter this is a typical URL=
URL=
http://nomads.ncep.noaa.gov/cgi-bin/...2Fofs.20170321

I wonder if it can be used & modified for the new global rtofs?

GrADS Data Server - info for /rtofs/rtofs_global20170528/rtofs_glo_2ds_forecast_3hrly_prog

and then adding something like:
&lev_0_m_below_sea_level=on&var_UOGRD=on&var_VOGRD =on&subregion=&leftlon=-71&rightlon=-68&toplat=44&bottomlat=41&dir=%2Fofs.20170321
from the Grib Filter above? -- but I don't think this is the correct server and I haven't got the correct filter defined..

Perhaps we could write noaa.. the emails are right there.

Alternatively, does anyone know how to extract data from a grads server?

[Moo]

Originally Posted by rgleason

So the fix is becoming clearer now. Can someone advise Saildocs? Or is there someone who is involved who can get this change made?

Can't you advise Jim Corenman of your findings? His contact is in Airmail. He is very helpful.

Cheers

[rgleason]

Thanks, Have sent an email to airmail3 #at airmail2000 #dot com

[Moo]

Originally Posted by rgleason

Thanks, Have sent an email to airmail3 #at airmail2000 #dot com

Have sent you a PM Rick.