Automatic grib downloading

[conachair]

This took a little while instead of doing more deserving items on the jobs list - but it works!

Briefly, the wonderful node-red is available online >
https://fred.sensetecnic.com/

Sort of swiss army knife fr data - take any input you can think of, fiddle with it then send it to anywhere you can think of. Pretty much. So now every 6 hours it sends an email to saildocs, gts the attachment when it's sent back and saves it into dropbox, once with data/time filename and once call 'current. Once the opencpn grib plugin opens the file called 'current', it does so every time you start the program. Always the latest grib automatically. Assuming you can get online to update dropbox anyway, but then if you can't get online downloading a grib ain't going to happen anyway



FRED is all online so will keep running all the time, though the free version needs you to log on once a day to keep it running.

Quick walk through - top left the Inject node is just trigger which sends something once every 6 hours, or every time you press the button on the left. To the right of that the create message function node creates the email text and sets some other things, it will run anytime something comes in to its input on the left from the inject node so once every 6 hours. Then it sends all that to the email node which sends off the email.

Then further down, bottom right is another email node - once a minute it checks for the latest emil and downloads it. THis the goes to a switch node which is set up to just let through emails from query-reply@saildocs.com. Next is a change node, this swaps the attachment around into a place where the drop box node expects to see it. Then there's another function node which renames the file with current date//time & path in dropbox the off it goes to the dropbox node to get uploaded. As well as this archived file, the flow splits off , waits for a few seconds then gets renamed as Current and uploaded. The msg nodes are just for debugging so you cn see what's going on to help figuring out why it doesn't work.
That's that

Node red code below, if you copy that, open up FRED then press CTRL I you can paste the code straight in. Probably won't work straight away, you might need to install the dropbox node which is easy, button on the control panel on the left.
So constant latest gribs plus saved past gribs .

[
    {
        "id": "17d3d13f.30215f",
        "type": "e-mail",
        "z": "c676bef7.1661e",
        "server": "smtp.gmail.com",
        "port": "465",
        "secure": true,
        "name": "",
        "dname": "send email",
        "x": 650,
        "y": 80,
        "wires": []
    },
    {
        "id": "5802db4f.e61384",
        "type": "e-mail in",
        "z": "c676bef7.1661e",
        "name": "get email",
        "protocol": "IMAP",
        "server": "imap.gmail.com",
        "useSSL": true,
        "port": "993",
        "box": "INBOX",
        "disposition": "Read",
        "repeat": "60",
        "x": 100,
        "y": 440,
        "wires": [
            [
                "711358ba.286ca8"
            ]
        ]
    },
    {
        "id": "bb666145.e08df",
        "type": "dropbox out",
        "z": "c676bef7.1661e",
        "dropbox": "",
        "filename": "",
        "localFilename": "",
        "name": "upload",
        "x": 550,
        "y": 260,
        "wires": []
    },
    {
        "id": "f95cb5a5.e021c8",
        "type": "debug",
        "z": "c676bef7.1661e",
        "name": "",
        "active": true,
        "console": "false",
        "complete": "true",
        "x": 670,
        "y": 300,
        "wires": []
    },
    {
        "id": "2b8cc36a.a0d37c",
        "type": "inject",
        "z": "c676bef7.1661e",
        "name": "",
        "topic": "saildocs",
        "payload": "send GFS:38N,25N,19W,1W|1.0,1.0|0,6,12|WIND,PRESS,APCP,TCDC,GUST",
        "payloadType": "str",
        "repeat": "21600",
        "crontab": "",
        "once": false,
        "x": 150,
        "y": 100,
        "wires": [
            [
                "f7c7a174.2cf3d"
            ]
        ]
    },
    {
        "id": "9519e8a0.2e7418",
        "type": "change",
        "z": "c676bef7.1661e",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "attachments.content",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 280,
        "y": 380,
        "wires": [
            [
                "11fb456.00987bb",
                "8d39fd28.bfb1a"
            ]
        ]
    },
    {
        "id": "79e9a194.8bfcc",
        "type": "debug",
        "z": "c676bef7.1661e",
        "name": "",
        "active": true,
        "console": "false",
        "complete": "true",
        "x": 610,
        "y": 140,
        "wires": []
    },
    {
        "id": "f7c7a174.2cf3d",
        "type": "function",
        "z": "c676bef7.1661e",
        "name": "Create message ",
        "func": "msg.payload = \"send GFS:48N,25N,29W,1W|1.0,1.0|0,6,12|WIND,PRESS,APCP,TCDC,GUST\\n\";\nmsg.to='xxxxx@saildocs.com';\nmsg.from=\"xxxx@gmail.com\"; //needed to set return-path in the header, otherwise that filed is ampty and saildocs will ignore the email\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 390,
        "y": 80,
        "wires": [
            [
                "17d3d13f.30215f",
                "79e9a194.8bfcc"
            ]
        ]
    },
    {
        "id": "711358ba.286ca8",
        "type": "switch",
        "z": "c676bef7.1661e",
        "name": "Filter",
        "property": "from",
        "propertyType": "msg",
        "rules": [
            {
                "t": "eq",
                "v": "query-reply@saildocs.com",
                "vt": "str"
            }
        ],
        "checkall": "true",
        "outputs": 1,
        "x": 110,
        "y": 380,
        "wires": [
            [
                "9519e8a0.2e7418"
            ]
        ]
    },
    {
        "id": "11fb456.00987bb",
        "type": "debug",
        "z": "c676bef7.1661e",
        "name": "",
        "active": false,
        "console": "false",
        "complete": "true",
        "x": 350,
        "y": 460,
        "wires": []
    },
    {
        "id": "8d39fd28.bfb1a",
        "type": "function",
        "z": "c676bef7.1661e",
        "name": "Add date to filename",
        "func": "var today = new Date();\n\ntoday.setDate(today.getDate() + 1);\n\nvar hour = today.getHours();\nvar day = today.getDate();\nvar month = today.getMonth()+1;\nvar year = today.getFullYear();\nvar folderdate = year +\"-\" + month;\n\n//node.warn(\"folderdate=\" + folderdate);\n//\nvar filedate= hour + '--' + day +'-' + month + '-' + year;\nnode.warn(\"filedate=\" + filedate);\n\n\nmsg.filename=\"Gribs/\" + folderdate +'/'+ filedate + \".grb\";\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 160,
        "y": 300,
        "wires": [
            [
                "bb666145.e08df",
                "f95cb5a5.e021c8",
                "d1f2f6bf.ad33f8"
            ]
        ]
    },
    {
        "id": "80c80a1f.bf06c8",
        "type": "function",
        "z": "c676bef7.1661e",
        "name": "Change filename to current",
        "func": "msg.filename=\"Gribs/\" + \"current.grb\";\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 340,
        "y": 220,
        "wires": [
            [
                "bb666145.e08df"
            ]
        ]
    },
    {
        "id": "d1f2f6bf.ad33f8",
        "type": "delay",
        "z": "c676bef7.1661e",
        "name": "",
        "pauseType": "delay",
        "timeout": "5",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "1",
        "rateUnits": "second",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": false,
        "x": 120,
        "y": 220,
        "wires": [
            [
                "80c80a1f.bf06c8"
            ]
        ]
    }
]

[Baikal]

Through Microsoft Flow is easier and free

[BlackSea]

Under Linux with Shell and Crontab much easier

[conachair]

Originally Posted by Baikal

Through Microsoft Flow is easier and free

Not heard of that one - thanks for mentioning it, can you rename the files with today's date, overwrite files & create folders on dropbox? Quick look at it, seems quite simple but not as powerful as node red.
Can you download web pages? Might be handy for synoptics.


Could you share a flow downloading renamed grib files as an example?

[conachair]

Originally Posted by BlackSea

Under Linux with Shell and Crontab much easier

Constantly running online when your computer is turned off?
Maybe you could share a script for others?

[Baikal]

Originally Posted by conachair

Not heard of that one - thanks for mentioning it, can you rename the files with today's date, overwrite files & create folders on dropbox? Quick look at it, seems quite simple but not as powerful as node red.
Can you download web pages? Might be handy for synoptics.


Could you share a flow downloading renamed grib files as an example?

I subscribed to receive grib files to outlook mail
on a ready-made template Flows configured Saving the mail attachment to the dropbox in the specified folder.
The filename grib is the same as it comes to the mail
Next, the task script removes old grib files from the folder

https://flow.microsoft.com

[BlackSea]

Originally Posted by conachair

Constantly running online when your computer is turned off?
Maybe you could share a script for others?

Runs your NodeRed if your Computer turned off?

[conachair]

Originally Posted by Baikal

I subscribed to receive grib files to outlook mail
on a ready-made template Flows configured Saving the mail attachment to the dropbox in the specified folder.
The filename grib is the same as it comes to the mail
Next, the task script removes old grib files from the folder

https://flow.microsoft.com

If you can rename the file then opencpn will always open the latest grib automatically. I have unlimited google drive space so hopefully will use IFTTT to move the gribs across to there, just cos I can Gets saved in monthly folders.

I forgot you could subscribe to saildocs! Would you need 4 subscriptions? From the emails it looks like each new grib gets posted at about 05:00/11.00/17.00/23.00 UTC.

[conachair]

Originally Posted by BlackSea

Runs your NodeRed if your Computer turned off?

Flow above is running online on FRED, no need for your computer to be turned on. Then when you boot up dropbox updates & the latest grib is downloaded.

[BlackSea]

Originally Posted by BlackSea

Runs your NodeRed if your Computer turned off?

Oh, sorry, I have not seen, that you use external Node instance somewhere in cloud.
Nevertheless, you can realize something easier without nodered in cloud

[Baikal]

Originally Posted by conachair

I forgot you could subscribe to saildocs! Would you need 4 subscriptions?

I have two subscriptions for different locations
for each his Flows, he searches for the name of the subject of the letter where the coordinates are indicated.

[conachair]

Originally Posted by BlackSea

Oh, sorry, I have not seen, that you use external Node instance somewhere in cloud.
Nevertheless, you can realize something easier without nodered in cloud

Apparently so with microsoft flows, though a quick look didn't make it obvious how easy it is, if at all, to rename attachments in different dropbox locations. Which is sort of the whole point, opencpn always opens the latest grib from dropbox.
Downside of node red online is you need to relog on every day fro the free version. I use it to download live/forecast weather data into a database from openweatheronline as well so don't mind paying a few pence a day, slowly moving everything across from the boat computer online
.
I'll leave this dashboard public for a while - does this link work?
https://paddyb.fred.sensetecnic.com/api/ui/#/0
(Edit - just added graphed live pressure data to that page, took about 2 minutes to set up)


Works well for off site data viewing as well, run a raspberry pi onboard sending data out with MQTT - then view boat data graphed online.

Seems to be easier ways to save an email online but haven't come across anything as powerful as node-red yet. Amazing free program.

[Baikal]

opencpn opens the last file by the date the file was created and not by the date in its name!
You can call it anything but it will open the one that was the last by date of the file metadata

[conachair]

Originally Posted by Baikal

opencpn opens the last file by the date the file was created and not by the date in its name!
You can call it anything but it will open the one that was the last by date of the file metadata

So it does! Didn't know that - thanks.

So with that if just downloading a grib a day then something like microsoft flow would probably be easier.