Climatology

[seandepagnier]

I have been hinting about this plugin for some time, and finally have it at a point where I think it could be useful.

I need some help with a few things:

1. The data files are quite large. I need somewhere to host them.

All the cyclone tracks are around 1.5 megs (compressed), and ocean current data will be around 3.5 megs. I need someone to generate the current data as well, but it is only a few gigabytes of data to process (into 3.5 megs)

Sea level pressure and temperature around 2 megs. So this is a total of 7 megs for everything besides wind atlas data (which is 10 to 50 megs depending on resolution)

For this reason I cannot distribute these files on this forum and I cannot put them in git (because it would take forever for people to clone especially when I delete some later and add new ones since git will clone the past history)

2. I cannot process all of the data and need someone who has a better internet connection to do this.

I generated a single map for january for the 1/4th degree resolution wind map resulting in a file 4 megabytes. If you had 10 years, the resulting file would still be the same size, it would just average the data over many years. I generated this from about 200 megs of raw data, and this is only for january of the year 2000, so 30 years worth is many gigs. I need someone who is willing to download all the data for 1980-2010 and compile wind atlases for each month, as my internet connection cannot deal with this (72 gigs of raw data) to produce a 48 meg database for other users to download. If you can only do a decade at first, that would be useful. It would also be possible to use the 6 hour data instead of daily which might give slightly better results but you would need to download much more data.

I don't think the programs that I wrote to compile the data will run on windows, so ideally do this in linux, but they should be fairly easy to port and make work if needed.

This is not including separate atlases for el-nino/neutral/la-nina which would require 3x the space. I will get to that later. I would like to offer 1/2 degree and 1 degree scales since these resolutions are still fairly useful and are a smaller download for end users as alternate options.

There are endless possibilities for extensions, and most of them require huge databases to implement. My next step after general improvements is to integrate with weather routing.

https://github.com/seandepagnier/climatology_pi

You can compile and run without any data files, but it won't be of much use (except feedback) Instructions for obtaining data are in the readme files

[cagney]

To build on Ubuntu I needed to install "libnetcdf-dev".
I still have this log error when trying to load the plugin:


Error: /usr/local/lib/opencpn/libclimatology_pi.so: undefined symbol: BZ2_bzReadClose

Thomas

[nohal]

Thomas...
Try with the following patch cannibalized from grib_pi... I did it without looking at the code at all and it will sure not work on Windows until src/bzip2 is copied form the grib plugin...

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 84eb643..3ea8ad8 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -94,6 +94,27 @@ IF(WIN32)
   ADD_DEPENDENCIES(${PACKAGE_NAME} ${PARENT})
 ENDIF(WIN32)
 
+IF(NOT UNIX)
+    SET(SRC_BZIP
+            src/bzip2/bzlib.c 
+            src/bzip2/blocksort.c 
+            src/bzip2/compress.c 
+            src/bzip2/crctable.c 
+            src/bzip2/decompress.c 
+            src/bzip2/huffman.c 
+            src/bzip2/randtable.c
+    )
+    ADD_LIBRARY(LIB_BZIP ${SRC_BZIP})
+
+    INCLUDE_DIRECTORIES(${PLUGIN_SOURCE_DIR}/src/bzip2)
+ENDIF(NOT UNIX)
+
+IF(UNIX AND NOT APPLE)
+    FIND_PACKAGE(BZip2 REQUIRED)
+    INCLUDE_DIRECTORIES(${BZIP2_INCLUDE_DIR})
+    TARGET_LINK_LIBRARIES( ${PACKAGE_NAME} ${BZIP2_LIBRARIES} )
+ENDIF(UNIX AND NOT APPLE)
+
 INCLUDE( FindNetCDF.cmake )
 
 #FIND_PACKAGE(netcdf REQUIRED)

Pavel

[seandepagnier]

Originally Posted by cagney

To build on Ubuntu I needed to install "libnetcdf-dev".
I still have this log error when trying to load the plugin:


Error: /usr/local/lib/opencpn/libclimatology_pi.so: undefined symbol: BZ2_bzReadClose

Thomas

Hmm... I thought libbzip was already linked.. strange that it works on my system and not yours. Anyway some tweaking to the cmakelists is likely needed here.

Currently libnetcdf-dev is a dependency of the plugin, but maybe I should make extractor programs for this data so we can host raw compressed data which would be smaller and also avoid the dependency. I'm sure it will be an issue on windows, and the libcdf stuff is a lot of extra code to inject into the tree somewhere.

Pavel,
I would like to see libbz2 libz and the json stuff all part of the main program and exported so any plugin can use them. We could just export the zuFile and json stuff.

In the future, more and more things will be needed. geomag routines would be nice (I made it thread safe in the weather routing version) and the list goes on.. I think it will simplify the build files, and reduce building time as well as avoid duplicate code. What do you think?

[nohal]

Originally Posted by boat_alexandra

I would like to see libbz2 libz and the json stuff all part of the main program and exported so any plugin can use them. We could just export the zuFile and json stuff.

In the future, more and more things will be needed. geomag routines would be nice (I made it thread safe in the weather routing version) and the list goes on.. I think it will simplify the build files, and reduce building time as well as avoid duplicate code. What do you think?

Me too
But taken the resources available, this is kind of "you want it, you make it" task... The idea always was to make the plugin API as small as possible and the plugins as self-contained as possible. With some obvious downsides it brings. I tried to take care of part of it introducing the plugin messaging API, but of course it's kind of useless when you need high performance...
My personal opinion is that it's good to export as much core services as we can to the plugins in a form of an API as high-level as possible. But at the other hand I don't have much time to invest to it. At the end, the final design decision is and will be Dave's, so propose stuff to be discussed and we will see. And as we know, if the proposal is backed by a patch, it's chances increase

Pavel

[seandepagnier]

more screenshots with vector display

There is both scalar and vector averaging for wind data. Could have monthly vector averages for currents too but I think this is probably less useful?

[seandepagnier]

_____

[Gilletarom]

Hello,

Boat-alexandra tells me the new Climayolgy plugin. I found the pot file.

But it seems to me that there is still no dll.

Without dll file, we do not immediately suggest a po file because we need to test our work before putting it online.

Gilletarom.

[rgleason]

Sean's questions about Climatology -my take. Post 170 Weather_routing

[rgleason]

Sean's questions about Climatology -my take. Post 170 Weather_routing

Don't know how I missed this hot thread. Sean it's awesome, I don't know how you find the time and have the talent, but I would like to help.

I have good internet service, an old Windows machine I could devote, and possibly a dedicated IP address which with apache server download might provide a download location.

Don't know the processor, but its running WinXP right now. How much trouble/time to load Linux, (which version and flavor) and get it running with your averaging programs? Could just leave it running 24/7 , after downloading the data in chunks? Later make it a server? ( Probably getting too ambitious here. Maybe someone else will be able to help too.)

Compressed scat data Summary from Sean

Cyclone tracks = 1.5 megs
Ocean current = 3.5 megs.
Sea level pressure & temp = 2 megs.
Wind data = 10-50 megs depends on resolution
Total of = 17-57 megs for everything
Need a place to host the files.

2000 January Wind 1/4 degree resolution
Download=200mb/yr Compressed=4mb (10 yrs will be same)

Need 1980-2010 = 30 years compile wind atlases for each month.
Internet Download 30 yrs daily data = 72 gigs of raw data ---> 48 mb
Could do 10 yrs with 6 hr data instead of daily data, will result in slightly better results but download much more data.

In addition the other data.

[seandepagnier]

Should be able to host datafiles on sourceforge.

As for the data processing, pavel said he would download the files.. but it could be a while. Maybe you could grab all the 6-hourly files, which should be better than the daily ones. Do you have about 200 gigs of storage you can spare for this endeavor indefinitely?

[rgleason]

Sean, I don't have 200gb of storage right now.. but might if I have a chance to figure this out.

Where do I download from?.... + your experience?
What Linux version do you reccomend? (after I get the the files.)

[rgleason]

I agree we need to get the data pronto, to protect the development.

[rgleason]

Thanks Sean

I began to realize that the Climatology averages might work better if they are available for download for 2days, weekly, monthly because the shorter the trip the more detail needed? For example, on a long trip of a month or 1.5 month, the monthly might work fine, but is it possible that weekly averages would be better?
Perhaps you can advise about that?

Thinking about algorithm.

Current position, single step to next average position:

1. Current Position -Wind Atlas (8 vectors + % time)
2. Calc 8 ship positions as vectors using boat polars
3. Calc 8 vectors X %Frequency for each vector
4. Sum the vectors to get a single resultant vector from current position to the next averaged position.

Variability, expression and representation:.

1. The 8 calculate ship positions in #2 above define a field which represents the "variability field for current step".
2. There will be similar variability field for the next step, and the next step..
3. Use those fields additively with user adjustable factors, to define a broad path following the optimal average path created (sort of like a Hurrican Cone that Noaa does, close in is narrow, and as unknowns increase it broadens).
4. Within the broad path occasionally (user adjustable, and a sensitivity variable. also how frequently it should be represented-days? ) show curved shaded bands which represent the range of variability in advancement of position. (I hope that is clear, but I could try to draw it)

I think discussion helps... even though it seems like a can of worms.

[rgleason]

Sean, I know you think what I am suggesting is not really viable, and I know you are way ahead of the curve on this compared to me, being the math whiz that you are, but I thought sketch might help a little with understanding.... so here goes. At the very least you'll get a chuckle.