Open Chart Standard

[dacust]

I am starting a new thread as it seems I was a little premature in my enthusiasm over the idea of a Marine GeoTIFF. Here is the opinion of an expert.

Hi Dan,

Just answered a similar mail from Jonas, but as it is in Swedish a cc will
not help much

In short what i said is I don't think geotiff is a good format for this
purpose. It is a complex, requires reading of the entire file, and rather
slow reading.
The strength with BSB (&WCI) is that only what is needed can be accessed
quickly.

An open format should be open from start to avoid any person/organisation be
held responsible, as there are a large number of file-format patents.

The format should handle partial reading, handle very large bitmaps, line or
block oriented with pointers for fast read access, easy access cal-header
for cataloging. I have many more ideas, and I know the mistakes made in the
WCI format

If you decide to create a new format I am glad to help. I already set up an
account on the OpenCPN forum. However my time is rather limited.


/Olle

Myself, I am VERY interested in whatever help Olle can give us.

-dan

[manimaul]

Dan,

I just posted my vote for KML/GML in the other forum. After reading Ollie's reply, I'll mention something pertinent about KML. KML raster tiles are 256px by 256px, which in my mind would fix the problem of slow/partial access to the parts of the file needed. As you know, it is an open format that is gaining support in many programs. It fits the bill.

Very basic anatomy of a KMZ raster chart file:
A kmz file is zip file containing an xml file with georeferencing bits
and directories containing 256 pixel by 256 pixel PNG files that stitch together.

I'm not sure if you've had the opportunity to play around with MaxSea Timezero. If you do you will notice that they are using map tiles similar to kml as well... Interesting...

I will post an example chart I've converted from BSB to KML shortly.

Will

[dacust]

Manimaul,

I never took KML seriously. I thought of it as a Google thing, and didn't realize what it really was. What I am embarrassed about is I just went on that impression and never actually checked it out.

I just now googled "KML standards" and see it is much more than I thought it was. Maybe it really is what we need.

KML Standards

-dan

[sidetrack]

Has anyone taken a look at Mapcruncher? It seems to produce tiled maps, which might fit the bill for a map in which you don't have to read the whole file.

[sinbad7]

Dan..

I don't know if it's any help but I have one of the Russian GeoTIFF maps from an area near Gibraltar if you need something to experiment with. Had a look at the header which looks pretty crazy,many sporadic characters and then several unreadable blocks before the main chart itself. Size around 6 MB. Let me know if you need it.

[idpnd]

Any open standard should be

- truly open/free (no legal/patent threats)
- widely in use already (so one isn't developing some sort of lonely cul-de-sac)
- supported by major industry forces/dynamics (such as google)
- technically viable (efficient at what it does, i.e. what geotiff apparently isn't)

KML was first championed by rhoel_asia who purports to be a GIS expert but has now gone a bit quiet (hi rhoel!).

Another advantage would be compatibility with the google earth universe of overlays and aerial pics..

Wonder what Olle would have to say about KML as an option? Any further alternatives?

[Rhoel_Asia]

Originally Posted by idpnd

KML was first championed by rhoel_asia who purports to be a GIS expert but has now gone a bit quiet (hi rhoel!).

He is still lurking, just a little busy as his Japanese boss is in Cambodia looking at the WebGIS he built. Fortunately, both are happy :-). he should have more time after a government viewing meeting on the 10th.

I am still working on the KML thing, recently completing a double blind test - I traced a google earth island, exported it as multi-levels of KML, converted it to shape files (with FME) and imported into ArcGIS 9.3. Did some renaming and colour stuff, then output the map file as a KMZ. This was then loaded back into GE and wayhey, it arrived bang on where it started. The file is available for download here. I deliberately chose an island out of the normal cruising grounds, just in case anyone was fool enough to use it for real - IT'S A TEST!. I have no depth information on the fringing reef - its only to get you to visual range of the island, not an autopilot plot to the landing stage; don't blame me if you scrape the paintwork on the coral.

My current thought is KML in OpenCPN will probably start as a chart-updating thing ... to mark simple Notice to Mariners information about lights being moved etc. You really do not want to tamper with the original ENC data - having an overlay is perfect. Later the tools might develop to make small boat chartlet/pilotage: There are big hurdles to jump, like building a light vector generator. If anyone has a really complex light arc on a chart, please send me a screen shot of it.

There are several possibilities floating around on how to approach this, and its good there is a dedicated thread to chart development> For now, I need to get the Japanese home and update my machine/s to Win7 ... this will leave me clear to reinstall my Borland C packages and get back to work.

Rhoel.

[idpnd]

He's still alive! Hehe

I think the OP is looking for an open format for georeferencing raster charts - would this be feasible?

[Rhoel_Asia]

Originally Posted by dacust

Manimaul,

I never took KML seriously. I thought of it as a Google thing, and didn't realize what it really was. What I am embarrassed about is I just went on that impression and never actually checked it out.

I just now googled "KML standards" and see it is much more than I thought it was. Maybe it really is what we need.

KML Standards

-dan

The history is it was created by keyhole who sold it to Google - they adopted it and sponsored it as a true open standard. It's related to GML and more importantly, both formats are deliberately on collision course ... at some point in the future, they will merge or skip happily along together as identical twins sisters. The current version on KML has been locked off, to allow people to have a stable environment to work in.
The real test will be when we start to build proper working charts with it - just how big/small will the data sets be? I have one KMZ world base map which is 3megs (the KML is 10megs). I have tested it against hi-res data of the local seas and I would be very happy to head off across oceans with it - not good enough to get you into port but enough detail to get you to the right bit of the world.

re Manimaul 256 tiles: I need to check this out in more detail, Fundamentally the KMl is point, line and polygon data. So I am not sure how the tiles refered to actually work ... it migh tbe a way some programs handle the page data but the file contains only the three basic data types.

I am testing to see if PHP SimpleXML can read KML data - be fun if it can ... it means we can database stuff and do other server-based processing: Not sure folks here want to go that route).

Rhoel

[dacust]

Originally Posted by Rhoel_Asia

...
re Manimaul 256 tiles: I need to check this out in more detail, Fundamentally the KMl is point, line and polygon data. So I am not sure how the tiles refered to actually work ... it migh tbe a way some programs handle the page data but the file contains only the three basic data types.

I am testing to see if PHP SimpleXML can read KML data - be fun if it can ... it means we can database stuff and do other server-based processing: Not sure folks here want to go that route).

Rhoel

Rhoel,

So, if you were to choose what format to display raster charts in a marine navigation program, what would it be? Must be public domain or open source. Be nice if the format and supporting tools already existed for at least the basics.

Remember, the goal is to be able to scan, geo-reference and use our own raster charts. And the resulting charts should be totally legal to distribute.

As you can see from earlier threads, it does not have to be turn-key. We can work to make it that way ourselves.

-dan

[Rhoel_Asia]

Originally Posted by dacust

Rhoel,

So, if you were to choose what format to display raster charts in a marine navigation program, what would it be? -dan

Short answer is i have no idea - I'm a vector kind of guy. The only time I play with geoTiffs is to vector off them.

However, some years ago I worked on the technical side of animation production ... we used to take pencil drawings and auto-vector the fill areas ready for ink & paint. The software detection was never 100% but it did work reasonably well, a 98% maybe. Basically, it looked for density and lines which joined up/had minor gaps (you could set the gap distance to suit).

Once it had a continuous line (same tone levels), it would thread a vector along it like water down a hose pipe. The key was getting a good thin line, done by using black and white level clamping - anyone who has played with the Ctrl + L setting of PhotoShop will understand how level clamping can turn a grey line on a lighter grey background into a pure black and white result. That's the principle to auto-vectoring, screen out the rubbish and hopefully what is left is your trace object.

A colour picker might help on colour charts to tell the converter the colour of land/drying area/shallows etc, as it would give an additional level of filtering ... it knows land has no blue bits: A black and white chart can have white land and white sea - which are the polygon object areas - is that a lake in the land or an island out at sea? Up to you!

The algorithm will be no fun to write either - what resolution the vector points are ticked off ... you don't want to have a thousand points along an 8" straight wharf edge when 8 might be enough; Conversely, a rocky coastline might need 100 points per inch to accurately describe the arcs.

Lots of questions: Are BSB only in colour? What resolution are people scanning stuff in? How are they stitching the A4 tiles together?

Some some agreed test data would be good to evaluate, preferably an actual dogeared chart which has been scanned and stitched - a warts and all real-world file.

In principle, it is feasible to auto vectorize a raster chart: A colour one would be easier than a grubby black and white. Once in a point, line or polygon vector, storing it as a KML would be possible, though the Mercator projection on some charts might make life interesting. Just as in the animation software, you would need to go back in and point edit some of the data - things like 6 can become 8, 5 -> 9, 8-> 9 etc.

So auto-vectoring is possible though it's no easy beast to write.

Is this approach an option?


Rhoel.

[jonasaberg]

Some extra info from my communication in Swedish with Olle:

WCI use RLE compression since it gives very fast decompression. He also note that the time needed for compression is of no importance compared to the decompression performance.

The new format should be completely open from start. Since he is the inventor of WCI he argues that he might in some aspects be held responsible for the format even if it is made open. He prefers a fresh start, also in the sense to correct mistakes created in the WCI format.

Olle also sets some requirements on the new open format:
-Handle large files, at leat 32k*32k.
-Row or block oriented
-Contain pointers to be able to quickly load a part of a map.
-Easy access to a calibrated data header

How does KML correspond to this? I have just given it a 5min glance and it seems like a markup language at first glance- not a graphical format - but maybe I need to read better and deeper.

/Jonas

[jonasaberg]

Oh, I get it. KML is a vector format.

It is going to be mayhem to quality control the auto vectoring. It will feel no safe at all navigating an area unless you trust your chart 100%.

/Jonas

[barnakiel]

Originally Posted by jonasaberg

Oh, I get it. KML is a vector format.

It is going to be mayhem to quality control the auto vectoring. It will feel no safe at all navigating an area unless you trust your chart 100%.

/Jonas

PLS expand. Meaning - vector = unsafe, auto vectoring = unsafe???

b.

[sping]

Hi everyone,

Not sure what I can add right now. I think OpenCPN is a very exciting development in progress and I think it is up to the dev team to decide the route to go.

In addition to my previous thinkings I may add that with a paletted format is much easier to change brightness.

I also agree that finding an existing open format would be best. Never did see one that fitts though.

If it is decided a new open format is the way to go, I will try to help when possible.



/Olle