Charts II: NGA - 2700 Charts

[LeaseOnLife]

Originally Posted by Netsurfer

Why would you like another wiki/ place to gather the worthy information?

Wordpress is great to document things, stable processes, etc

What is "worthy information". At times we don't know what we know. A Wiki is a bit more like brain storming. Creating pages with some thoughts, other users adding/correcting info. This is often an unstructured mess of pages. Good info could then be moved in a more structured space, like wordpress.

But can't we have both? Wordpress has plug-ins, and one plugin is
http://wordpress.org/extend/plugins/wordpress-wiki/

Make one page for

• image conversion
• header creation
• kap creation
• whatsoever

I see pages also for

• existing tools (i.e mc2bsbh): command line or guis: usage, quirks, advantages, best practices to install and use the tool
• explain the content of the hdr (and the .bsb file). Where should info be taken from to fill the hdr. can this be automated?
• Compatibility Matrix for the different charting programs, document the special needs (i.e RayTech needs a .bsb)
• the different download and stitch scripts, prerequisites...
• quality: at what points do we check for quality, and how
• document test cases
• and so many more

As I said, unstructured thoughts, sometimes the structure comes after the content. And that is what I like Wikis for.

[Rapport]

Originally Posted by LeaseOnLife

.bsb file content to satisfy Fugawi and RayTech

Here is an early result of my trials:

!Copyright 1996, NDI. All Rights Reserved
    !Comment...
CHT/NA=VANCOUVER ISLAND JUAN DE FUCA STRAIT TO ,NU=3001
    !above line needed by RayTech but not by Fugawi
CHF/GENERAL
    !Fugawi and RayTech don't need CHF/    
CED/SE=3001, RE= 300101, ED=01/30/96
    !RayTech needs above string,  Fugawi does not    
VER/1.1
    !RayTech needs VER/  Fugawi does not
CHK/1,300101
    !Fugawi and RayTech don't need CHK/ (but maybe if the bsb catalog has more that one chart)    
CGD/UNKNOWN
    !Fugawi and RayTech don't need CGD/ 
ORG/Nautical Data International Inc., 1 Military Rd., P.O. Box 127, Station C S
    t. John's, NF, A1C 5H5, on behalf of: Canadian Hydrographic Service, 615 Bo
    oth St., Room 237, Ottawa, Ontario, K1A 0E6
    !Fugawi and RayTech don't need ORG/  
K01/NA=VANCOUVER ISLAND JUAN DE FUCA STRAIT TO ,NU=3001,TY=BASE,FN=300101.kap
    !above line needed by Fugawi

One .bsb file can describe more than one .kap, but that gets more complicated. If we create a .bsb file per .kap, the following lines, should satisfy both, Fugawi and Raytech:

CHT/NA=VANCOUVER ISLAND JUAN DE FUCA STRAIT TO ,NU=3001
CED/SE=3001, RE= 300101, ED=01/30/96
VER/1.1
K01/NA=VANCOUVER ISLAND JUAN DE FUCA STRAIT TO ,NU=3001,TY=BASE,FN=300101.kap

Anybody care to try and verify?

Re BSBs and Raytech.
Ive had no problem running the kapgen created NZ charts on coastal navigator and Fugawi, without BSB files.
Kiwitech, the progenator of Raytech and iexpedition, needs the polyclonal (-p) of mc2bsbh to run. I'm not sure if Raytech doesnt. Raytech needs BSB, and
Raytech for me needs CHK , depending on Ty, (base, insert, extension) and is fussy about how the files are filed. Easier to use Opencpn.

[bcn]

This morning I was playing a bit with the workflow and resolutions/file sizes.
All done in GIMP

Using the 53093 (this the smallest 100% jpg quality chart for me) I tried the following steps:

original 100% 13.020kB
resize 50% 4.667kB
reduce palette
256 colors 4.776kB (!)
64 colors 4.694kB
32 colors 4.682kB

converting png (compression is 5)
256 colors 7.242kB
32 colors 1.674kB

For me the 100 zoom for the 32c png and the corresponding 50 zoom of the original 100% jpg have only a very slight difference. The 200 zoom for the png is blurred and corresponds to a 200 zoom of the original in quality.

If there is interest I can upload the samples.

Hubert

[cagney]

Originally Posted by bcn

This morning I was playing a bit with the workflow and resolutions/file sizes.
All done in GIMP

Using the 53093 (this the smallest 100% jpg quality chart for me) I tried the following steps:

original 100% 13.020kB
resize 50% 4.667kB
reduce palette
256 colors 4.776kB (!)
64 colors 4.694kB
32 colors 4.682kB

converting png (compression is 5)
256 colors 7.242kB
32 colors 1.674kB

For me the 100 zoom for the 32c png and the corresponding 50 zoom of the original 100% jpg have only a very slight difference. The 200 zoom for the png is blurred and corresponds to a 200 zoom of the original in quality.

If there is interest I can upload the samples.

Hubert

My understanding is that the first step must be a strait conversion to png, before anything else is done. The reason for this is that each operation on a jpg contributes to data loss.

Thomas

[LeaseOnLife]

Originally Posted by Rapport

Re BSBs and Raytech.
Ive had no problem running the kapgen created NZ charts on coastal navigator and Fugawi, without BSB files.
Kiwitech, the progenator of Raytech and iexpedition, needs the polyclonal (-p) of mc2bsbh to run. I'm not sure if Raytech doesnt. Raytech needs BSB, and
Raytech for me needs CHK , depending on Ty, (base, insert, extension) and is fussy about how the files are filed. Easier to use Opencpn.

Thanks for the info!

I am at the point where Raytech imports the chart (bsb+kap file), shows the chart in the list, shows the outline of the chart at the correct position, but no picture within the outline-frame. I tried 16, 64, 127 and 128 colors, RGB as well as indexed, no difference.

Since you have been down that road before, could you upload a homemade kap + bsb which does work with Raytech? From there I might be able to see what I am doing wrong.

"polyclonal (-p) of mc2bsbh"
I tried (mc2bsbh v0.9), it does not complain about the -p switch, but the header is identical to one created without -p switch. Could you check on that?

OpenCPN: that is what I navigate with, RayTech is only for Radar + Overlay....

[Netsurfer]

Originally Posted by LeaseOnLife

I tried (mc2bsbh v0.9), it does not complain about the -p switch, but the header is identical to one created without -p switch.

If we know what we need to end up with the header it should be relatively easy to write a small script to perform the needed action.

That's what I propose anyway:
Let's do it e.g. by a web form. You enter the needed data (or choose from drop-down lists where possible, checkboxes, radio buttons etc.) and get the final header as return.

I think this way round we are much more flexible as when using some of the existing helper tools which do not perfectly suit our needs.

Gunther

[LeaseOnLife]

Originally Posted by Netsurfer

That's what I propose anyway:
Let's do it e.g. by a web form. You enter the needed data (or choose from drop-down lists where possible, checkboxes, radio buttons etc.) and get the final header as return.

I support that! For all the chart info, the first part of the header, with as many fields pre-filled, from the NGA/NOAA website or the related pdfs.

There are (at least) two more portions of the kap-header. The calibration (REF and PLY..) plus the color definitions (IFM and RGB). Keeping these on the server should then enable us to re-run a kap creation easily, in case we find a mistake/improvement in the headers, or the according image.

I say at least, because I found ARE/ADN sections in the NOAA charts, corresponding with Notice of Mariners, I think

[Netsurfer]

Originally Posted by LeaseOnLife

For all the chart info, the first part of the header, with as many fields pre-filled, from the NGA/NOAA website or the related pdfs.

There are (at least) two more portions of the kap-header. The calibration (REF and PLY..) plus the color definitions (IFM and RGB). Keeping these on the server should then enable us to re-run a kap creation easily, in case we find a mistake/improvement in the headers, or the according image.

Yes, that's exactly the idea behind this ...!

I am already gathering every piece of information about each chart in a database.

So if we are going to add the needed information for the header as well (including REFs and PLYs) we could easily update and recreate any kap.

When doing the calibration for the original image size it should be possible to simply re-calculate them for any other image size.

Additionally we will have consistent headers not depending on the used conversion tool.

Gunther

[Netsurfer]

Originally Posted by LeaseOnLife

Wordpress is great to document things, stable processes, etc

What is "worthy information". At times we don't know what we know. A Wiki is a bit more like brain storming. Creating pages with some thoughts, other users adding/correcting info. This is often an unstructured mess of pages. Good info could then be moved in a more structured space, like wordpress.

But can't we have both? Wordpress has plug-ins, and one plugin is
WordPress › WordPress Wiki « WordPress Plugins

I had a look at it.
IMHO this makes things unnecessarily complicated as you have to deal with another wiki syntax. And experience showed that there are very few users contributing at all - see opencpn.org and opencpn.de!

I see pages also for

• existing tools (i.e mc2bsbh): command line or guis: usage, quirks, advantages, best practices to install and use the tool
• explain the content of the hdr (and the .bsb file). Where should info be taken from to fill the hdr. can this be automated?
• Compatibility Matrix for the different charting programs, document the special needs (i.e RayTech needs a .bsb)
• the different download and stitch scripts, prerequisites...
• quality: at what points do we check for quality, and how
• document test cases
• and so many more

As I said, unstructured thoughts, sometimes the structure comes after the content. And that is what I like Wikis for.

I agree with your points but I would prefer another solution:
I could easily set up a temporary wordpress blog (like the ones already present at opencpn.info) which we could use for the 'unstructured' things. If things become clearer we could transfer them to the opencpn.info/en/ in a clean structured way.

How about this?

Gunther

[nohal]

Folks...
As the grabbing progresses so nicely with almost 700 jpegs uploaded, I've started to work on the toolchain to produce the charts in a way as automatic as possible.
I've setup an organization on github to host all the involved scripts - The repository is accessible on web at https://github.com/ngacharts/ngacharts
My goal is to be able to do the whole process of image conversion, calibration and distribution at any time on the whole set or just a part of the charts automatically on any machine with the needed tools.

Everybody is welcome to join and host any homebrew tools there. As a free bonus we get a wiki and bugtracker we can use there, so no need to spend time setting up anything temporary anywhere.

So far I started with the pre-processing scripts. Right now it adjusts the levels of the image to make everything that's almost white white and almost black black, resizes to 50% and produces a 32 color palette png.

That's just an initial take and any input on this matter is welcome. I have to say I was positively surprised with the quality and processing time - From 35000, being 89 MB, the script produces 5.9 MB png in 4 minutes on my server machine.
The goal is to eliminate any manual labor, so only tools scriptable from the command line should be used. The solution is currently totally Unix-centric.

The next step is to invent a way to do the calibration and QA - I like the idea of an online "generator" for the headers - certainly can reflect the latest acquired knowledge faster than any other tools. The resulting headers I would really like to store on github though - more on that later when I'm done with some more workflow thinking and documentation.

Gunther, could you please provide me with the list of info you are currently gathering about the individual charts, so I can count with what can be automated and what not?

Pavel

[Netsurfer]

Pavel

just some thoughts ...

Originally Posted by nohal

Everybody is welcome to join and host any homebrew tools there. As a free bonus we get a wiki and bugtracker we can use there, so no need to spend time setting up anything temporary anywhere.

Fine ..., so let's start using it.

So far I started with the pre-processing scripts. Right now it adjusts the levels of the image to make everything that's almost white white and almost black black, resizes to 50% and produces a 32 color palette png.

About the colors: I have already seen charts where the land area is yellow (e.g. chart 700). And on many other charts the land area is grey. Or is this eliminated by your noise reduction scripts?
I also do not understand why you are creating a 32 color image?
When converting to a palette image format there will be a maximum of 256 colors. Also it is possible to get the number of used colors. Wouldn't it be more efficient to get this value first and afterwards producing an image with the "optimal" color count (assumed it does not exceed our 127/128 max)?

As the charts are scanned from paper charts some of them are also slightly skewed/crooked.

The next step is to invent a way to do the calibration and QA

What does 'QA' means (quality assurance)?
Hmmm ..., as much as I like the idea of automation I couldn't think of any other way as doing the calibration manually/ by hand. But maybe you have an idea ...

Gunther, could you please provide me with the list of info you are currently gathering about the individual charts, so I can count with what can be automated and what not?

I sent you an email.

Gunther

Addendum: And there are also charts with harbour plans or several parts of a river etc. (good examples are 37235 and 37246 / upload coming soon)!

[cagney]

Gunther/Pavel

Some info just have to be retrieved manually from he chart picture, just once hopefully.

GD=Geodetic Datum. Look at this:on the tracker
PR=Projection
PP Latitude where PR is valid. Mid long if tmerc.
UN Sounding units. Perhaps we can assume "Meters" unless the chart is very old.
SD Sounding Datum
DTM Corrections to adjust the charts to WGS84. Available on a few charts

REF and PLY
We should get the exact corner lat&long, from the printed information in the SW and NE corners of the charts.
We also need the corresponding pixels to a very good accuracy. I zoom in at 200% to sort the pixels out.
For most charts the 4 corner will do, but not for all, there are a few crap shots around!! Some charts - mainly inserts, needs more PLY points as the area is not rectangular.

Using the 4 corners has an other advantage. QA at a glance, the PLY line should follow the inner border very accurately.
The quality should be tested further by zooming in and having the mouse pointer over a printed lat/long crossing, and read the values. How accurate is acceptable? I suggest that up to and including the 4:th decimal should be correct with normal rounding off rules. Repeat for ~4 similar points spread out. We mustn't compromise on this, and produce a chart as accurate as possible.

Another thing is that if we make the picture a different size, only the pixel count for the corners change. There will never be any doubt as to exactly which reference points that were used originally.
I also suggest that we follow a "semi-standard" and start REF and PLY in the SW corner and proceed clockwise.

Quite a few charts have inserts, named xxxxxxA etc. For these, the corner positions, often more than 4 and the corresponding pixel count must be retrieved, as well as the Name and scale for the inserts.
The name and scales are available in the pdf catalogs for each region, so hopefully this can be automated as well.

We also have to make up our minds how to handle those chart pictures that are turned 90 degrees. In the end its just a question of the pixels, nothing else changes.

IFM and RGB etc is not our problem, imgkap/libbsb handles that.
The only thing here, is if we should use imgkap to produce a dusk and night palette, but I suggest that we don't, as OpenCPN and many other plotters takes care of dimming the whole screen.

Did I miss something?

Thomas

[cagney]

To speed up the downloading I'm using an XP box as well, with ~3GB memory.
I use the python script to try to semi-automate the down loading.
There is a problem with Windows memory handling. After 1 or 2 downloads the script stops, with the message that the next picture is to big to fit in memory. There is no problem just starting the script directly again, however.
Has anyone else had this problem?

Thomas

[Surveyor]

Originally Posted by cagney

To speed up the downloading I'm using an XP box as well, with ~3GB memory.
I use the python script to try to semi-automate the down loading.
There is a problem with Windows memory handling. After 1 or 2 downloads the script stops, with the message that the next picture is to big to fit in memory. There is no problem just starting the script directly again, however.
Has anyone else had this problem?

Thomas

Yes. It is very frustrating. I asked this question on the author's talk page:

"I am using your batch script and love it! Except one thing which is this: I am on a machine with a total possible ram of 2gig. I use FreeRAM XP Pro set to clear the ram when it gets below a certain number. This works great on slow machines, the ram gets cleared before your script knows there isnt enough and continues 8 times out of 10. On fast machines the script is too fast and starts to process the next url before the memory manager can clear the ram and the script stops. I am wondering if you can insert a 5 second pause between url processes in order for the ram to catch up. Is this possible? Thanks in advance.

It's easy to do yourself. Add "time.sleep(5)" under "for row in range(self.yTiles):" at line 313. Inductiveload (talk) 00:31, 9 July 2011 (UTC)"

I have not tried it yet but this may help. If nothing else have it pause for 30 seconds, clear ram manually then let it run.

EDIT: I would like to edit out the 100% are you sure question too.

[cagney]

Thanks Jim.
I'm testing the version below right now.
dezoomify.py.doc
Thomas