Charts II: NGA - 2700 Charts

[wdkester]

I am trying to convert a mapcal header and png file to create a bsb/kap chart. No matter what type of graphics file I use (bmp, tiff, png, jpg) I get a microsoft error as follows:

EventType : clr20r3 P1 : kap-file utility.exe P2 : 1.0.0.0
P3 : 4bb61efa P4 : microsoft.visualbasic P5 : 8.0.0.0 P6 : 4889f422
P7 : 345 P8 : 4f P9 : system.invalidcastexception

I'm using the chart 24402 downloaded from the Opencpn repository. It has passed thru a utility, nconvert -out png -resize 50% 50% -colors 32 24402.jpg. I realize that this doens't follow the DACUST flow chart, but would this cause a microsoft error? Am I doing something wrong?

[nohal]

David...
Not much info... The system.invalidcastexception means that there's a problem with datatype conversion, but nothing else can be said from the error.
Where does the "kap-file utility.exe" come from and how do you invoke it?

Pavel

[wdkester]

Originally Posted by nohal

David...
Not much info... The system.invalidcastexception means that there's a problem with datatype conversion, but nothing else can be said from the error.
Where does the "kap-file utility.exe" come from and how do you invoke it?

Pavel

I got the utility here KAP File Utility

But dacust says that another Opencpn guy (Netsurfer) wrote it. I was hopeing he would reply.

I'm trying to get up to speed so that I can help the project.

[HappySeagull]

wdkester...I've had no luck with nconvert for charts.It always gives me an error when I use its tifs with tif2bsb.

[HappySeagull]

PS.Nohal,I'm wrong about scale.It actually works well as you said on the wiki-using the chart's illustrated scale along with the 50% reduction in image size....the kaps I've made line up nicely with other charts I have...

[wdkester]

Originally Posted by HappySeagull

wdkester...I've had no luck with nconvert for charts.It always gives me an error when I use its tifs with tif2bsb.

I rather liked it because with it you could reduce size and color at the same time.

What are you using?

I really really want to avoid command line utilities as they (for me) are so error prone, and take a lot of time to learn.

Rather than the entire conversion process, could we agree on the next step? Going from the jpg to ?? Do we reduce the size? Do we reduce the color? What utility do we use? Do we put the file in the repository?

[dpons]

Originally Posted by brak

There is an issue in these charts that will make building good quality BSB challenging.

The charts appear to be scanned directly from paper charts. This is different from specially crafted images used in raster charts. Consider the tile (zoomed in) below.

What should be a uniform color, is actually a collection of "paint pixels" - artifact of offset printing.
BSB is best encoded from data that has large uniform color surfaces, that can be compressed in RLE. This data would result in terrible RLE - larger than original size.

They need some sort of color/graphic processing to turn pixelated color areas into solid colors.

I had good luck with the Gimp, using Filters->Blur->Selective Gaussian Blur. Radius set to 5, delta to 80-100, It does not disappear, yet is much less annoying.

This is your pic after treatment:

[dacust]

Originally Posted by dpons

..., It does not disappear, yet is much less annoying.

...

The annoying part is not really what we are trying to fix. The pixleated stuff does not compress well, so it makes the images take up MUCH more space on disk.

Example: one I saw that had homogenous colors took 22mb.
A similar chart with the pixleated look was 223mb.

So, the question is, the treatment you did to it, did it reduce the storage space required? If so, cool.

As a separate issue, it's also cool that you found a way to make it look nicer.

-dan

[nohal]

Originally Posted by wdkester

Rather than the entire conversion process, could we agree on the next step? Going from the jpg to ?? Do we reduce the size? Do we reduce the color? What utility do we use? Do we put the file in the repository?

We (especially Netsurfer) are pretty busy working on a web based system for the calibration of the whole set. With the 2725 charts (and way more resulting KAPs) to process, it's probably the only way not to end up with a total mess.
The plan is to progress with the calibration starting with the rectangular charts, as they are easier to handle, extending the system to the more complicated cases, composed of more charts in one image or needing rotation or other preprocessing as we progress.
We should be able to add features to import the headers created locally to the database, but please make sure to record very exactly the preprocessing steps you used (cropping coordinates, down-scaling, image rotation, filters applied, palette reduction etc.) so that we can reproduce and/or recalculate them server-side.
Please stay tuned for more info on this subject after the weekend.

As far as the toolset goes, from my experience, the fastest viewer for Windows is Google Picasa (IrfanView also reported to work well, if I remember correctly)
For the resizing, rotation, colorspace conversion etc. I'm using GraphicsMagic from the command line, which is much more resource efficient than ImageMagick, while being almost 100% parameter compatible.
From the image editors to do the image manipulation, Photoshop and Gimp work well, while Windows Paint, Paint.net and Pinta are not able to open the images at all (at least on my machine with 32-bit Win 7 and 3 gigs of RAM).
To produce the KAPs, we will most likely end up with a slightly modified version of imgkap (TBD, but for your local use the one available now is absolutely OK)

To maintain the highest possible accuracy, we would like to calibrate at full resolution - But don't even try it in tools like MapCal, it's a waste of your precious time...

And don't fear the command line, you will find out that you are using about 3 or 5 commands repeatedly all the time

Cheers

Pavel

[HappySeagull]

Originally Posted by wdkester

I rather liked it because with it you could reduce size and color at the same time.

What are you using?

I really really want to avoid command line utilities as they (for me) are so error prone, and take a lot of time to learn.

Rather than the entire conversion process, could we agree on the next step? Going from the jpg to ?? Do we reduce the size? Do we reduce the color? What utility do we use? Do we put the file in the repository?

With irfanview's batch gui,I'm reducing size to 50%,and saving as bmps for MapCal to use(large pngs aren't working).But I can do everything all at once there as to down-sampling colours and all for tifs etc.....128 works well.16 needs a palette change to bring back the white....then,calibrated,bordersand I add a comment:"OpenCPN NGA Project" Next,I make a stack of headers with mc2bsbh.
The cmd line can be totally avoided with a batch file.For mc2bsbh, I just use:

@echo
mc2bsbh CHARTCAL.DIR
@echo so,several hdrs should appear...
@pause

in a ".txt" file renamed to".bat"and I edit these to fill in the PP and SD values..such a nice header file it makes!Finally,onto tif2bsb in a batch to make kaps.

But with Nohal and netsurfer working up an online calibrator?
Well,that'd be fantastic!and solve a lot of the confusion of personal recipes...

[dpons]

Originally Posted by dacust

The annoying part is not really what we are trying to fix. The pixleated stuff does not compress well, so it makes the images take up MUCH more space on disk.

Example: one I saw that had homogenous colors took 22mb.
A similar chart with the pixleated look was 223mb.

So, the question is, the treatment you did to it, did it reduce the storage space required? If so, cool.

As a separate issue, it's also cool that you found a way to make it look nicer.

-dan

After 50% scaling, conversion to 127 colors, blurring and converting to tiff, they are about 1/5th the original size. I don't know how that compares to other's procedures. I use LZW compression on tiff, though, which is non-loosy. JPG's have much better compression ratio.

BTW, convert goes nuts in my pc (debian, 4 GB RAM). Doing all the stuff in gimp is much faster, although needs operator interaction.

--Diego

[HappySeagull]

I'm not compressing tifs at all but the kap for 51007_NGA CAPE FINISTERRE TO CADIZ weighs in at 1.38mb (half-sized)from the original 5.89mb jpeg.(336 mb as a bmp!)

[phiggins]

WDKester and myself have developed a fairly simple way to create kap charts from the NGA jpgs with some modifications to GE2KAP. Here's how we do it:

1. Reduce the size of the JPG to a png using:
nconvert -out png -resize 20% 20% xxxxx.jpg

2. Use Mapcal to calibrate the png. Must have at least 3 calibration points and there also must be a point in the upper right corner and lower left of the chart. Usually charts have the lat/lon information for those points. The modifications to GE2KAP will crop the chart to those two points. The reason we do this is so charts will be displayed properly in OpenCPN if quillting is on.

3, Save the calibration data which Mapcal saves in the CHARTCAL.DIR

4. Start GE2KAP and enter the png image file in the overlay field. The modifications to GE2KAP will check if there is a CHARTCAL.DIR and see if there is an entry for the image. If there is, it will use the calibration data and crop the image to the UR and LL points and produce a chart. You can either overlay the chart on GE or create a chart directly from the image without using GE.

The GE2KAP version is V6.0.2 which I haven't uploaded to my website yet. If you want to try it, send me an email:

phiggins()gmn-usa.com (replace the () with a @)

[nohal]

Paul,
could you please avoid the cropping and use PLYs instead? Or at least save the original coordinates of the two points used for cropping in the header comments so that we can use such headers in our automated process?

Thanks

Pavel

[cagney]

phiggins

Regarding your #2. No need to crop the chart at all, for the stated reason.
OpenCPN only shows the area enclosed by the PLY polygon, when in quilting mode.

Thomas