Charts II: NGA - 2700 Charts

[allanpeda]

Originally Posted by cagney

I think that a correct prefix to these DMA charts (sorry, old habit) should be
NGAxxxxx_short_version_of_name. Don't know where you guys get NOAA from in this connection?
Look at their web site with names as "NGA Chart: 27160 " etc.
Problems with spaces in names can pop up in unexpected places. The last thing I heard of was in the code for windows installer in OpenCPN.... Best to avoid it altogether by using underscores.
Thomas

Will do. I grabbed NOAA* from out of the blue.

[allanpeda]

Originally Posted by Amgine

on *nix (including Mac), nice is your friend for running intensive scripts while not losing all of your cpu...

I reniced all the way up to 20, without it my system would have been crippled.

[Surveyor]

I have 84 charts stitched. 91004 to 92470 I think. The python script running in batch mode is nice. I dont have to be sitting there, it just does it and keeps going. My pc is kind of old with only 2gig of ram so it does get bogged down and the script will stop when it senses the ram running low. I installed a ram manager that has helped some but didnt completely eliminate the problem. So, the script will do a good run of about 10 charts, sometimes more, sometimes less and by then enough leak has occurred to cause it to stop.

I am on my toughbook and an anonymous neighbor's wifi atm . The pc is upstairs churning through the last of the latest list I gave it. I am going to give it a go on my computer at work tomorrow. It is a powerhouse CAD machine with 8gig of ram. My employees will just have to live with the slower facebook performance!

[LeaseOnLife]

Originally Posted by cagney

I think that a correct prefix to these DMA charts (sorry, old habit) should be
NGAxxxxx_short_version_of_name. Don't know where you guys get NOAA from in this connection?
Look at their web site with names as "NGA Chart: 27160 " etc.
Problems with spaces in names can pop up in unexpected places. The last thing I heard of was in the code for windows installer in OpenCPN.... Best to avoid it altogether by using underscores.
Thomas

No spaces, really, sic, what else to avoid?
Some names have a dot or comma:

• Cape St. Francis to Ystervarkpunt

Or other characters which can cause problems in a script:

• Limpopo to Ponta do Ouro (Mozambique)
• Raas Aantaara to Raas Binna including `Abd Al Kuri
• Idi to Assab including Jabal Zuqar and Jaza'ir Hanish

I will leave this to proper programming/scripting and keep the names as they appear on the NGAViewer website or in the region-pdfs.

[NavPak ECS]

NGA probably has all these charts in 16 color Packbits TIFF format, and it would be great if they had a public release, but they will contain so many foreign copyrights..... failing that I would like to share my experience of making charts for what it is worth. In the meantime, I think it is important to capture the tiles in something loss less before one of the foreign HOs complain.

After scanning thousands of charts, I worked out a system that worked well for me. The charts were scanned at 200 DPI. NOAA BSB charts are 254 DPI which is a little too high and results in an over-zoomed condition. The charts came out of the scanner as 24 bit TIFFs that look like Braks first thumbnail in this thread. The clarity is good, but way too many colors. Using Paintshop Pro, I downsize to 16 colors then use the Color Replacer tool to consolidate the palette. The Color Replacer tool allows you to move colors to a different palette index.

Below is a clip from the MapSetup help file that explains one way to do it:

When you look at the 16 color palette, you will probably see 2 or 3 shades of white. If you change these to bright white, then you will probably see an immediate improvement in the clarity of the image. Also on the palette, you will probably see various shades of dark gray or near black. If you change these all to black, you will probably see another improvement in the clarity of the image. When changing colors on the palette, you should carefully watch the changes to the features on the chart. For example, one of the dark gray colors may be used predominately in light house symbols, so it will look better if changed to dark magenta, rather than black. To see effects of changing a palette color, you can temporarily change to red. Then you can see if the color is mostly a land or sea component, or used primarily somewhere else. For example, you may see 2 similar shades of light blue. By changing these to red, you can see if it is mostly used to show shallow water, or if it is mostly shown in the land areas.

If the color is mostly used to show shallow water, then change it to the same color used for shallow water, or if the color is predominately used to show land, then change it to the same color used for land.
Changing some of the colors to darker shades will improve the appearance of the chart, and sometimes the color, contrast, and brightness tools in the image editor can be used to enhance the image. These tools will generally have the same effect as manually editing the palette, described above, except they will act on the whole palette in one operation. With a little practice, you can quickly edit the palette and brighten up the image.

CONSOLIDATE PALETTE
This is another optional step, which will reduce the file size, and improve display speed. A consolidated palette means much less work for the computer, and a smaller file size.
After editing the colors of the chart, you may have multiple whites. These can be consolidated to one white palette entry, to improve compression ratio and improve decompression speed. The RLE and TIF Packbits compression are similar. Lines are compressed, to reduce repetitive color entries. For example, a compressed image row could be:
4xBlack, 48xTan, 128xBlue, 128xWhite….
If the palette is not consolidated, the same line could look like this:
1xBlack#1, 2xBlack#2, 1xBlack#3, 4xTan#1, 24xTan#2, 20xTan#1, etc… Clearly the first example is more efficient. With a little practice, you can quickly edit and consolidate the palette in one step. For example: an image has 3 shades of dark gray. Change the darkest one to black, and use the Color Replacer or Mover tool in the image editor to move the other 2 dark grays to the black palette position, then turn the gray palette entries to bright red to show that they are not used in the image.When you look at the palette, you will probably see 2 or 3 shades of white. If you change these to bright white, then you will probably see an immediate improvement in the clarity of the image. Also on the palette, you will probably see various shades of dark gray or near black. If you change these all to black, you will probably see another improvement in the clarity of the image. When changing colors on the palette, you should carefully watch the changes to the features on the chart. For example, one of the dark gray colors may be used predominately in light house symbols, so it will look better if changed to dark magenta, rather than black. To see effects of changing a palette color, you can temporarily change to red. Then you can see if the color is mostly a land or sea component, or used primarily somewhere else. For example, you may see 2 similar shades of light blue. By changing these to red, you can see if it is mostly used to show shallow water, or if it is mostly shown in the land areas.

If the color is mostly used to show shallow water, then change it to the same color used for shallow water, or if the color is predominately used to show land, then change it to the same color used for land.
Changing some of the colors to darker shades will improve the appearance of the chart, and sometimes the color, contrast, and brightness tools in the image editor can be used to enhance the image. These tools will generally have the same effect as manually editing the palette, described above, except they will act on the whole palette in one operation. With a little practice, you can quickly edit the palette and brighten up the image.

End excerpt from help file.

The resulting image made from a DMA chart would have about 5 colors. Before doing the above. I would crop the excess collar, sometimes moving notes closer to the neat lines for a tighter crop.

The image is then saved as a TIFF with Packbits compression, same as the government topos. The NOAA BSB charts compress a little better, but these homemade TIFFS display faster, inspite of the pixalation. I think that is because Packbits is simpler than NOS compression (KAP and CAP charts). Anyway you guys are planning to make BSB, but I hope the stitched images will be available in PNG or something loss less.

I think it is important to reduce the colors as much as possible to get acceptable performance on slow hardware. The DMA charts are only about 5 colors so the palette should be crunched down to about that and fit into a 4 bit per pixel format (standard 16 color image). This should then convert to a BSB more gracefully than a JPG which is designed for photographs with color gradients.

Nuff said; getting tired.

[r.fairman]

I think the chart counts is going to be way less than 2700 The latest North Amearica catalog publised June 2011 has been reduced form approx 150 charts to approx 22 ckarts.this is a reduction of 85% If this is repeated over the whole NGA catalog th chart count will be 400 charts. Have any of the converters found their areas catalogs have been reduced The conversion process is certainly worthwile. I suspect NGA are reviewing their catalogs one by and removing charts for some reason I appeciate NAvPaks excellent post regarding chart colors but our prority must be to get hese charts saved in their native format ASAP and convert later. This is resource that is dwindling and once gone may never be again on offer.
Brak any ideas
regards

Richard

[LeaseOnLife]

Here is my count:

   Region       # of charts                                                
         1              20     North America
         2              619    Central & South America 
         3              129    UK, Iceland, Greenland, Western Europe
         4              206    Scandinavia, Northern Russia
         5              465    Western Africa, Mediterranean, Black Sea
         6              304    Eastern Africa, Southern Asia
         7              264    Autralia, New Zealand, Indonesia, Papua New Gui.
         8              135    South Pacific Islands
         9              484    Eastern Asia, South Eastern Russia, Philippines
         10             99     Miscellaneous 
                                                                               
       total           2725

In the NGA-pdfs describing each region, one can see the charts which have been removed, usually detail charts, like harbor charts. Regarding the US, everything seems available from NOAA.

I am scraping Region 6, some 85 files so far out of 304. Every single chart, out of the 304, listed on http://www.charts.noaa.gov/NGAViewer...werTable.shtml has been available, so far.

For these 85 files I need 2.9Gbytes, full size jpgs, output of cagney's ruby script. Files are here: http://www.4shared.com/dir/toctSpJ5/NGA-CHARTS.html

And yes, THANKS to NavPak ECS, your post gave me a better understanding regarding the colors (and the effort needed).

[LeaseOnLife]

Reading brak's and NavPak's posts regarding color space, and trying to understand the implications, I have to wonder:

Do we have to go with a 4-bit color space? My thinking goes along these lines:

• use 8-bit color space instead, 256 colors.
• with 256 colors we don't necessarily have to go with indexed colors, my experiments with direct-8-bit colors look good to me. Some users might want to spend extra efforts and do go for indexed colors. But I doubt the indexing can be automated.
• to reduce filesize and amount of pixels to be handled in memory, we could reduce the image in size. I have good results with a 50% reduction (50% witdth x 50% height).
• leave the borders for now, the manual calibration step can take care if this if we want to.

The above steps are easy enough to automate and the results aren't bad at all. Have a look at the 3 png images in http://www.4shared.com/dir/eOjG7lw9/...ed_charts.html

I would love to test these charts in opencpn, but I haven't been through the Mapcal-gif-tiff-kap process yet. I could use some help here!

I did calibrate the images for Fugawi, they work fine on my low power netbook. The FX3 files are available for download too.

[cagney]

Originally Posted by LeaseOnLife

.......
I would love to test these charts in opencpn, but I haven't been through the Mapcal-gif-tiff-kap process yet. I could use some help here!
.....

Here is one of your charts - 4shared folder - My 4shared

The "Mapcal-gif-tiff-kap process" is not recommended.....Use "imgkap" instead and produce a kap directly from a png or even a jpg. Using convert twice for each file on top of everything else will really slow things down.
The main thing is to create a "template" equal to the beginning of the text header in a kap file, that contains good geo-referencing. As the 4 corners of these charts has a very precise geographical location, it seems natural to me to use these as reference points, just converting the positions to decimal degrees and noting the exact pixel location for each corner.
It seems natural for me just to start with template template/header file and just fill in the specific for each chart.

Brak talked about inserts o the charts. I have a crop script for that, using convert. To white out the remainder on the main chart is just a question of marking an area with gimp and hit "delete" and "save".

jonasaberg talked about the quality of the charts. I agree that we need some kind of quality standard an peer review. I also think that there is a need for, say two, different sizes/qualities, one full size/max-quality, and one size, much smaller, that is happy as long as the chart don't pixilate at zoom x2.


There is a thread dealing with imgkap here: http://www.cruisersforum.com/forums/...ool-56600.html

Thomas

[LeaseOnLife]

Thomas

thanks for pointing me towards the imgkap process!

Trying to read thru the whole thread and learn....

In the meantime, could you repeat the kap generation on 61591 with less colors, 256, or 128? I think the kap file now has 24 bit colors and the size is 2.5x the source png.

[cagney]

Originally Posted by LeaseOnLife

Thomas

thanks for pointing me towards the imgkap process!

Trying to read thru the whole thread and learn....

In the meantime, could you repeat the kap generation on 61591 with less colors, 256, or 128? I think the kap file now has 24 bit colors and the size is 2.5x the source png.

That can't be right as a kap contains a maximum of 128 colors! The size is often bigger than the picture to a varying degree. There s an option in imgkap to reduce colors, that is not recommended at the moment.
Tiff2bsb has this option, but I generally find that imgkap produces a touch smaller kap files.

To get you guys going creating your own charts here is a brief summary of the way I do it. Just covering a standard "case"

The whole trick more or less centers around creating a "header" file ( must end in .kap for use with "imgkap", a silly restriction in my mind)
Lets take an example and make chart 57472 Approaches to Saldahna Bay.
The final header should look something like this(not including line numbers):


Remember to always use + for N and E and - for S and W.
I use GIMP for the pixel-count an image info and software calculator to transform degrees minutes and seconds to a decimal value.
In linux speed things up by remembering that if you "paint" a value you can then paste that value by a middle click.
My calculators value is also available with a middle click after a CTRL+C
If you in the end prefer to use "MapCal",or something else, instead of "my" method, no problem, as long as the header file. the end result, is OK.

Lets look at the lines above:

1 This is the BSB version. 2.0 or 3.0 is OK . OpenCPN don't need this line. But other plotters do.
2 The Name of the chart
3 This line is a continuation of line 2 (max 80 chars /line) and as such starts in the 5:th column -> leave 4 spaces blank.
The values represents the chart number, x an y in pixels and DU is the resolution 72x72 dpi
4 SC is the chart scale
5 PP is the latitude for the scale. Use the values from the chart.If not available, use mid latitude.
For Transverse Mercator charts always use the mid longitude. Leave the other values as is.
6 UN units for depth. SD date for sounding , get both from the chart.
7 Using the NOAA std SE: chart edition RE raster edition = 1 !! ED date of edition in US format. imgkap gets this wrong at the moment.
As a stopgap, I, for the moment write all info in SE

REF: Geo referencing points. Use at least 2 positioned diagonally. But better follow my example and use the 4 corner of the chart starting in the lower left
(SW) corner and proceed clockwise.
The values are: pixel x, pixel y, lat, long note the different order between pixels and lat/long.
PLY: These values represents a polygon following the inner (useful) border of the chart. In OpenCPN it is equal to
what will be seen when quilting.
In the real world, just copy the lat/long values from REF x to PLY x (to start with)
More PLY values can be added (later) if necessary

DTM is the values in seconds of arc to make the chart agree with WGS84. This often means to apply the reverse sign to any
corrections mentioned on the chart, if any( mostly a feature of older charts)

Call the header for ex 57472hd.kap- To create a chart use a similar command line to this.
$imgkap 57472.jpg 57472hd.kap 57472-Approaches_to_Saldahna_Bay.kap
This produces a chartfile 57472-Approaches_to_Saldahna_Bay.kap

Test the chart for accuracy. The red PLY line should follow the "inner" line of the chart

Note that if you download 57472 and try this out you will probably have to change the pixel values as I have made the size smaller, 50%, I think.
No guarantee.

Thomas

[HappySeagull]

Thomas (and others)
PER
..." must end in .kap for use with "imgkap"

you can use ".txt" for the header file in the newest version...

[NavPak ECS]

<< to reduce filesize and amount of pixels to be handled in memory, we could reduce the image in size. I have good results with a 50% reduction (50% witdth x 50% height). >>

Your PNG 61594, reduced 50%, looks like plenty of resolution for a finished chart, indicating that they are coming out of Dezoomify at very high rez. As Cagney said, if it doesnt pixilate at zoom 50% then the rez is probably high enough. The 256 colors looks good but the file size is still big; maybe fine for some purposes; just depends on what you want. As someone said, some of the images are coming out of Dezoomify pixelated and some have good solid colors. Your chart 61594 looks to be the latter, in which case no manual processing is required to downsize the colors. In Paintshop Pro, I downsized it to 16 colors in 2 mouse clicks. The processing time was about 2 seconds. The file size went from 3129 kb to 827 kb. I was surprised about that; would have expected about half size.

If it is easier to grab the data now in 256 colors, then by all means there is nothing wrong with it. As you suggested, later it will be easy to downsize. Manual processing as I described previously only seems to be necessary for pixilated images.

[LeaseOnLife]

Thanks all, I think the consensus is to keep grabbing and stitching. The reduction plus calibration can come at a later time. In the mean time we can work on some "best practices" for reduction and calibration.

But we need storage space, now. I use a 4shared account. A free 4shared account gives 10Gbytes. The exchange between 4shared members is fast and easy, the windows desktop client is OK, from my Gentoo box I use webdav for uploads. Download by non members is handicapped by a delay and lots of ads. The owner of the account has to login every 30 days, otherwise the files get deleted.

For Region-6, I am at 135 files, using 4.1 GB:

http://www.4shared.com/folder/__i2G9Rw/Region-6.html

Extrapolating, I will need all 10Gb for Region-6, for 305 files.

Looking at an estimate of 100Gb for all 2725 charts, we need 10-12 accounts, exclusively for the full size charts.

Is this the best way to go?

[jonasaberg]

I think the best is if someone can gather all the files locally, create a torrent, and put it on the net.
I think many of us is willing to share a torrent for a long time since this is non-copyrighted material.

The best is if we can organize so that someone can gather all the data in one place, create the torrent and start to share.

So, I see it as:
1. Download all the file fragments
or/and
2. Download and stitch them and create (big uncompressed) files.

3. Create torrent and share.

4. Now everybody interested in stitching/georeffing and compressing them can get the raw files and work on them.

5. For storing finished georeferenced files the same system could be used, but it would be better with a website with a system for peer review and some kind of moderation where a superuser can remove incorrect files.