Cruisers Forum
 


Join CruisersForum Today

Reply
 
Thread Tools Rating: Thread Rating: 3 votes, 5.00 average. Display Modes
Old 12-07-2011, 22:39   #196
Registered User
 
LeaseOnLife's Avatar

Join Date: Apr 2008
Location: out cruising again
Boat: Sailboat
Posts: 1,046
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by Netsurfer View Post
Why would you like another wiki/ place to gather the worthy information?
Wordpress is great to document things, stable processes, etc

What is "worthy information". At times we don't know what we know. A Wiki is a bit more like brain storming. Creating pages with some thoughts, other users adding/correcting info. This is often an unstructured mess of pages. Good info could then be moved in a more structured space, like wordpress.

But can't we have both? Wordpress has plug-ins, and one plugin is
http://wordpress.org/extend/plugins/wordpress-wiki/

Quote:
Make one page for
  • image conversion
  • header creation
  • kap creation
  • whatsoever
I see pages also for
  • existing tools (i.e mc2bsbh): command line or guis: usage, quirks, advantages, best practices to install and use the tool
  • explain the content of the hdr (and the .bsb file). Where should info be taken from to fill the hdr. can this be automated?
  • Compatibility Matrix for the different charting programs, document the special needs (i.e RayTech needs a .bsb)
  • the different download and stitch scripts, prerequisites...
  • quality: at what points do we check for quality, and how
  • document test cases
  • and so many more
As I said, unstructured thoughts, sometimes the structure comes after the content. And that is what I like Wikis for.
__________________

__________________
LeaseOnLife is online now   Reply With Quote
Old 13-07-2011, 03:06   #197
Registered User

Join Date: Jul 2010
Location: Marlborough, NZ
Boat: Farr 1104
Posts: 13
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by LeaseOnLife View Post
.bsb file content to satisfy Fugawi and RayTech

Here is an early result of my trials:
Code:
!Copyright 1996, NDI. All Rights Reserved
    !Comment...
CHT/NA=VANCOUVER ISLAND JUAN DE FUCA STRAIT TO ,NU=3001
    !above line needed by RayTech but not by Fugawi
CHF/GENERAL
    !Fugawi and RayTech don't need CHF/    
CED/SE=3001, RE= 300101, ED=01/30/96
    !RayTech needs above string,  Fugawi does not    
VER/1.1
    !RayTech needs VER/  Fugawi does not
CHK/1,300101
    !Fugawi and RayTech don't need CHK/ (but maybe if the bsb catalog has more that one chart)    
CGD/UNKNOWN
    !Fugawi and RayTech don't need CGD/ 
ORG/Nautical Data International Inc., 1 Military Rd., P.O. Box 127, Station C S
    t. John's, NF, A1C 5H5, on behalf of: Canadian Hydrographic Service, 615 Bo
    oth St., Room 237, Ottawa, Ontario, K1A 0E6
    !Fugawi and RayTech don't need ORG/  
K01/NA=VANCOUVER ISLAND JUAN DE FUCA STRAIT TO ,NU=3001,TY=BASE,FN=300101.kap
    !above line needed by Fugawi
One .bsb file can describe more than one .kap, but that gets more complicated. If we create a .bsb file per .kap, the following lines, should satisfy both, Fugawi and Raytech:

Code:
CHT/NA=VANCOUVER ISLAND JUAN DE FUCA STRAIT TO ,NU=3001
CED/SE=3001, RE= 300101, ED=01/30/96
VER/1.1
K01/NA=VANCOUVER ISLAND JUAN DE FUCA STRAIT TO ,NU=3001,TY=BASE,FN=300101.kap
Anybody care to try and verify?
Re BSBs and Raytech.
Ive had no problem running the kapgen created NZ charts on coastal navigator and Fugawi, without BSB files.
Kiwitech, the progenator of Raytech and iexpedition, needs the polyclonal (-p) of mc2bsbh to run. I'm not sure if Raytech doesnt. Raytech needs BSB, and
Raytech for me needs CHK , depending on Ty, (base, insert, extension) and is fussy about how the files are filed. Easier to use Opencpn.
__________________

__________________
Rapport is offline   Reply With Quote
Old 13-07-2011, 03:48   #198
bcn
Registered User

Join Date: May 2011
Location: underway whenever possible
Boat: Rangeboat 39
Posts: 2,758
Re: Charts II: NGA - 2700 Charts

This morning I was playing a bit with the workflow and resolutions/file sizes.
All done in GIMP

Using the 53093 (this the smallest 100% jpg quality chart for me) I tried the following steps:

original 100% 13.020kB
resize 50% 4.667kB
reduce palette
256 colors 4.776kB (!)
64 colors 4.694kB
32 colors 4.682kB

converting png (compression is 5)
256 colors 7.242kB
32 colors 1.674kB

For me the 100 zoom for the 32c png and the corresponding 50 zoom of the original 100% jpg have only a very slight difference. The 200 zoom for the png is blurred and corresponds to a 200 zoom of the original in quality.

If there is interest I can upload the samples.

Hubert
__________________
bcn is offline   Reply With Quote
Old 13-07-2011, 03:53   #199
Registered User

Join Date: Dec 2005
Location: WNA
Boat: Dufour 35
Posts: 3,247
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by bcn View Post
This morning I was playing a bit with the workflow and resolutions/file sizes.
All done in GIMP

Using the 53093 (this the smallest 100% jpg quality chart for me) I tried the following steps:

original 100% 13.020kB
resize 50% 4.667kB
reduce palette
256 colors 4.776kB (!)
64 colors 4.694kB
32 colors 4.682kB

converting png (compression is 5)
256 colors 7.242kB
32 colors 1.674kB

For me the 100 zoom for the 32c png and the corresponding 50 zoom of the original 100% jpg have only a very slight difference. The 200 zoom for the png is blurred and corresponds to a 200 zoom of the original in quality.

If there is interest I can upload the samples.

Hubert
My understanding is that the first step must be a strait conversion to png, before anything else is done. The reason for this is that each operation on a jpg contributes to data loss.

Thomas
__________________
cagney is offline   Reply With Quote
Old 13-07-2011, 05:01   #200
Registered User
 
LeaseOnLife's Avatar

Join Date: Apr 2008
Location: out cruising again
Boat: Sailboat
Posts: 1,046
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by Rapport View Post
Re BSBs and Raytech.
Ive had no problem running the kapgen created NZ charts on coastal navigator and Fugawi, without BSB files.
Kiwitech, the progenator of Raytech and iexpedition, needs the polyclonal (-p) of mc2bsbh to run. I'm not sure if Raytech doesnt. Raytech needs BSB, and
Raytech for me needs CHK , depending on Ty, (base, insert, extension) and is fussy about how the files are filed. Easier to use Opencpn.
Thanks for the info!

I am at the point where Raytech imports the chart (bsb+kap file), shows the chart in the list, shows the outline of the chart at the correct position, but no picture within the outline-frame. I tried 16, 64, 127 and 128 colors, RGB as well as indexed, no difference.

Since you have been down that road before, could you upload a homemade kap + bsb which does work with Raytech? From there I might be able to see what I am doing wrong.

"polyclonal (-p) of mc2bsbh"
I tried (mc2bsbh v0.9), it does not complain about the -p switch, but the header is identical to one created without -p switch. Could you check on that?

OpenCPN: that is what I navigate with, RayTech is only for Radar + Overlay....
__________________
LeaseOnLife is online now   Reply With Quote
Old 13-07-2011, 05:28   #201
Registered User
 
Netsurfer's Avatar

Join Date: Jan 2010
Location: Cologne, Germany
Boat: Beneteau Oceanis 331
Posts: 557
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by LeaseOnLife View Post
I tried (mc2bsbh v0.9), it does not complain about the -p switch, but the header is identical to one created without -p switch.
If we know what we need to end up with the header it should be relatively easy to write a small script to perform the needed action.

That's what I propose anyway:
Let's do it e.g. by a web form. You enter the needed data (or choose from drop-down lists where possible, checkboxes, radio buttons etc.) and get the final header as return.

I think this way round we are much more flexible as when using some of the existing helper tools which do not perfectly suit our needs.

Gunther
__________________
Deutschsprachige Community- und Support-Website unter OpenCPN.de
Netsurfer is offline   Reply With Quote
Old 13-07-2011, 05:37   #202
Registered User
 
LeaseOnLife's Avatar

Join Date: Apr 2008
Location: out cruising again
Boat: Sailboat
Posts: 1,046
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by Netsurfer View Post
That's what I propose anyway:
Let's do it e.g. by a web form. You enter the needed data (or choose from drop-down lists where possible, checkboxes, radio buttons etc.) and get the final header as return.
I support that! For all the chart info, the first part of the header, with as many fields pre-filled, from the NGA/NOAA website or the related pdfs.

There are (at least) two more portions of the kap-header. The calibration (REF and PLY..) plus the color definitions (IFM and RGB). Keeping these on the server should then enable us to re-run a kap creation easily, in case we find a mistake/improvement in the headers, or the according image.

I say at least, because I found ARE/ADN sections in the NOAA charts, corresponding with Notice of Mariners, I think
__________________
LeaseOnLife is online now   Reply With Quote
Old 13-07-2011, 05:46   #203
Registered User
 
Netsurfer's Avatar

Join Date: Jan 2010
Location: Cologne, Germany
Boat: Beneteau Oceanis 331
Posts: 557
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by LeaseOnLife View Post
For all the chart info, the first part of the header, with as many fields pre-filled, from the NGA/NOAA website or the related pdfs.

There are (at least) two more portions of the kap-header. The calibration (REF and PLY..) plus the color definitions (IFM and RGB). Keeping these on the server should then enable us to re-run a kap creation easily, in case we find a mistake/improvement in the headers, or the according image.
Yes, that's exactly the idea behind this ...!

I am already gathering every piece of information about each chart in a database.

So if we are going to add the needed information for the header as well (including REFs and PLYs) we could easily update and recreate any kap.

When doing the calibration for the original image size it should be possible to simply re-calculate them for any other image size.

Additionally we will have consistent headers not depending on the used conversion tool.

Gunther
__________________
Deutschsprachige Community- und Support-Website unter OpenCPN.de
Netsurfer is offline   Reply With Quote
Old 13-07-2011, 05:55   #204
Registered User
 
Netsurfer's Avatar

Join Date: Jan 2010
Location: Cologne, Germany
Boat: Beneteau Oceanis 331
Posts: 557
Re: Charts II: NGA - 2700 Charts

Quote:
Originally Posted by LeaseOnLife View Post
Wordpress is great to document things, stable processes, etc

What is "worthy information". At times we don't know what we know. A Wiki is a bit more like brain storming. Creating pages with some thoughts, other users adding/correcting info. This is often an unstructured mess of pages. Good info could then be moved in a more structured space, like wordpress.

But can't we have both? Wordpress has plug-ins, and one plugin is
WordPress › WordPress Wiki WordPress Plugins
I had a look at it.
IMHO this makes things unnecessarily complicated as you have to deal with another wiki syntax. And experience showed that there are very few users contributing at all - see opencpn.org and opencpn.de!

Quote:
I see pages also for
  • existing tools (i.e mc2bsbh): command line or guis: usage, quirks, advantages, best practices to install and use the tool
  • explain the content of the hdr (and the .bsb file). Where should info be taken from to fill the hdr. can this be automated?
  • Compatibility Matrix for the different charting programs, document the special needs (i.e RayTech needs a .bsb)
  • the different download and stitch scripts, prerequisites...
  • quality: at what points do we check for quality, and how
  • document test cases
  • and so many more
As I said, unstructured thoughts, sometimes the structure comes after the content. And that is what I like Wikis for.
I agree with your points but I would prefer another solution:
I could easily set up a temporary wordpress blog (like the ones already present at opencpn.info) which we could use for the 'unstructured' things. If things become clearer we could transfer them to the opencpn.info/en/ in a clean structured way.

How about this?

Gunther
__________________
Deutschsprachige Community- und Support-Website unter OpenCPN.de
Netsurfer is offline   Reply With Quote
Old 13-07-2011, 06:47   #205
Registered User

Join Date: Feb 2010
Location: On the go. Not in Prague.
Posts: 4,016
Re: Charts II: NGA - 2700 Charts

Folks...
As the grabbing progresses so nicely with almost 700 jpegs uploaded, I've started to work on the toolchain to produce the charts in a way as automatic as possible.
I've setup an organization on github to host all the involved scripts - The repository is accessible on web at https://github.com/ngacharts/ngacharts
My goal is to be able to do the whole process of image conversion, calibration and distribution at any time on the whole set or just a part of the charts automatically on any machine with the needed tools.

Everybody is welcome to join and host any homebrew tools there. As a free bonus we get a wiki and bugtracker we can use there, so no need to spend time setting up anything temporary anywhere.

So far I started with the pre-processing scripts. Right now it adjusts the levels of the image to make everything that's almost white white and almost black black, resizes to 50% and produces a 32 color palette png.

That's just an initial take and any input on this matter is welcome. I have to say I was positively surprised with the quality and processing time - From 35000, being 89 MB, the script produces 5.9 MB png in 4 minutes on my server machine.
The goal is to eliminate any manual labor, so only tools scriptable from the command line should be used. The solution is currently totally Unix-centric.

The next step is to invent a way to do the calibration and QA - I like the idea of an online "generator" for the headers - certainly can reflect the latest acquired knowledge faster than any other tools. The resulting headers I would really like to store on github though - more on that later when I'm done with some more workflow thinking and documentation.

Gunther, could you please provide me with the list of info you are currently gathering about the individual charts, so I can count with what can be automated and what not?

Pavel
__________________
nohal is offline   Reply With Quote
Old 13-07-2011, 08:05   #206
Registered User
 
Netsurfer's Avatar

Join Date: Jan 2010
Location: Cologne, Germany
Boat: Beneteau Oceanis 331
Posts: 557
Re: Charts II: NGA - 2700 Charts

Pavel

just some thoughts ...
Quote:
Originally Posted by nohal View Post
Everybody is welcome to join and host any homebrew tools there. As a free bonus we get a wiki and bugtracker we can use there, so no need to spend time setting up anything temporary anywhere.
Fine ..., so let's start using it.

Quote:
So far I started with the pre-processing scripts. Right now it adjusts the levels of the image to make everything that's almost white white and almost black black, resizes to 50% and produces a 32 color palette png.
About the colors: I have already seen charts where the land area is yellow (e.g. chart 700). And on many other charts the land area is grey. Or is this eliminated by your noise reduction scripts?
I also do not understand why you are creating a 32 color image?
When converting to a palette image format there will be a maximum of 256 colors. Also it is possible to get the number of used colors. Wouldn't it be more efficient to get this value first and afterwards producing an image with the "optimal" color count (assumed it does not exceed our 127/128 max)?

As the charts are scanned from paper charts some of them are also slightly skewed/crooked.

Quote:
The next step is to invent a way to do the calibration and QA
What does 'QA' means (quality assurance)?
Hmmm ..., as much as I like the idea of automation I couldn't think of any other way as doing the calibration manually/ by hand. But maybe you have an idea ...

Quote:
Gunther, could you please provide me with the list of info you are currently gathering about the individual charts, so I can count with what can be automated and what not?
I sent you an email.

Gunther

Addendum: And there are also charts with harbour plans or several parts of a river etc. (good examples are 37235 and 37246 / upload coming soon)!
__________________
Deutschsprachige Community- und Support-Website unter OpenCPN.de
Netsurfer is offline   Reply With Quote
Old 13-07-2011, 08:27   #207
Registered User

Join Date: Dec 2005
Location: WNA
Boat: Dufour 35
Posts: 3,247
Re: Charts II: NGA - 2700 Charts

Gunther/Pavel

Some info just have to be retrieved manually from he chart picture, just once hopefully.

GD=Geodetic Datum. Look at this:on the tracker
PR=Projection
PP Latitude where PR is valid. Mid long if tmerc.
UN Sounding units. Perhaps we can assume "Meters" unless the chart is very old.
SD Sounding Datum
DTM Corrections to adjust the charts to WGS84. Available on a few charts

REF and PLY
We should get the exact corner lat&long, from the printed information in the SW and NE corners of the charts.
We also need the corresponding pixels to a very good accuracy. I zoom in at 200% to sort the pixels out.
For most charts the 4 corner will do, but not for all, there are a few crap shots around!! Some charts - mainly inserts, needs more PLY points as the area is not rectangular.

Using the 4 corners has an other advantage. QA at a glance, the PLY line should follow the inner border very accurately.
The quality should be tested further by zooming in and having the mouse pointer over a printed lat/long crossing, and read the values. How accurate is acceptable? I suggest that up to and including the 4:th decimal should be correct with normal rounding off rules. Repeat for ~4 similar points spread out. We mustn't compromise on this, and produce a chart as accurate as possible.

Another thing is that if we make the picture a different size, only the pixel count for the corners change. There will never be any doubt as to exactly which reference points that were used originally.
I also suggest that we follow a "semi-standard" and start REF and PLY in the SW corner and proceed clockwise.

Quite a few charts have inserts, named xxxxxxA etc. For these, the corner positions, often more than 4 and the corresponding pixel count must be retrieved, as well as the Name and scale for the inserts.
The name and scales are available in the pdf catalogs for each region, so hopefully this can be automated as well.

We also have to make up our minds how to handle those chart pictures that are turned 90 degrees. In the end its just a question of the pixels, nothing else changes.

IFM and RGB etc is not our problem, imgkap/libbsb handles that.
The only thing here, is if we should use imgkap to produce a dusk and night palette, but I suggest that we don't, as OpenCPN and many other plotters takes care of dimming the whole screen.

Did I miss something?

Thomas
__________________
cagney is offline   Reply With Quote
Old 13-07-2011, 08:45   #208
Registered User

Join Date: Dec 2005
Location: WNA
Boat: Dufour 35
Posts: 3,247
Windows memory problem.

To speed up the downloading I'm using an XP box as well, with ~3GB memory.
I use the python script to try to semi-automate the down loading.
There is a problem with Windows memory handling. After 1 or 2 downloads the script stops, with the message that the next picture is to big to fit in memory. There is no problem just starting the script directly again, however.
Has anyone else had this problem?

Thomas
__________________
cagney is offline   Reply With Quote
Old 13-07-2011, 08:51   #209
Registered User
 
Surveyor's Avatar

Join Date: Mar 2010
Location: Ft. Pierce, FL - Bislig, Philippines - Gladstone QLD
Boat: 1968 Alberg 30 #329
Posts: 451
Re: Windows memory problem.

Quote:
Originally Posted by cagney View Post
To speed up the downloading I'm using an XP box as well, with ~3GB memory.
I use the python script to try to semi-automate the down loading.
There is a problem with Windows memory handling. After 1 or 2 downloads the script stops, with the message that the next picture is to big to fit in memory. There is no problem just starting the script directly again, however.
Has anyone else had this problem?

Thomas
Yes. It is very frustrating. I asked this question on the author's talk page:

"I am using your batch script and love it! Except one thing which is this: I am on a machine with a total possible ram of 2gig. I use FreeRAM XP Pro set to clear the ram when it gets below a certain number. This works great on slow machines, the ram gets cleared before your script knows there isnt enough and continues 8 times out of 10. On fast machines the script is too fast and starts to process the next url before the memory manager can clear the ram and the script stops. I am wondering if you can insert a 5 second pause between url processes in order for the ram to catch up. Is this possible? Thanks in advance.

It's easy to do yourself. Add "time.sleep(5)" under "for row in range(self.yTiles):" at line 313. Inductiveload (talk) 00:31, 9 July 2011 (UTC)"


I have not tried it yet but this may help. If nothing else have it pause for 30 seconds, clear ram manually then let it run.

EDIT: I would like to edit out the 100% are you sure question too.
__________________
Jim - Bahala na!
http://svcookie.blogspot.com/
KK4GGF
Surveyor is offline   Reply With Quote
Old 13-07-2011, 09:16   #210
Registered User

Join Date: Dec 2005
Location: WNA
Boat: Dufour 35
Posts: 3,247
Re: Charts II: NGA - 2700 Charts

Thanks Jim.
I'm testing the version below right now.
dezoomify.py.doc
Thomas
__________________

__________________
cagney is offline   Reply With Quote
Reply

Tags
charts

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off




Copyright 2002- Social Knowledge, LLC All Rights Reserved.

All times are GMT -7. The time now is 15:57.


Google+
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Social Knowledge Networks
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.

ShowCase vBulletin Plugins by Drive Thru Online, Inc.