Charts II: NGA - 2700 Charts

[Netsurfer]

Thomas

Originally Posted by cagney

REF and PLY
We should get the exact corner lat&long, from the printed information in the SW and NE corners of the charts.

Yes, and from this we can simply calculate the NW and SE points, at least for the charts where the two corner coordinates are given (not for all charts).

We also need the corresponding pixels to a very good accuracy. I zoom in at 200% to sort the pixels out.
For most charts the 4 corner will do, but not for all, there are a few crap shots around!! Some charts - mainly inserts, needs more PLY points as the area is not rectangular.

Using the 4 corners has an other advantage. QA at a glance, the PLY line should follow the inner border very accurately.

For rectangular charts this is very simple. I propose to calculate the PLYs by setting them a little bit smaller (about 5-10 pixel) than the outer edges of the chart. The lat/lng values could be simply calculated.

The quality should be tested further by zooming in and having the mouse pointer over a printed lat/long crossing, and read the values. How accurate is acceptable? I suggest that up to and including the 4:th decimal should be correct with normal rounding off rules. Repeat for ~4 similar points spread out. We mustn't compromise on this, and produce a chart as accurate as possible.

When the SW and NE coordinates are given why do you want to round the decimal degree value?

Another thing is that if we make the picture a different size, only the pixel count for the corners change. There will never be any doubt as to exactly which reference points that were used originally.
I also suggest that we follow a "semi-standard" and start REF and PLY in the SW corner and proceed clockwise.

Hehe ..., I suggested several times before to use them counter-clockwise.
But anyway, whether clockwise or counter-clockwise doesn't matter in the end as long as point follows the other for the rectangle/ polygon.

Quite a few charts have inserts, named xxxxxxA etc. For these, the corner positions, often more than 4 and the corresponding pixel count must be retrieved, as well as the Name and scale for the inserts.
The name and scales are available in the pdf catalogs for each region, so hopefully this can be automated as well.

I don't think so ...!
Though you might convert the pdf to another format which you are able to extract the information from as text it would be still difficult to set them up to the correct chart.
In generall: How to deal with such charts?
In my opinion we

• use the original image for the "main" chart with the PLYs set correctly
• have to copy rectangular images from the original one for each plan
  these ones could be named something like 37235_plan_[a-z].jpg
• calibrate and set PLYs for them

In my understanding there is no other way as to make a single chart (kap) out of each plan integrated in a chart.

We also have to make up our minds how to handle those chart pictures that are turned 90 degrees. In the end its just a question of the pixels, nothing else changes.

Why not simply turning them before any other processing? This is possible without any loss of quality to the original JPG.

For the charts where the SW and NE corners are not given:
The attached image shows the upper left corner of a chart. As you can see in such cases it won't make much sense to use the edge as you have to guess the right lat/lng values.
And about the accuracy: The more you zoom in the more pixel are used for a line. So it's the small pink '+' sign in the middle which you should use for the pixel coordinates.

Gunther

[cagney]

Gunther
See comments below.

Originally Posted by Netsurfer

Thomas

Yes, and from this we can simply calculate the NW and SE points, at least for the charts where the two corner coordinates are given (not for all charts).



For rectangular charts this is very simple. I propose to calculate the PLYs by setting them a little bit smaller (about 5-10 pixel) than the outer edges of the chart. The lat/lng values could be simply calculated.

When the SW and NE coordinates are given why do you want to round the decimal degree value?

Hehe ..., I suggested several times before to use them counter-clockwise.
But anyway, whether clockwise or counter-clockwise doesn't matter in the end as long as point follows the other for the rectangle/ polygon.

I don't think so ...!
Though you might convert the pdf to another format which you are able to extract the information from as text it would be still difficult to set them up to the correct chart.

In generall: How to deal with such charts?
In my opinion we

• use the original image for the "main" chart with the PLYs set correctly
• have to copy rectangular images from the original one for each plan
  these ones could be named something like 37235_plan_[a-z].jpg
• calibrate and set PLYs for them

In my understanding there is no other way as to make a single chart (kap) out of each plan integrated in a chart.

Why not simply turning them before any other processing? This is possible without any loss of quality to the original JPG.

For the charts where the SW and NE corners are not given:
The attached image shows the upper left corner of a chart. As you can see in such cases it won't make much sense to use the edge as you have to guess the right lat/lng values.

And about the accuracy: The more you zoom in the more pixel are used for a line. So it's the small pink '+' sign in the middle which you should use for the pixel coordinates.

Gunther

Thomas

[nohal]

Originally Posted by Netsurfer

About the colors: I have already seen charts where the land area is yellow (e.g. chart 700). And on many other charts the land area is grey. Or is this eliminated by your noise reduction scripts?

Not at all and I don't think it should be doing anything about it (although it of course could be handled once done manually for the first time)

I also do not understand why you are creating a 32 color image?
When converting to a palette image format there will be a maximum of 256 colors. Also it is possible to get the number of used colors. Wouldn't it be more efficient to get this value first and afterwards producing an image with the "optimal" color count (assumed it does not exceed our 127/128 max)?

It's an experimental value resulting in a pretty good combination of quality and size. The charts are black, white, blue and gray in most cases, it means the palette ideally would be even smaller (even if you count the antialiasing half-tones) But as it comes from a scan, it's non-trivial to achieve. No dogma here, our experimenting has to come up with the ideal settings.

As the charts are scanned from paper charts some of them are also slightly skewed/crooked.

Of course some charts need special "treatment" Not much can be done automatically, but the workflow I'm working on takes into account that for some charts there will have to be special scripts handling them. That's why not everything can be blindly generated from the database. An that's why we will need to store stuff in git as well.

What does 'QA' means (quality assurance)?
Hmmm ..., as much as I like the idea of automation I couldn't think of any other way as doing the calibration manually/ by hand. But maybe you have an idea ...

Yep, for that we need people's eyes and brains. Same like for identifiing the charts demanding special treatment.

Addendum: And there are also charts with harbour plans or several parts of a river etc. (good examples are 37235 and 37246 / upload coming soon)!

Same as above. But same as above, once we know how the chart is composed, we can script a machine to do it next time we need it.

Pavel

[cagney]

Originally Posted by cagney

Thanks Jim.
I'm testing the version below right now.
Attachment 29596
Thomas

No that wasn't right!!
Use this version instead, has run 3 in a row by now.
dezoomify2.py.doc

Thomas

[LeaseOnLife]

Originally Posted by Netsurfer

For rectangular charts this is very simple. I propose to calculate the PLYs by setting them a little bit smaller (about 5-10 pixel) than the outer edges of the chart. The lat/lng values could be simply calculated.

I don't think so, maybe I misunderstand. During my calibration I found the charts are not scanned perfectly, lines are not perfect horizontal or vertical. Don't calculate REF points, only humans can define a REF point. And YES, if possible we should use the corners.

If the chart is rectangular, and all 4 corners have been used for REF, then surely we can re-use them for PLY.

About inset charts:

Originally Posted by Netsurfer

In generall: How to deal with such charts?
In my opinion we

• use the original image for the "main" chart with the PLYs set correctly
• have to copy rectangular images from the original one for each plan
  these ones could be named something like 37235_plan_[a-z].jpg
• calibrate and set PLYs for them

In my understanding there is no other way as to make a single chart (kap) out of each plan integrated in a chart.

Makes sense to keep the complete size chart for each inset chart, otherwise the splitting and cropping will be done differently every time and REF/PLY can't be re-used. To make it repeatable, keep the full rectangular chart, white-out the areas not used, as suggested by Thomas. Image compression should take care of the white space and the file should not be that much bigger than a cropped chart.

One more thought: Should we perform and save the calibration (REF & PLY) for the full size chart, before resizing? When resizing, the script could recalculate the pixels from there? That would allow us to resize to any smaller size without loosing accuracy. Caveat: MapCal on my netbook scrambles to work on xx-MByte charts.

[nohal]

Originally Posted by LeaseOnLife

One more thought: Should we perform and save the calibration (REF & PLY) for the full size chart, before resizing? When resizing, the script could recalculate the pixels from there? That would allow us to resize to any smaller size without loosing accuracy. Caveat: MapCal on my netbook scrambles to work on xx-MByte charts.

VERY good question. Of course it would be the best and most accurate, but I really fear that for practical reasons we will have to calibrate the preprocessed 50% charts (if 50% is the size we decide is the one we will use for our charts).

Pavel

[Netsurfer]

Pavel ...

Originally Posted by nohal

VERY good question. Of course it would be the best and most accurate, but I really fear that for practical reasons we will have to calibrate the preprocessed 50% charts (if 50% is the size we decide is the one we will use for our charts).

What reasons do you mean by "practical reasons"?
Maybe it would be possible to just copy small parts of the edges from the fullsize image for the calibration?

Gunther

[LeaseOnLife]

Originally Posted by Netsurfer

Maybe it would be possible to just copy small parts of the edges from the fullsize image for the calibration?

How do I get the correct pixel count for a REF point on an image which doesn't have all pixels from X0/Y0 ?

Different question:
Would a cal tool via the web be too hard to code, too hard for the server to handle the image processing?

[nohal]

Originally Posted by Netsurfer

Pavel ...

What reasons do you mean by "practical reasons"?
Maybe it would be possible to just copy small parts of the edges from the fullsize image for the calibration?

Gunther

I don't know how on your machines, but on my core i7 wit 3 gigs of ram, even opening any of the full size images takes ages in anything but Picasa.
Also Mapcal does not support JPG, so you would have to convert the fullsize image to PNG or BMP which again will take tons of time.
All this and more seems pretty impractical to me Of course like always, if there's a good reason not to go with scaled down version, I will be easily convinced.

Pavel

[Netsurfer]

Originally Posted by LeaseOnLife

How do I get the correct pixel count for a REF point on an image which doesn't have all pixels from X0/Y0 ?

If you know (and you know) the original size and the coordinates of the part you copy than you know the pixel coordinates for the original by any given coordinates of the copied one.

Different question:
Would a cal tool via the web be too hard to code, too hard for the server to handle the image processing?

What would be the advantage of doing so?
All you need (for the "normal" rectangular charts with given position for the SW and NE corners) is a graphic program where you can get the according pixel coordinates from.

Gunther

[Netsurfer]

Originally Posted by nohal

I don't know how on your machines, but on my core i7 wit 3 gigs of ram, even opening any of the full size images takes ages in anything but Picasa.
Also Mapcal does not support JPG, so you would have to convert the fullsize image to PNG or BMP which again will take tons of time.
All this and more seems pretty impractical to me Of course like always, if there's a good reason not to go with scaled down version, I will be easily convinced.

First of all I don't see why we should use MapCal.
Second about the speed: Have you tried to use the GD Lib with PHP?
I am using it to stitch the downloaded tiles. It takes less than a minute to stitch 4000 tiles with 200 MB. I guess (though not tried yet) it should be able to copy the 4 corners (let's say each 512 x 512 px) pretty fast.

Gunther

[nohal]

Originally Posted by Netsurfer

First of all I don't see why we should use MapCal.
Second about the speed: Have you tried to use the GD Lib with PHP?
I am using it to stitch the downloaded tiles. It takes less than a minute to stitch 4000 tiles with 200 MB. I guess (though not tried yet) it should be able to copy the 4 corners (let's say each 512 x 512 px) pretty fast.
Gunther

Gunther,
there's no problem to cut the 4 corners with 4 imagemagick calls taking no time of course. The problem is that this is a perfect approach just for a "perfect" chart. I really don't know what's the percentage of such charts in the portfolio and how bad are the others. I need to see numbers and do some trials to be able to make qualified decisions, the thoughts and feelings are not enough.
No need to use Mapcal if we have something better people can intuitively use. Right now we don't. Once the online header generator is ready with the appropriate instructions to use it, we have no problem.

Pavel

[Netsurfer]

Pavel

Originally Posted by nohal

The problem is that this is a perfect approach just for a "perfect" chart. I really don't know what's the percentage of such charts in the portfolio and how bad are the others. I need to see numbers and do some trials to be able to make qualified decisions, the thoughts and feelings are not enough.

That's the point. Because I got the impression that you are already suggesting other methods ... "I really fear that for practical reasons we will have to calibrate the preprocessed 50% charts"!

No need to use Mapcal if we have something better people can intuitively use. Right now we don't. Once the online header generator is ready with the appropriate instructions to use it, we have no problem.

Sorry, but what does any header generating tool has to do with the calibration?
First we need a method to get at least 2 (better 4) REF points. That means we need the pixel coordinates and the according lat/lng values from each chart image. The most accurate way of doing this would be to use the fullsize image. So before thinking about any other way we should have a look if there will be a practical solution for this. And only if not, then we have to look for alternatives. That's my opinion.

Gunther

[nohal]

Originally Posted by Netsurfer

Pavel

That's the point. Because I got the impression that you are already suggesting other methods ... "I really fear that for practical reasons we will have to calibrate the preprocessed 50% charts"!

Sorry, but what does any header generating tool has to do with the calibration?
First we need a method to get at least 2 (better 4) REF points. That means we need the pixel coordinates and the according lat/lng values from each chart image. The most accurate way of doing this would be to use the fullsize image. So before thinking about any other way we should have a look if there will be a practical solution for this. And only if not, then we have to look for alternatives. That's my opinion.

Gunther

Gunther,
things without context look so nice and easy
By the header generation tool I of course meant something, that (at least) stores the input REFs & PLYs and does at least a formal check of the input etc.
All this is a natural part of Mapcal and we have currently no replacement for that (apart from a totally manual approach). Not talking about Mapcal's intuitivity for anybody used to use it, which is of course questionable.
With Mapcal, using the fullsize images is almost impossible though. That's a fact. There's a question: Is the tradeoff in accuracy if we use scaled down images with Mapcal acceptable or not? I really don't know as I didn't try yet. We are just discussing it to find the best way ahead.

Pavel

[bcn]

As proposed by THomas test of transformation/size reduction starting with jpg-->png first
All in GIMP. First value with png quality 5/ second with quality 0 (lower is better)

Original 100% 53093.jpg 13.020kB

PNG 100% 21.323kB 375.151kB

PNG 50% px 6.676kB 93.793kB

PNG 32colors 1.326kB 31.269kB

The result in that workflow at a zoom of 200% is slightly better when doing reductions in jpg. Very subtle differences. The improvement between quality 5 and 0 at 200% is only visible with imagination.
No filters applied.

Hubert