Charts II: NGA - 2700 Charts

[bcn]

Notice to calibrators:

we are about half through the insets now.
Keep on going with the pace!

Hubert

[bcn]

We forecast to finish the "punching out inset" phase this weekend.

We will end up with aprox. 1750 chartlets and panels.

Pavel, would it be possible to run a check the charts for completeness of the datum infos? I've seen quite a lot of the composed charts that were incomplete.

Hubert

[nohal]

Hubert...
Yes, I will definitely run such a check and also probably do some "normalization" to the datum names we invented during the base chart calibration before copying the data to the insets as the default values.

Pavel

[wdkester]

We need to be careful, in the next phase, of insets on the same chart having different data. 25610 has one inset in fathoms and one in feet.

[bcn]

Absolutely right - especially in the case of old charts there are complete sets without datum - neither soundings nor projection. And a lot of single insets from groups that are without complete datum information.

Hubert

[bcn]

Just to report that we have finished the first phase for the insets, identifying all correctly (hopefully) with their corresponding rectangular border lines.

In total we have now 1698 insets processed that are waiting for the next steps - geo-referencing and exact outlining for quilting.

Not all of the insets will be usable but our estimate is that we might end up with about 4.000 charts and chartlets from the NGA collection when finished sometime away from today.

There are regions of course that already covered by other projects (and with an excellent quality) like New Zealand

Hubert

[P_Dub]

Excellent work bcn, I am very thankful for those of you that did it.
Best Regards.
P_Dub

[manimaul]

Originally Posted by bcn

Not all of the insets will be usable but our estimate is that we might end up with about 4.000 charts and chartlets from the NGA collection when finished sometime away from today.

There are regions of course that already covered by other projects (and with an excellent quality) like New Zealand

Hubert

After an r-sync with the server, I'm showing 2,514 .kap files. The status information on opencpn.info is a bit cryptic to me. Is there a simple status page somewhere? I'm also curious what kind of load the machine is under that is doing the processing.

And, how can we help going forward?

Will

[nohal]

Will...
You can find some stats at NGA Charts project stats or just pull the database from github and you have all the data daily to do whatever processing you want...
Complete processing of one chart takes up to ca. 5 minutes on a single Xeon X3440 with 8 gigs of RAM (depends on the size of the chart, the need for rotation and the load on the machine as the process is reniced to 19, so everything else has precedence)

The next step is to modify the calibration page to do the insets - The calibrators were as usually much faster doing their job than I can find the time to write the code It's probably not very realistic to ask for help with this page, but there are two tasks that need to be done and are separate that come to my mind:
1) Tooling to pull the updates from the NGA site - unfortunately it's not as easy as parsing the table on say Region 1 - North America NGA Nautical Charts - NGA as far as we've seen often just uploads new images to the subfolders of Index of /NGAViewer without changing the catalogs.
2) Release tooling - scripts to package the whole release, by region and prepare the .torrent files for them + the same for increment since the last complete release - of less interest for you

Anyway I will try to have the calibration page by the end of the week, but it really depends on the workload I will receive from my real work, which is especially in the last weeks totally unpredictable as the managers have obtained new toys drawing some graphs.

Pavel

[bcn]

Everybody should also be aware that the calibration process for the insets will most probably slower than for the main charts:
- only a part of the panels and charlets carry coordinates in their corners
- others have not even any grid on it
Means that referencing will be much more individual and manual and in not few cases most probably we will have to decide not to include those.

And we should remember too that in the case of the main charts only a minority has the special PLY's applied to remove legends and texts from the quilted areas.

With respect of packaging charts for "delivery" the ideal would be a mechanism to select (sub)regions. What about a wildcard scheme? 54xxx would give you the set 54000-54999 for instance.

Hubert

[manimaul]

Originally Posted by nohal

1) Tooling to pull the updates from the NGA site - unfortunately it's not as easy as parsing the table on say Region 1 - North America NGA Nautical Charts - NGA as far as we've seen often just uploads new images to the subfolders of Index of /NGAViewer without changing the catalogs.

Simple solution perhaps: The root directory listing has a modification date next to each chart directory. Parse the date string out to an epoch and cache. Also, It looks like all of the charts were updated in May and June of 2011. If this is going to be a regular thing, perhaps an update every July would work.

Will

[nohal]

Will...
Unfortunately we have to go even further - check the dates on the xml files in the respective directories. But my problem is not about not knowing what and how to do - it's more about when someone will have time to do it

Pavel

[bcn]

Applying for Google Summer of Code?

[z.stadler]

Pavel,
Are you aware that for at least some of the rotated charts, such as chart 24270, the inset shape was defined on the original chart rather than on the north-up chart!?
Zeev

[nohal]

Zeev...
Yep, I am, on the list

Pavel