Beta Test / Technical

[Hakan]

Dave..
PS - PaEC now also successfully tested on a W-XP machine - DS

[cagney]

Originally Posted by bdbcat

cagney/hakan...

New code in github, this time with lots of log output to track the SENC build decision tree. Thomas, I hope you may be able to see what is happening as you rebuild SENCs with updates.

I will try the ENC Netherlands set, too.

Thanks
Dave

Dave
Repeating test with renamed updates etc using this mornings git master.
Both PaEC took about 8 min. Afterwards O is still processing charts as I pan around.
Excerpts from the log attached.
PaEC-log.txt.pdf
/Thomas

[bdbcat]

Hakan...

Are you able to PaEC for the ENC Netherland set now? I tried it with cm93 on-screen, as you reported. No problem with current github. Took about 4 minutes. Some warnings in the log, no errors.

Dave

[bdbcat]

Thomas...

Still some trouble here. It is not picking up the updates correctly. A logfile entry for a cell with updates should look like this:

20:22:19 EDT: Preparing to apply ENC updates, target final update is   12.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.001.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.002.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.003.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.004.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.005.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.006.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.007.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.008.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.009.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.010.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.011.
20:22:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US5MD21M.012.

I will investigate on a virtual 64 box.

Dave

[Hakan]

Originally Posted by bdbcat

Hakan...

Are you able to PaEC for the ENC Netherland set now? I tried it with cm93 on-screen, as you reported. No problem with current github. Took about 4 minutes. Some warnings in the log, no errors.

Dave

My report here: http://www.cruisersforum.com/forums/...ml#post2221603 was for all my ENCs NL included and CM93 on screen. And resent Github

A "Funny" observation: If I run PaEC with raster charts on screen O switch to vector, CM93, after finalized PaEC.

Håkan

[bdbcat]

cagney...

New code in github correcting missing log messages. I don't think this actually fixes the problem, though.

Anyway, more log to read and think about.
After you do PaEC once, then again, the log should be instructive.
Something like this, maybe?

00:28:19 EDT: S57chart::Checking SENC file: /home/dsr/.opencpn/SENC/US2EC03M.S57
00:28:19 EDT:     Rebuilding SENC due to incremental cell update.
00:28:19 EDT: last update recorded in SENC: 9   most recent update file: 10
00:28:19 EDT: Preparing to apply ENC updates, target final update is  10.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.001.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.002.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.003.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.004.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.005.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.006.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.007.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.008.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.009.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.010.

Dave

[Hakan]

Dave
Observations:
1
When I run PaEC a second time the log say:
"07:51:54: ParseAllENC() count = 743"
But there are 1466 files after the first run?

2
For the BE_INLAND files a lot of warnings appear in the log, like this chart:

08:01:06: S57chart::Checking SENC file: C:\ProgramData\opencpn\SENC\7W7GENO3.S57
08:01:06:     Rebuilding SENC due to missing SENC file.
08:01:07:    Warning 1: For feature "SMCFAC", attribute "DATSTA" ignored, not in expected schema.

08:01:07:    Warning 1: Illegal feature attribute id (ATTF:ATTL[3]) of 17106
on feature FIDN=1437142426, FIDS=212.
Skipping attribute, no more warnings will be issued.
08:01:07:    ENC contains no useable M_COVR, CATCOV=1 features:  C:\Users\admin\Documents\Charts\ENC\BE_INLAND\Kanaal Gent-Oostende\7W7GENO3.000
08:01:07:    ENC contains no M_COVR features:  C:\Users\admin\Documents\Charts\ENC\BE_INLAND\Kanaal Gent-Oostende\7W7GENO3.000
08:01:07:    Calculating Chart Extents as fallback.
08:01:09: S57chart::Checking SENC file: C:\ProgramData\opencpn\SENC\7W5BRUS1.S57

Are these warnings due to mistake from those who made the charts or due to our code?
You could download and check the Belgium inland charts if you like.
Håkan

[cagney]

Originally Posted by bdbcat

cagney...

New code in github correcting missing log messages. I don't think this actually fixes the problem, though.

Anyway, more log to read and think about.
After you do PaEC once, then again, the log should be instructive.
Something like this, maybe?

00:28:19 EDT: S57chart::Checking SENC file: /home/dsr/.opencpn/SENC/US2EC03M.S57
00:28:19 EDT:     Rebuilding SENC due to incremental cell update.
00:28:19 EDT: last update recorded in SENC: 9   most recent update file: 10
00:28:19 EDT: Preparing to apply ENC updates, target final update is  10.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.001.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.002.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.003.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.004.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.005.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.006.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.007.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.008.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.009.
00:28:19 EDT:  S57: Applying feature updates from /home/dsr/.opencpn/SENC/US2EC03M.010.

Dave

Dave
I've done the same thing again, using this mornings code.
Both runs were almost identical and took about 8 min each.

See attached file. Note that the first run ends and the second run starts at line 1155.

PaEC-logg2.txt.pdf

Thanks!
/Thomas

[bdbcat]

cagney...

Once more, please, using this mornings github. More log messages.

We'll get it....

Thanks
Dave

[cagney]

Originally Posted by bdbcat

cagney...

Once more, please, using this mornings github. More log messages.

We'll get it....

Thanks
Dave

Dave

Sure you will get it, and I don't mind, while waiting for "Invest 97L" (Matthew) in Port Egmont.

Very fast run, only 8 sec, but nothing much happened.
The log only contained these 2 lines for each run

13:29:33 AST: ParseAllENC() count = 516
13:29:40 AST: Chart cache purge

O crashed a couple of times, at least once it was when pressing PaEC the second time.




/Thomas

[bdbcat]

cagney...

I see another bug. But you can work around it temporarily by making sure you display some (any) ENC or cm93 chart before pressing PaEC.

Thanks
Dave

[cagney]

Dave

Another go. Both runs took about 8 min.
Second run starts at line 1303.

PaEC-logg3.txt.pdf

/Thomas

[bdbcat]

cagney...

.. the Aha moment arrives...

The trouble with repeated PaEC is that you have duplicates of some cells (in different directories), and those duplicates have different edition/update specifications. So what happens is that these two duplicates "compete" with each other for who gets to create and own the common SENC file. And the SENC file is built over and over, flopping from one cell ownership to the other.

I implemented some code to stop the war, but I'm not sure how we want to handle this case.

Duplicate files in the same directory tree are pruned during addition to the database. But in your case, and mine too, the duplicates are in different trees as specified in options->charts->Chart Files. So they are not pruned. You will see them as duplicate buttons on the chart bar, at certain geographic locations. I think this will be a common situation when using the chart downloader and arranging ENCs by region. The same small scale chart (e.g.US2EC03) will appear in multiple regions. But the two copies of US2EC03 may not have the same edition/update numbers.

My solution is to create the SENC for the latest, most modern, of the two charts, and assign it to both cells. This is a little inconsistent, but at least ensures that the "best" chart will always be chosen.

Maybe we need to implement a "global" pruning step, to absolutely eliminate duplicates? But I recall some discussion where certain chart group architectures wanted to retain duplicates. Ring any bells?

Another alternate solution would be to tie the SENC to the originating cell more closely (by some name hash), so that there may be duplicate SENCs, and not just the single SENC allowed by the current model.

This may require more thought/discussion.

Comments, anyone?

Anyway, new code in github now. Should run fast on repeated PaEC, and not create SENCs as you pan around.

Thanks for your patience.
Dave

[cagney]

Dave

Aha ...good catch

Latest test. Git master from earlier this morning.
As usual I have renamed the last update for three charts in the first round, and restored them for the second run.
It is much quicker now. PaEC now takes about 3.5 minutes.There still seems to be a bit of flip-flopping going on with a few files.It does create a few SENCs as I pan around.
PaEC-logg4.txt.pdf

My solution is to create the SENC for the latest, most modern, of the two charts, and assign it to both cells. This is a little inconsistent, but at least ensures that the "best" chart will always be chosen.

I think that this is good enough for now. Test it in the upcoming beta series.
Please mention this issue in the notes for the first beta.

Maybe we need to implement a "global" pruning step, to absolutely eliminate duplicates? But I recall some discussion where certain chart group architectures wanted to retain duplicates. Ring any bells?

Very weak bells ...

Another alternate solution would be to tie the SENC to the originating cell more closely (by some name hash), so that there may be duplicate SENCs, and not just the single SENC allowed by the current model.

This may be the best solution, but is it necessary?

Thanks

Thomas

[transmitterdan]

Originally Posted by cagney

I think that this is good enough for now. Test it in the upcoming beta series.
Please mention this issue in the notes for the first beta.


Very weak bells ...


This may be the best solution, but is it necessary?

Thanks

Thomas

My opinion has evolved on this question. Originally I though all duplicate source chart files would in general be identical. But it appears they are not in all cases.

So it seems we could use either a hashed file name approach or just retain the same directory structure within SENC that the chart folders have (but there are degenerate cases where the latter will fail). We have many cases where there are chart files with duplicate names within various chart file trees. The SENC folder tries to "flatten" this out into a single folder and we end up with the challenge of one SENC file standing in for multiple chart files of the same name.

With a hashed file name the original source file name (minus the full path name) is embedded in the S57 file so it's sort of possible to cross reference the hashed file back to it's source. I worry that this loss of source location will cause problems/confusion even with a hash solution.

I agree that this issue needs some collective contemplation. But it seems to me a bad practice to have cached data that can't be unambiguously linked to the parent data in a bidirectional way. I believe that's a necessary condition to be able to know if each piece of the cache is valid or invalid.