Applying Notices to Mariners (NTMs)

[nohal]

Richard...
These are actually a combination of 3 objects: The buoy, eg. BOYCAN68 (bitmap symbol), a topmark (TOPMAR05 - vector symbol) and a light (LIGHTS13 - vector symbol, rotated) The presentation library in OpenCPN takes care of this. I don't know how you plan to implement the NTMs, so hard to advice what to do...

Pavel

[r.fairman]

I am not sure if you know that NGA charts are available as hard copy from the Noaa agent Oceangraphix. Their FAQ states that before printing the charts are updated to the latest version and NTMs from the Noaa database I'm not sure if this all charts including NGA or just the NOAA charts. I have had a look at a NGA chart in my area that needs NTMs applied and from the image shown it has not been updated. Any thoughts The noaa site also talks about there old relationship with Maptech and describes an update service via email where a digital patch is applied to users charts. Again this could be marketing hype where the patch is just a new version of the existing chart. I principle we need digital patches to update NGA charts to NTMs. Easy to say not so easy to implement.

Richard

[nohal]

Richard...
Updates to KAP charts similar to what's possible for ENC are part of the BSB standard, version 3.
But I really doubt it's useful for us as there's no documentation available and I actually have never seen any chart with updates (I probably have not seen a BSB v3 chart at all) apart from the one included in Maptech's test data. What's pretty sure is that there are probably (at least close to) no chartplotters able to use it (including OpenCPN) in our price range.
The other possibility is, as you say, to distribute binary diffs between the various versions of our BSBv2 charts. Of course - quite some work, but definitely less than the BSBv3 way. What I'm afraid of is the size of such updates - we would have to do some testing to see if it's reasonably doable and worth the effort. What to base it upon is probably clear from the technical point of view - xdelta or librsync
The third option I would still consider is GPX - I know it has a lot of limitations, but is by far the easiest and fastest to implement and most of the code can be reused later if either of the binary options shows feasible.

Pavel

[r.fairman]

Now i'm confused I thought that Opencpn could read the bsb 3 chart format I can certainly open 2 of the charts in the maptech test data namely the new chart edition chart and the full file replacement charts I simply renamed the fie extention to .kap What i cannot open are the patch files which I assume Maptech used to overlay the original chart to apply NTM.s etc. If the patches are simply overlays it still does not answer the question how were the update made to the patches I have a horrible feeling it was done manually. Where i am present is i can generate a gpx from the NTM and i think it is possible to automatically apply the correct Nav Object as the waypoint on the chart What is needed is a method of merging the waypoint layer with the underlying chart In the ideal world we could apply the waypoint from the gpx to a layer over the original chart image and then merge an recalibrate. This could be part of a chart amendment plugin

regards

Richard

[nohal]

Richard...
the BSB format versions (at least up to BSBv3) are from the most part backward compatible - it means the BSBv3 chart still includes all the BSBv2 features - thus OpenCPN displays it "somehow" and ignores the bells and whistles introduced by v3.
I share your horrible feelings about the update workflow used
It should be possible to rob part of the OpenCPN code to do the NTM rendering, so the amount of new code needed to create the updates is not enormous, but still someone has to do it.

Pavel

[sgschwend]

This is all very fascinating!!

Is the idea to place a "WP" on the chart that when selected shows a footnote?
Doesn't the notices come out every Thursday?

[r.fairman]

At the lowest level a waypoint per chart would open a text file of the all NTMs for that chart. I think new ntms are issued every 2 weeks. NOAA has a query faciity whereby all ntms for a chart or a range of charts, can be downloaded as a comma separated text file so the ntm is never more than 2 weeks old The NTM text file should be stored locally so can be read offline.

The next possible level is to generate a gpx file for each chart(s) which is what I am trying to automate. The gpx file will display waypoints on the chart with the correct nav object or text or cover an existing object that has to be deleted The output from NOAA is loadable into excel the main task is to automatically reformat the data in the spread sheet which is mostly cut and paste. a heading row has to be added which has the heading sym,desc,name,latitude,longitude. In the sym column the codes for the nav object have to be added, This is done by searching on the text description of the nav object than inserting the code. The sym code is the same as the xpm image file name stored in the UserIcon directory. (N Pavel kindly exported most of the nav object from cmap 93 in png format which i converted to xpm's.) The excel sheet is then automatically converted to GPX. In an ideal world a user could every 2 weeks auto query the ntms for a range of charts he uses and then open the resulting gpx There may be a serverside method of parsing the ntm text file direct to gpx but that is beyond my limited skills

The highest level of applying ntms would be to merge the displayed nav objects with the underlying kap image file. Some manual drawing may be needed in such cases as submarine cable runs In effect the chart would become a new edition . The user would then only to have to query and generate the gpx ntm since the edition date.

I hope this makes sense. Writing excel macros is a real pain

regards

richard

[sgschwend]

This is what I found at the USCG Navigation Center, week updates, perhaps your source is built from this one?
Local Notice to Mariners - USCG Navigation Center

Also there is a document call the "light list" which runs in a similar updating process. You can see a link to it on the left column.

I know that keep the chart updated is a big requirement for commercial vessels, you idea sounds like it could really help minimize the pencil work (which I understand is the way it is done now). I guess I am assuming that electronic charts could be substituted for paper in the regulatory eyes, not likely the current case but that is where it is headed.

[r.fairman]

Thanks for the reply i query nga ntms at

Maritime Safety Information

If you scroll down the page you can query a range of NTM's
I think NOAA updates its chart for local US waters fairly frequently. This is done for obvious national reasons but i suspect they easily update their vector charts from the NTM's and then use the vectors to update the raster charts by hand and then recalibrate. A majority of the NGA chart are for non US waters. NTM's are frequent but updated charts (new editions) a lot less frequent as i assume there is no national interest and no NGA vestors to update to use as ref to update the NGA rasters.

For purely interest i downloaded a full years ntms for ALL charts It cam to 1.7 MB So all historic ntms for all charts should be in the region of 20MB
Thanks for the feedback

regards

Richard

[PjotrC]

Originally Posted by r.fairman

I think new ntms are issued every 2 weeks.

NTMs are issued every week by most Hydrographic Authorities, NOAA, British Admiralty and others.

I use the waypoint method to annotate electronic charts, but very selectively - e.g. I do not bother to enter the new depth of 200 meters... etc., which of course I do on the paper chart. I just include a short information in the description field, and the NTM number in waypoint name (or track name - if it is a pipeline or a prohibited area). I do it only in preparation for a voyage, mostly with Temporary and Preliminary Notices, which do not go on paper charts in a usable way.

The waypoint method is good, but very often the NTM comes in the form of a chartlet, which is not easy to use if not pasted on the original...

The advantage of mark/waypoint method is that the annotation stays there for all kinds of charts. The disadvantage - that one needs anyhow to manually structure the marks into groups, otherwise they tend to cover the chart too much at low scale. Just try to enter 100 or so of measurement buoys deployed in the Baltic or North Sea...

Interesting project...

Piotr

[r.fairman]

Does anyone have the file names of the existing waypoint icons in OpenCPN. Creating a gpx from the NTM's and finding and replacing (macro) the sym field with the displayed icon name in openpn does not work. Importiing your own icons work if you use the file name. The displayed name for the existing icons does not seem to be the same as the file name

Some progress on the project. Find and replace (macro) in excel will insert the sym field for the correct nav object to be displayed. Linked ntms such as new piers breakwaters or submarine cables can be coded in the gpx to display in opencpn as track lines I am looking to how to put the code in the gpx as a route.

Full application of 40 ntm changes for a single chart now takes 30mins

The 40 changes brings this chart fully uptodate for all NTMs since 1999

I am lookng for a free spreadsheet program that can use macros to get away from my Microsoft dependency I have tried Open Office but it is very slow

regards

Richard

[nohal]

Richard...
Default OpenCPN icons: https://github.com/nohal/OpenCPN/blo...eman.cpp#L1178 - until line 1222 - the first parameter to the MAKEICONARRAYS macro is the name you are looking for.

I'm afraid the best free spreadsheet you can find out there really is OpenOffice (or currently Home » LibreOffice clone, which gets more updates)
Maybe time to become a programmer is coming even for you It would be pretty nice to let the servers do the work and have to download just the fresh preprocessed NTMs every week without the need to do anything and wait.

GPX route: <rte> elements after the waypoints.
Minimalistic example (the wpt is there just to show it comes first, to add more points to the route, just add subsequent rtept elements):

<?xml version="1.0" encoding="utf-8" ?>
<gpx version="1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.topografix.com/GPX/1/1" xsi:schemaLocation="http://www.topografix.com/GPX/1/1 http://www.topografix.com/GPX/1/1/gpx.xsd">
    <wpt lat="37.086119821" lon="19.286151872"/>
    <rte>
        <rtept lat="10.567943897" lon="15.937784081"/>
        <rtept lat="11.567943897" lon="15.937784081"/>
    </rte>
</gpx>

The same with the OpenCPN extensions:

<?xml version="1.0" encoding="utf-8" ?>
<gpx version="1.1" creator="OpenCPN" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.topografix.com/GPX/1/1" xmlns:gpxx="http://www.garmin.com/xmlschemas/GpxExtensions/v3" xsi:schemaLocation="http://www.topografix.com/GPX/1/1 http://www.topografix.com/GPX/1/1/gpx.xsd" xmlns:opencpn="http://www.opencpn.org">
    <wpt lat="37.086119821" lon="19.286151872">
        <time>2011-12-02T09:17:02Z</time>
        <sym>triangle</sym>
        <type>WPT</type>
        <extensions>
            <opencpn:guid>4ba2ffff-570e-4e27-bac2-7f825256ffff</opencpn:guid>
            <opencpn:viz>1</opencpn:viz>
            <opencpn:viz_name>1</opencpn:viz_name>
        </extensions>
    </wpt>
    <rte>
        <extensions>
            <opencpn:start></opencpn:start>
            <opencpn:end></opencpn:end>
            <opencpn:viz>1</opencpn:viz>
            <opencpn:guid>510affff-9adf-4c21-9173-046a16100000</opencpn:guid>
            <gpxx:RouteExtension>
                <gpxx:IsAutoNamed>false</gpxx:IsAutoNamed>
                <gpxx:DisplayColor>Black</gpxx:DisplayColor>
            </gpxx:RouteExtension>
        </extensions>
        <rtept lat="10.567943897" lon="15.937784081">
            <time>2011-11-28T15:11:09Z</time>
            <name>WP1</name>
            <sym>diamond</sym>
            <type>WPT</type>
            <extensions>
                <opencpn:guid>120d0000-6936-4398-a4a8-c07732db0000</opencpn:guid>
                <opencpn:viz>1</opencpn:viz>
                <opencpn:viz_name>0</opencpn:viz_name>
                <opencpn:auto_name>1</opencpn:auto_name>
            </extensions>
        </rtept>
        <rtept lat="11.567943897" lon="15.937784081">
            <time>2011-11-28T15:11:09Z</time>
            <name>WP1</name>
            <sym>diamond</sym>
            <type>WPT</type>
            <extensions>
                <opencpn:guid>120d0000-6936-4398-a4a8-c07732db0001</opencpn:guid>
                <opencpn:viz>1</opencpn:viz>
                <opencpn:viz_name>0</opencpn:viz_name>
                <opencpn:auto_name>1</opencpn:auto_name>
            </extensions>
        </rtept>
    </rte>
</gpx>

If you want to study the official docs: GPX 1.1 Schema Documentation

Pavel

[r.fairman]

As a test I have manually reformatted the output from NGA Ntm's using Lotus Symplony. this took about 2 hours with one hour still to go. This was for all the NTM's for all the charts in region five for the past 12 months I estimate that about 400 man hours (or seagull hours) are needed to create gpx files to apply all the NTM's for all the NGA charts


Lotus Symphony is free and supports macros. I have also converted to KMZ for display in GE.

Here is the file

This may be of use for users in the future to decide if their area is affected by a NTM

I have converted some linked areas to tracks some can be seen on the east and south cost of the main island of the azores which identify anchorages. also on the west coast of Spain some marine reserves are outlined.

Mouse over a point will display the description Clicking on an an outlined area also displays the description.

Once I have searched and replaced the symphony file to insert the nav objects in will post the gpx file

[PjotrC]

I tried another approach and the results seem promising... zero manual editing and good performance...

I wrote two simple Python scripts:

1. ntmcsv2db.py
2. ntmdb2gpx.py

Script 1 reads an NTMChartCorrections.csv file as downloaded from NGA Maritime Safety Information web page, for any chart set, with coordinate format chosen as decimal minutes, PC format. The script does the following:

- splits file contents into individual .txt files per chart per NtM
- writes the individual notices into a directory tree with 10 subdirectories: one for every region 1-9, and one (0) for all other charts
- within every region creates directories for referenced charts, and writes there files with NtM texts
- generates a .csv file with all coordinate points found anywhere in the notices, referenced to chart number, NtM number, NtM type and NtM text file location

Script 2 converts the above produced points .csv into .gpx with "Additional Information" links pointing to NtM text.

The purpose of this excercise was to get a feeling as to the numbers involved and any performance issues. Certainly this is not a definitive way of handling NtMs, but it works surprisingly well.

Results:

I tried this with a real folio of ca. 60 NGA charts I had been maintaining for some years.
I downloaded all NtMs for this set, starting at the original editions, plus the full NtM of week 52/11, ca. 180 charts in all. This resulted in 1500 Notices (i.e. cases of application of a Notice to a Chart).

Script 1 produced 180 directories and filled them with 1500 files (<30 seconds, including logging to console).

Script 1 produced also the .csv file with ca. 4000 points.

Script 2 converted this into a 1200 KB file, that can be loaded into OCPN as a Layer, again in a couple of seconds.

Results for Region 5 and NtMs for past 12 months: 350 charts, 800 Notices, 3000 correction points, <5 seconds total compute time.

OpenCPN 2.5.0 handles the 4000 waypoints layer very well. The potentially limiting factor is generating the WP names in the Route Manager Dialog Tab. Without listing individual names (not really needed) there is no performance problem in OCPN.

The directories with text files are useful anyhow to store and browse individual Notices.

The Notice Texts might in some cases still be parsed further, but I doubt if this is needed or worthwile to do now.

Some more difficult points still need to be addressed:

1. instead of having one layer with all notices, the display should be adapted to the scale currently shown on screen
2. as much help as possible should be offered for handling of the chartlets supplied with NtMs

These will not be possible with simple GPX approach... a plugin interfacing to some database is needed here...

Anyhow, writing Python is probably a lot easier than using spreadsheet macros...

Piotr

[r.fairman]

wow fantastic Could we display the NTM description ie wreck, submarine cable etc as the waypoint name then action for each waypoint plotted would be clear. Or is too much of a hit on the OpenCpn performance Names can of course be hidden to reduce chart clutter. is it also possible to parse the cvs or the final gpx to display the nav object.
.
for example in your screen shot could NTM 34 on the chart display wreck as the name and the wreck nav object.

keeping the additional information to the NTM is also desirable. belts and braces. Such a solution as yours far exceeds what commercial chart providers supply

In region 5 alone I have counted many tens of wrecks not plotted in the NGA charts but included in the NTM,s Many of these are also not listed in up to date electronic charts. From commercial sources.

Anyway a major move forward

Regards

richard