OpenCPN Grid and Chart Datum

[seasniffer]

Hi,
playing with OpenCPN and a few italian (from hydrographic institute) paper scanned coast maps, it seems that OpenCPN grid does not exactly follow Mercator paper grid of the maps. If I tune the middle of the map using datum offsets in KAP format, grid at the edges do not match anymore. Differences are about 0.05 / 0.1 ' on a 100000 scale map.

With reference to "American Practical Navigator" (Bowditch) PDF last 2002 bicentennial edition, pag. 23 and similar topics about chart precision, or similar tutorials and books on the topic, it seems that the grid should depend about datum choosen for the map. In case of italian paper (old) maps, datum is Rome 1940, not WGS84.

Problems in the grid could also be added while scanning the raster images from paper, but to me it seems a rare case, if a flat scanner is used, (not a picture taken with a camera).
This is a quirk topic, but the question is:

(1)Does openCPN grid refer latitude and longitude based upon the chart datum, or the WGS84 system ?

(2)How it is possible to improve positions on map with OpenCPN, in order that map paper positions are the same as GPS positions, e.g. current GPS boat position from NMEA sources matches current boat position on map ?

I apologize for the heuristic explanation of the problem, but I suspect that topic is quite clear in spite of its complexity.

Hope someone can help me ....

Alfredo

[cagney]

Originally Posted by seasniffer

Hi,
playing with OpenCPN and a few italian (from hydrographic institute) paper scanned coast maps, it seems that OpenCPN grid does not exactly follow Mercator paper grid of the maps. If I tune the middle of the map using datum offsets in KAP format, grid at the edges do not match anymore. Differences are about 0.05 / 0.1 ' on a 100000 scale map.

With reference to "American Practical Navigator" (Bowditch) PDF last 2002 bicentennial edition, pag. 23 and similar topics about chart precision, or similar tutorials and books on the topic, it seems that the grid should depend about datum choosen for the map. In case of italian paper (old) maps, datum is Rome 1940, not WGS84.

Problems in the grid could also be added while scanning the raster images from paper, but to me it seems a rare case, if a flat scanner is used, (not a picture taken with a camera).
This is a quirk topic, but the question is:

(1)Does openCPN grid refer latitude and longitude based upon the chart datum, or the WGS84 system ?

(2)How it is possible to improve positions on map with OpenCPN, in order that map paper positions are the same as GPS positions, e.g. current GPS boat position from NMEA sources matches current boat position on map ?

I apologize for the heuristic explanation of the problem, but I suspect that topic is quite clear in spite of its complexity.

Hope someone can help me ....

Alfredo

1/ Yes, for recognized Geodetic Datums. See georef.c in the source code for a list.
2/ First start with a perfect picture of the paper chart, then make a proper georeferencing. OpenCPN is very good at handling small imperfections. To say what's the problem with your chart is impossible without seeing it. Do you notice any difference between single mode display and quilting? Any complaints from OpenCPN about the chart in your opencpn.log file?
93 - 185 meters is not very much on a chart in 1:100,000.
1mm on the chart corresponds to 100m, probably equal to the accuracy of the printing process itself. Then there are other errors in the process of creating the chart itself.
When was the survey that your chart is based on done? What methods were used, etc.

Thomas

[seasniffer]

Thanks, Thomas.

(1)Map with grid problem (the last investigated) is "003 DA PORTOFINO A SAN ROSSORE". These raster files are taken from Carte Nautiche Coste Italiane or similar sites, and are scanned copies of official maps from hydrographic italian office. The scanned version lacks the borders with lat and log rules and are rotated, to be useless for nautical use (and for avoiding copyright problems ...). These jpg files must be rotated back and calibrated (I used MapCal II) using the map existing grid or known point info. I used the grid from the map itself. Then bsb and imgkap tools do the remaining job to produce KAP files.
Up to now, I did not find updated info about this maps on internet, I will have to call by phone the hydrographic sale office and get info about s57 ENC version of these maps, because they have no on-line shop and I never saw one of these new ENC maps, that are listed in the new catalog. It seems that in the last 40 years, not a lot of improvements were made about italian coast recognition and mapping. Rome 1940 datum is still the base.

(2)I tryed to enable quilting in OpenCPN, but I do not see any change about grid behaviour playing with this option.

Paper printing precision seems to be more than 1 mm, I rotated the maps with gimp 2.6 and 2.8 and zooming the map I did not find anomalies. Of course the original paper of official maps (I have some of these, bought in nautical shops many years ago) is quite robust, and its printing quality is quite high, more then 1 mm. These paper maps are quite large and need a special scan process to be rasterized and become ENC charts.
In conclusion, before buying s57 or proprietary format ENC charts, I would like to try OpenCPN to do the job. ENC market in Italy seems to be quite completely a private proprietary market, with need for a dedicated program to use them, like the complete electronic equipments already on board of sailing boats when You buy a boat.

Any other idea ?

Alfredo

[HappySeagull]

Alfredo,
just a couple of comments ...
- RE: Grid...Is the chart a Mercator one? Mercator describes the grid spacing as a ratio ( latitude versus longitude on a flat projection at the chart's position).
This is NOT the same thing as "Datum".
-avoid jpegs. Use png or bmp format.
- I don't know about Rome1940 datum, but if I were you, I would just recalibrate the image to WGS84.

-....Therefore, given the above, the lines of latitude and longitude may be artifacts of the cartographer's art but still useful to see if the chart is actually a "flat" straight-sided, projection....



-and, all that said, Mercator and most other projections are cartographers' attempts to take a spherical piece of Planet Earth and lay it flat- therefore, THERE WILL BE ERRORS....usually at the edges as to landmarks and so on.


-Be sure what datum your gps is using and . Again, WGS84 is the best-choice (even though,I have read that gps actually do their primary calculation as UTM....)
Try to use pngs , bmps, or tiff format from the first scan. AVOID JPG or jpegs. Change any jpeg you get to a png before saving it. There is aLossless Jpeg format out there but unless you know what I am talking about, stay with the png,or tiff or bmp.

[seasniffer]

Hi HappySeagull ,

I used a PNG file as imput to Mapcal II and imgkap, the original jpg from the site was downloaded, rotated, cropped and color reduced to a 16 color palette, then saved as PNG, so jpg was not used for calibration.
I tried to change datum to/from WGS84 or Rome 1940 in KAP file, but the OpenCPN grid does not change in such situation. It changes if You change datum offet (DTM kap field) and it gets completely different if projection is set to transversal Mercator instead of Mercator, (but this was just a test to see if something change ..).

I tried to attach (but I did not succed ...) two snapshots of
(1)center-right part of map as rendered in OpenCPN 3.0.2, and
(2)left part of map. You see that meridians do not match on both pictures.

Using DTM KAP offset, the grid is shifted, but its shape is not modified. Torre Scola is deemed to be 44 03,1 N 009 51,5 E in a 2011 regatta description sheet. I put a waypoint ("dfgdfg") at those coordinated in openCPN. You may see the shift (waypoint does not match Torre Scola), so, there is something wrong. Same happens if I use other objects reference, the coordinates do not match completely with the grid also for other objects (e.g. ODAS 1 oceanographic buoy).

I had an idea : I will next try to calibrate with objects instead of using grid lat/long crossing points on map, and I will post the results of calibration and checks in my next reply.

Alfredo

[cagney]

I tried to change datum to/from WGS84 or Rome 1940 in KAP file, but the OpenCPN grid does not change in such situation.

Not quite true. I have tested and it does work. In your case the difference between Rome 1940 and WGS84 is only (0.04', -0.02'), which translates to 0.75mm,at 1:100,000, in latitude, so I suspect that the change isn't noticed. Try a couple of other GD and you will see a difference!

Thomas

[HappySeagull]

gee, Alfredo- I had a look at the image you are converting and it's not that great... After rotating it 3.7 degrees (roughly speaking) I couldn't get the distortion out of it well enough to really even see such a small discrepancy, such as Thomas pointed out.
Just as a comparison, a "real" kap I have for 1:100,000 scale is an image that's 11,000x8,000 pixels- a lot bigger than the one I saved (4506x3018) from your link. I don't think it reasonable to puzzle over small discrepancies in any chart I could make from that image....
That said, I would happily use them if I had no other, but I think it's an excellent idea you have to make "mark/wpt s" placed for lights and landmarks from positions in an official List of Lights and also from other sources- even Google Earth-just to give a little more surety close to shore. I just wouldn't worry overmuch about the grid-I sure couldn't get the lines straight at that scale, on that image....
If I'm missing something speak up. Maybe I'm not downloading full-size or something...?

[seasniffer]

I am aware about low quality of images we are speaking of. I cannot find right now something more reliable.

But after many trials with openCPN, MapCal/SeaClear and other stuff, I realized that is difficult to get a precision higher that 0.1 nautical mile if it is not clear what we want to get and how.

A few question infact arise:
(1)Which cohordinates does show openCPN ? WGS84, or does it read from the map datum ?

(2)Where are common (recent times) sense cohordinates taken from ? - e.g. if i read a PDF on the Internet that explains me that a buoy is blabla lat and blabla long, non always it is also explained in the PDF the exact reference system used.

(3)When I correct an old datum chart with aux offset (e.g. 0.04 ' S, 0.02' E), do I obtain what ? - do I obtain a practical method to read on map old cohords and get new ones ?, but the old grid on paper still refers to old cohords, so it is impossible to get new cohords and matching grids on ENC, at the same time ? Is it ? I think that the two grids, if they ever match, they will lead to wrong wgs84 cohords.

(4)I had a confirmation from Italian Hidrographic Office and dealers, that at present on the market in Italy it is not possible to get un-encrypted ENCs. They sell S57/S63 ENCs /ECDIS for professional use (commercial vessels). The official reference system for leasure yatching is still on paper. Some providers (e.g. Garmin/Navionics) sell additional information, but do not substitute old style maps, for coastal navigation. Anyway, they sell the viewer sw together with the maps.

This justifies the idea of processing raster images, but needs an additional skill to understand how to deal with projections.

(5)SeaClear, MapCal and OpenCPN manuals should specify this stage in a more compact way, information at present is distribuited in a lot of different paragraphs.

Who deals with ENCs and not paper maps, will never create himself doubts about the maths - like a pc user that does not worry about all the sw bugs he is navigating upon, because he is not aware of them !!! http://www.cruisersforum.com/images/smilies/smile.gif

When both way to make routes come afloat, differences are quite evident.

Have You any idea how to sequentially indicate the process to get a cohordinate match between paper and raster ENC, starting from the scan, and ending with a set of waypoints and routes that will lack only the GPS precision, but everything else is predictable ?


I lost this hope ....http://www.cruisersforum.com/images/smilies/frown.gif

Alfredo

[GPS-Marco]

Ciao Alfredo,

Let me try to answer some of your questions.

The image you used is a poorly scanned image of the original chart. It is not only rotated but it is also distorted (Scanner problem: the grid lines are not exactly parallel/perpendicular one each other and the vertical scale is slighly different by the horizontal scale). Moreover the Datum is Rome 1940 and not WGS84.

Rotation and Datum are not big issues (can be perfectly corrected) but the distorsion is a big problem. To face this issue the KAP format foresees a number of reference points that can be placed spread all over the chart to instruct the software utilizing the calibration (OpenCPN, Seaclear/MapCal or other) on how to correct the distorsion. Of course, this methods cannot be perfect (at least, in theory, you have a ref point for each pixel of the image...).

The preferred method to modelize the chart distorsion is to evaluate a third degree bidimensional polynomial that converts geographic lat/lon to X/Y chart pixel and viceversa. These Polynomials may be either calculated by the software by utilizing the reference points or can be directly included in the KAP chart itself. OpenCPN uses the polynomials included in the KAP (if they are present) otherwise it calculates the polynomials by itself.

Anyway these third degree polynomials cannot correct large distorsions so you must expect some errors (at pixel level) at the end of the process.

Now, about your chart. I've tried to calibrate the chart by rotating it (3.7°), converting to 63 colours with Gimp and by using MapCal with 39 reference points (one for each crossing of the printed grid). The result is not that bad and OpenCPN shows errors in the order of 10 pixel at most (as you said, in the order of 0.1 NM at most).

Note anyway that to exactly evaluate this error, i.e. to compare OpenCPN GRID with Chart GRID, you have to remove the grid shift due to Datum Conversion (the 2 grids will never overlap due to the distorsion but also due to the Datum difference: Roma 1940 for Chart and WGS84 for OpenCPN). To remove shift (just to compare grids) open the Kap file with a text editor (e.g. Notepad++) and change the ,GD=Roma 1940, to something like ,GD=Unknown, then save and reopen OpenCPN by instructing a chart data base reconstruction (Note: DTM shall remain to 0,0).

Now the question is: is it possible to reduce the final error? Sometimes, like in your case, Yes.

Unfortunately the Polynomial evaluation made by OpenCPN is not always the best possible. Sometimes exist polynomials that better fit the chart distorsions so they can better correct the scanning errors. I developed an experimental version of mc2bsbh that calculates the polynomials with a refined algorithm and insert them directly in the KAP header file. OpenCPN will find those polynomials in the KAP and will use them. Somtimes they are better than the OpenCPN internal calculated ones.

I attached to this post 5 images that shows what happens in the most distorted location of your chart. You can compare the grid crossing (below San Rocco) and a Waypoint (a red x) that should be on the signal of the harbour (Santa Margherita).

1) Error by using internal OpenCPN polynomials and without datum shift (GD=Unknown)
2) Error by using internal OpenCPN polynomials and with datum shift (GD=Roma 1940)
3) Error by using external calculated polynomials and without datum shift (GD=Unknown)
4) Error by using external calculated polynomials and with datum shift (GD=Roma 1940)
5) Verification of the Waipoint position with respect to a reference chart

As you can see, picture 3 shows a grid with a near perfect calibration but note that the right chart to use is the fourth that shows a perfect calibration of the objects (and not of the grid)

The modified version of mc2bsbh (mc2bsbh-poly.exe) can be found Here #531

Ciao, Marco.

[jonasaberg]

Originally Posted by GPS-Marco

Unfortunately the Polynomial evaluation made by OpenCPN is not always the best possible. Sometimes exist polynomials that better fit the chart distorsions so they can better correct the scanning errors. I developed an experimental version of mc2bsbh that calculates the polynomials with a refined algorithm and insert them directly in the KAP header file. OpenCPN will find those polynomials in the KAP and will use them. Somtimes they are better than the OpenCPN internal calculated ones.

Marco,
very good explanation, as always.

Is your polynomials always as good or better than the ones produced internally by OpenCPN, or are they sometimes worse also?

If they are always as good or better, maybe it should be suggested to Dave to change to your method in the OpenCPN code?

/Jonas

[seasniffer]

Ciao Marco,
thanks very much for Your very very very clear explanation.
I was suspicious about distorsion in the original paper map (or it scanned version), but You confirmed that.

I was not aware of polynomial capabilities of KAP format, and these are good news for me, since I use kap as OpenCPN input maps.

I did not exactly understand when You say "the vertical scale is slighly different by the horizontal scale".
As I know, in Mercator nautical charts this should be common practice, they tought us to use nautical compasses to compensate for this, on paper maps, when it is necessary to calculate distances (based upon average latitude of the route).

This discussion is very interesting, and as soon as I have results to post, I will try to upload them. (I had problems in uploading images to the forum, I have to solve this issue ...).

So Marco,

let's stay tuned. !!!

[GPS-Marco]

Originally Posted by jonasaberg

Marco,
very good explanation, as always.

Is your polynomials always as good or better than the ones produced internally by OpenCPN, or are they sometimes worse also?

If they are always as good or better, maybe it should be suggested to Dave to change to your method in the OpenCPN code?

/Jonas

Hi Jonas.

I've not tested it so extensively to be able to say that the polynomials I calculate are always better. Anyway they seem to be very reliable and very good for every chart I've tested.

I don't know how much work would be to add the algo to OCPN. The source code I've written is available but I've not the right experience and skill to understand if it can be easily integrated in OCPN. Also licence of some library I use should be investigated.

Ciao, Marco.

[seasniffer]

Some test result :

(1)An experiment with a mountain map (1:25000) with UTM grid passed through my flat scanner demonstrated that is very simple and easy to get distortion of the grid by scanning process and the raster output is unusable ...

(2)Using OpenCPN, MapCal, OkMap and AFTrack (on Nokia phones) the KAP BSB files with Rome Datum (with or with no offset configured) behave in the same way, grids do not match, but the difference is the same, whilst different programs.

(3)I did not succed in finding reasonable coordinate sets for known light houses or buoys in Italy, on the Internet there is a lot of confusion and different values, anyway they match with 0.1 NM precision, when they make sense, as already "forecasted" in this thread.

(4)My books on the topic (e.g. "Manuale dell'ufficiale di rotta", very similar to "American practical Navigator") do not identify the datum used for coordinates of lights in official hidrographic institute pubblication (that is the source for internet descriptions ...) .
Because of this mess, I think I will directly buy one of this lists, together with some new paper nautical charts, because >finally< there is no free sure information about object coordinates, unless You pay for it buying charts or light lists from the hydrographic office.
Then I will compare with GPS coordinates and see what happens.

Hope this helps ...