MBTiles for OpenCPN

[nohal]

Dave...

We have first real set with 512x512 tiles - https://github.com/netAction/signalk...-coastline-map.

To my surprise, the best results for the missing bounds are so far always achieved from the highest resolution layer in my (growing) collection of problematic sets, leaving us with the simplest possible algorithm for now. PR sent.

Some more observations:

• In quilted mode (only with GSHHG+1 mbtiles set in my current setup), the mbtiles set is always excluded for me as soon as I zoom in or out.
• Switching to single chart mode causes the chart canvas to stop reacting to me completely. Starting O in single chart mode is fine.
• In quilted mode, scale is changed when I switch to a mbtiles regardless of "Preserve scale..." setting.

Pavel

[nohal]

Originally Posted by nohal

Switching to single chart mode causes the chart canvas to stop reacting to me completely. Starting O in single chart mode is fine.
In quilted mode, scale is changed when I switch to a mbtiles regardless of "Preserve scale..." setting.

diff --git a/src/viewport.cpp b/src/viewport.cpp
index cdb97fd2d..42e437912 100644
--- a/src/viewport.cpp
+++ b/src/viewport.cpp
@@ -168,6 +168,7 @@ wxPoint2DDouble ViewPort::GetDoublePixFromLL( double lat, double lon )
         lat0_cache = clat;
         switch( m_projection_type ) {
         case PROJECTION_MERCATOR:
+        case PROJECTION_WEB_MERCATOR:
             cache0 = toSMcache_y30(clat);
             break;
         case PROJECTION_POLAR:
@@ -183,6 +184,7 @@ wxPoint2DDouble ViewPort::GetDoublePixFromLL( double lat, double lon )
 
     switch( m_projection_type ) {
     case PROJECTION_MERCATOR:
+    case PROJECTION_WEB_MERCATOR:
         toSMcache( lat, xlon, cache0, clon, &easting, &northing );
         break;
 
@@ -277,6 +279,7 @@ void ViewPort::GetLLFromPix( const wxPoint2DDouble &p, double *lat, double *lon
     double slat = 0.0, slon = 0.0;
     switch( m_projection_type ) {
     case PROJECTION_MERCATOR:
+    case PROJECTION_WEB_MERCATOR:
         //TODO  This could be fromSM_ECC to better match some Raster charts
         //      However, it seems that cm93 (and S57) prefer no eccentricity correction
         //      Think about it....

Seems to help with this one, but I really don't know much what I'm doing here...

[seandepagnier]

Originally Posted by bdbcat

Pavel...

Bounds: Error in this parm is not really critical. Not really used for Raster quilting.

It might be useful to sort the order for generating texture cache once that is implemented.

In MBTiles, we scan the lowest zoom level tiles to build the valid chart region (for quilting). I think it might be fair to simply toss out tiles that are all black in this scan. We end up with a non-rectangular coverage region, but that is OK.

mbtiles can cover widely different areas at each zoom level... Wouldn't there be a different boundary (and coverage region) for each zoom level?

What will be more problematic is if the minzoom tileset is not contiguous. This will lead to trouble in quilting, I think, as the logic tries to figure out what to do about "holes" in the coverage region.

Why is this an issue? The LLRegion should fully support multiple regions, and regions with holes in it. It is already used in vector charts. Again, we need different coverage regions for each zoom level.

In the worst case we would need to "logical-OR" all of the tiles in the dB to generate the correct coverage area. And still probably need to detect and deal with holes.

I would imagine that only the tiles from the correct zoom level are used, and quilted with other charts. Maybe if no other options are left (similar to quilting cm93 with raster charts) then lower zoom levels could be used?

[bdbcat]

Sean...

re:
Why is this an issue? The LLRegion should fully support multiple regions

Because LLRegion is expensive and not without corner-case errors when doing complex logical ops.

Far simpler, I think, to arrange for the tileset to "cover" a fixed region for the entire set. Other wise, we have to build a "composite" architecture like cm93. We may be forced to this, anyway, but I'd rather exploit a simpler solution first if we can.

Dave

[bdbcat]

Pavel...

re:

• In quilted mode (only with GSHHG+1 mbtiles set in my current setup), the mbtiles set is always excluded for me as soon as I zoom in or out.

Sorry, what tileset used?

Thanks
Dave

[nohal]

Originally Posted by bdbcat

Pavel...

re:

• In quilted mode (only with GSHHG+1 mbtiles set in my current setup), the mbtiles set is always excluded for me as soon as I zoom in or out.

Sorry, what tileset used?

Thanks
Dave

Dave...

As far as I can say any I have so far. All the "problematic ones" and NOAA 23 (NW Florida).
The only exception is the latest download, NOAA 08 (The Pacific Islands) - that does not show at all, ever (although it sometimes makes it to the piano and the chart outline seems OK).

Pavel

[bdbcat]

Pavel...
As you zoom in from start, at some point the tileset appears, yes?

This is all controlled by the declared chart scale, shown on the piano rollover of the tileset. And this is equal to the OSMStreetmaps scale of the minZoom of the tileset. Logic may be too simplistic here, though. Thus, the experiments.


For fun, try to boost the Options->Display->Advanced->Raster zoom factor. Should have some effect.

Or, look at Quilt.cpp:830

return scale * 4 * mod; // MBTiles are fast enough Boost the "4" upward to experiment.

Dave

[nohal]

Originally Posted by bdbcat

Pavel...
As you zoom in from start, at some point the tileset appears, yes?

This is all controlled by the declared chart scale, shown on the piano rollover of the tileset. And this is equal to the OSMStreetmaps scale of the minZoom of the tileset. Logic may be too simplistic here, though. Thus, the experiments.


For fun, try to boost the Options->Display->Advanced->Raster zoom factor. Should have some effect.

Or, look at Quilt.cpp:830

return scale * 4 * mod; // MBTiles are fast enough Boost the "4" upward to experiment.

Dave

Dave...
Spot on. From some earlier test I have had the raster chart zoom/scale weighting slider set to 5. And that turns out to be the necessary condition needed to reproduce. With anything from -5 to 4, everything works fine.
With the exception of NOAA 08, no change in behaviour there.

Pavel

[seandepagnier]

Originally Posted by bdbcat

Sean...

re:
Why is this an issue? The LLRegion should fully support multiple regions

Because LLRegion is expensive and not without corner-case errors when doing complex logical ops.

I thought we fixed all the corner case errors... Are there still cases you found where the results of logical region operations are incorrect?

As far as expensive... Can't the regions be constructed when the chart is first scanned, and stored in the chart database? Or are you concerned with quilting performance?

We should consider offloading quilting operations to a background thread. They do not need to execute every frame, and can be out of sync or even lag by a few seconds on slow processors to improve fluidity of panning and zooming.

Far simpler, I think, to arrange for the tileset to "cover" a fixed region for the entire set. Other wise, we have to build a "composite" architecture like cm93. We may be forced to this, anyway, but I'd rather exploit a simpler solution first if we can.

simpler, but, this is not the nature of mbtiles. A lot of mbtiles are currently produced from wherever the user panned, or arranged to scan the coastlines. You cannot expect a rectangular region.

We could detect if the mbtiles is rectangular, and enable specific optimizations for this case, but otherwise, I think you are looking at a "composite" architecture for general support.

With this, multisampling on zooming across quilts becomes a separate project (for later). This would work with all different types of charts together and be quite cool.


Another serious thing to consider is partial transparency which mbtiles supports. It is normally a bitmap (1 bit transparency) but this "significantly" complicates region operations. I think we would have to have two regions, one which includes all tiles, and one which includes only fully opaque tiles. The opaque tiles are used for quilting calculations, and the all tiles region can be used for rendering detection. The charts (as always) would have to be rendered in the correct order.

[bdbcat]

Sean...

re:
You cannot expect a rectangular region.

I don't expect or need a rectangular region.

What I would like is for the region made by combining all minZoom tiles to be:
a) contiguous, no holes
b) large enough to encompass all larger scale tiles.

This is how NOAA sets are built, and the natural way for GUI tileset creators to work.

Do you have a source for useful tilesets that don't work this way? I'd like to see them, please.

On the LLRegion errors:
There is some precision problem, somewhere. If I make a region by successively adding tiles ( defined by LLBox ), sometimes the resulting region outline contains a small sliver of very tiny width or height at tile boundaries

So, I added a small hack to mbtiles.cpp:526 which masks the problem.
I grow the tiles by one meter so that adjacent tiles overlap and thus always combine properly.

Not a happy hack. Needs fixing at LLRegion source, really.

Dave

[seandepagnier]

Originally Posted by bdbcat

Sean...

What I would like is for the region made by combining all minZoom tiles to be:
a) contiguous, no holes
b) large enough to encompass all larger scale tiles.

Some mbtiles like the noaa ones have a single tile for minzoom at level 0. It may encompass all larger scale tiles, but it doesn't make a very nice border.


It is certainly possible to make mbtiles files with only a single zoom level, and/or plenty of holes in them.

This is how NOAA sets are built, and the natural way for GUI tileset creators to work.

Do you have a source for useful tilesets that don't work this way? I'd like to see them, please.

Not at the moment.

On the LLRegion errors:
There is some precision problem, somewhere. If I make a region by successively adding tiles ( defined by LLBox ), sometimes the resulting region outline contains a small sliver of very tiny width or height at tile boundaries

If the tiles are on a perfect grid, so the coordinates are identical it should work... but the tesselator has always hated nearly identical segments.

So, I added a small hack to mbtiles.cpp:526 which masks the problem.
I grow the tiles by one meter so that adjacent tiles overlap and thus always combine properly.

You should be able to grow by a bit less than a meter.

Not a happy hack. Needs fixing at LLRegion source, really.

Dave

I will take a look, would be nice to fix the tesselator, but this hack should be ok for now.

[Baikal]

I'm building CPN on the latest changes
Many fatal errors take off

[nohal]

Originally Posted by Baikal

I'm building CPN on the latest changes
Many fatal errors take off

Follow https://github.com/nohal/OpenCPN/wik...n-Windows---O5 we are in the middle of changing the build and dependencies on Windows.

[BlackSea]

Hi,

once more again. Could anybody explain, which requirements must be satisfied for mbtiles?
- Which schema? XYZ or TMS?
- Which metadata must be presented?

just 2 cent
Could it be useful for some operations to use GDAL driver for mbtiles? ( MBTiles )

gdalinfo world.mbtiles
Driver: MBTiles/MBTiles
Files: world.mbtiles
Size is 32768, 16640
Coordinate System is:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],
    AUTHORITY["EPSG","3857"]]
Origin = (-20037508.342789243906736,11897270.578531112521887)
Pixel Size = (1222.992452562820063,-1222.992452562820063)
Metadata:
  ZOOM_LEVEL=7
  format=png
  version=1
  name=World
  description=Worldmap in zoom 3,4,6,7
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-20037508.343,11897270.579) (180d 0' 0.00"W, 72d23'44.54"N)
Lower Left  (-20037508.343,-8453323.832) (180d 0' 0.00"W, 60d14'23.32"S)
Upper Right (20037508.343,11897270.579) (180d 0' 0.00"E, 72d23'44.54"N)
Lower Right (20037508.343,-8453323.832) (180d 0' 0.00"E, 60d14'23.32"S)
Center      (       0.000, 1721973.373) (  0d 0' 0.01"E, 15d17' 3.07"N)
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
  Overviews: 16384x8320, 8192x4160, 4096x2080, 2048x1040
  Mask Flags: PER_DATASET ALPHA 
  Overviews of mask band: 16384x8320, 8192x4160, 4096x2080, 2048x1040
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
  Overviews: 16384x8320, 8192x4160, 4096x2080, 2048x1040
  Mask Flags: PER_DATASET ALPHA 
  Overviews of mask band: 16384x8320, 8192x4160, 4096x2080, 2048x1040
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
  Overviews: 16384x8320, 8192x4160, 4096x2080, 2048x1040
  Mask Flags: PER_DATASET ALPHA 
  Overviews of mask band: 16384x8320, 8192x4160, 4096x2080, 2048x1040
Band 4 Block=256x256 Type=Byte, ColorInterp=Alpha
  Overviews: 16384x8320, 8192x4160, 4096x2080, 2048x1040

[nohal]

Originally Posted by BlackSea

Could anybody explain, which requirements must be satisfied for mbtiles?
- Which schema? XYZ or TMS?

XYZ

- Which metadata must be presented?

With the current code none (Your world.mbtiles posted earlier works just fine with the current master code including https://github.com/OpenCPN/OpenCPN/p...a3fc76c6e435dc)

Pavel