NEW - wcst_import should generate a reasonable default tiling

[Bang Pham Huu]

wcst_import should generate a reasonable default tiling for rasdaman collection in case this setting is not specified by user in the ingredients file.

Currently, for a 3D coverage, if tiling is not specified, petascope creates this tile

TILING ALIGNED [0:*,0:*,0:*] tile size 4194304

which is not good as it has these tiles in rasdaman:

 "[5:5,35136:35199,0:64999]",
        "[5:5,40320:40383,0:64999]",
        "[5:5,40384:40447,0:64999]",
        "[5:5,40448:40511,0:64999]",
        "[5:5,40512:40575,0:64999]",
        "[5:5,40576:40639,0:64999]",
        "[5:5,40640:40703,0:64999]",
...

It has to consider multiple details:

• datacube dimension
• which axes are spatial
• Number of bands and types
• What is primary access pattern: spatial slices, or across non-spatial axes, or something inbetween?
  • gdal slicer imports 2D data organized in spatial slices, so it's very likely the access will be on spatial slices and the tiling should focus on the spatial axes
  • netcdf/grib slicers probably import 3D+ data, so there can be some extension of the tiles to the non-spatial axes
• Insitu: if original files are tiled/chunked (tiff and netcdf especially), the tiling in rasdaman should be a multiple of the chunk sizes

wcst_import should show a WARNING that no tiling was specified, and print the generated tiling.

[Dimitar Misev]

DM will implement a function in coverage_util.py with signature:

def generate_tiling(number_of_axes: int,
                    spatial_axes_grid_indices: list[int],
                    band_base_type_sizes: list[int],
                    chunk_sizes_from_file: list[int] = None):
    """
    Generate a rasdaman aligned tiling scheme based on the provided parameters before importing a created coverage

    @param number_of_axes the coverage's number of axes, e.g. 3 (time, lat, long axes)
    @param spatial_axes_grid_indices a list of grid indices (0-based) of the spatial X/Y axes (e.g. Long, Lat axes in TIFF file);
                                     note: set to None if coverage has no X/Y spatial axes
    @param band_base_type_sizes a list of sizes in bytes of the datacube bands
    @param chunk_sizes_from_file an optional chunk specification read from
    one of the input files; when the input file is chunked, e.g. netcdf or tiff (note: set to None if file is GRIB format),
    this parameter should be set as a list of sizes for each axis. If a tiff file
    has 512x512 Block size, this parameter would be a list [512,512].
    @return if the tiling can be determined, a string specifying the tiling, e.g.
    "aligned [0:0,0:511,0:511] tile size 4194304". If the tiling cannot be
    determined, None is returned.
    """
    '''
    TODO: This function should be filled in next release to generate proper tiling if possible
    '''
    return None

This is then called in wcst_import to generate a tiling scheme.

wcst_import checks if the tiling parameter exists in the end, and throws error if not, with a hint for the user:

Hint: if not sure what tiling to specify in the ingredients file, the following is a reasonable default:
"options": { "tiling": "ALIGNED [0:*,0:*,0:*] tile size 4194304" }

[Bang Pham Huu]

wcst_import code to throw error if tiling not specified and the call of method generate_tiling function in applications/wcst_import/master/importer/importer.py are done in wcst_import, the rest is to implement the content of function generate_tiling.