.. _v3800: FORCE v. 3.8.00 =============== Release: 31.01.2025 - **FORCE L1AS** - force-level1-sentinel2 was finally removed after being deprecated for a while now. Use force-level1-csd. Contributions to substitute this program using the Copernicus Data Space Ecosystem endpoint are welcome and appreciated! - force-level1-csd currently suffers from a change of Google's way of indexing images. The previously used mechanic has been exchanged for BigQueries. This is not implemented yet. In addition, it may generate costs if it would be implemented. This is an ongoing issue which hase not yet been resolved: https://github.com/davidfrantz/force/issues/334. The current implication is that images newer than August 28, 2024 cannot be downloaded. - force-level1-csd: regardless of the previous bullet, support for S2C has been implemented. As GCS does not hold Landsat Collection 2 data, the option to download Landsat has been disabled. Use force-level1-landsat instead - force-level1-landsat has been updated to reflect changes in USGS' authentication. - **FORCE L2PS** - Support for Sentinel-2C (as well as a future Sentinel-2D) was added. - Coregistration has been enabled for Landsat, too. A potential use case is to better align tier 2 data. - force-level2 was refactored. Little has changed for users. The logfiles are more verbose, and log messages generated by ``force-level2`` and by the core process ``force-l2ps`` are more cleary separated. The queue file has got an additional status, wherein unsuccessfully processed images get ``FAIL`` status instead of ``DONE``. The wrapper script ``forcel-l2ps_`` was discontinued; ``force-level2`` is now directly calling ``force-l2ps``. - A new parameter ``STRICT_WATER_VAPOR`` was introduced, which will result in process abortion when the water vapor database does not hold a day-specific value, i.e. the usage of the climatology is forbidden. Applies to Landsat only. - A new parameter ``FILE_AOI`` was introduced. Users can now use an area of interest by specifying the path to a polygon-based vector file. This file will be used to mask the Level 2 products to the boundary of this AOI. The AOI will be reprojected (if necessary), and then rasterized to match the processed image. This operation is performed after the radiometric correction, but before the reprojection and tiling. It has little effect on computation time, but will considerably reduce output volume for images that are at the edge of your study area. - force-level2-report additionally outputs the compiled log data in a csv file for further inspection. - **FORCE HLPS** - Support for Sentinel-2C (as well as a future Sentinel-2D) was added. - Analogous to Python UDFs, R UDFs can now be used through FORCE. This applies both to the TSA and UDF submodules. Three new parameters were added: ``FILE_RSTATS``, ``RSTATS_TYPE``, ``OUTPUT_RSP``. At least two functions must be present in the UDF: ``force_rstats_init``, as well as ``force_rstats_pixel`` or ``force_rstats_block`` (depending whether ``RSTATS_TYPE`` is set to PIXEL or BLOCK). To run PIXEL-type functions, you need to install the ``snow`` and ``snowfall`` packages. To run the FORCE components that are now compiled against the R API (mostly force-higher-level, but also force-qai-inflate), you need to provide two environment variables, i.e. ``R_HOME`` and ``LD_LIBRARY_PATH``. You can do it like this: - export R_HOME=$(R RHOME) - export LD_LIBRARY_PATH=$R_HOME/lib - force-higher-level parameters.prm In the pre-built Docker container, everything is pre-configured already. - Added a new parameter: ``STREAMING = FALSE`` can disable the streaming strategy of FORCE HLPS. When TRUE (previous behaviour, and still the default), FORCE will perform reading, computing and writing at the same time. If FALSE, these operations are done after one another in sequential mode. Disabling streaming might be necessary for some UDFs that otherwise produce threading conflicts with the internally used OpenMP functionality. When using R UDFs, FORCE will issue a warning and gracefully disable streaming. - Added a new parameter: ``PRETTY_PROGRESS``. HLPS will display progress information on screen. When TRUE (previous behaviour, and still the default), the progress information overwrites itself to produce a pretty displayal. However, this can cause error messages (or printing in UDFs) to be overwritten. If disabled (FALSE), the progress information will be simply be appended on screen (stdout). - Removed warning message that no output is produced when using the sampling submodule. It is now checked whether the files are actually written and will only warn if no input was detected or if no files were written. The behaviour for all other submodules stays the same. - It is now possible to specify the product types that should be used. This is to give the user more freedom with respect to data sources to be used. There are two new parameters: ``PRODUCT_TYPE_MAIN`` and ``PRODUCT_TYPE_QUALITY``. The main product is usually a reflectance product like ``BOA``. When using composites, you may use BAP. This can be anything, but make sure that the string can uniquely identify your product. As an example, do not use ``LEVEL2`` as this will not filter products apropriately. Note that the product should contain the bands that are to be expected with the sensor used, e.g. 10 bands when sensor is SEN2A. The quality product type should be a bit flag product like ``QAI``. When using composites, you may use INF. This can be anything, but make sure that the product contains quality bit flags as outputted by FORCE L2PS. As an exception, it is also possible to give ``NULL`` if you don't have any quality masks. In this case, FORCE will only be able to filter nodata values, but no other quality flags as defined with ``SCREEN_QAI``. Feature requested by Marcel Schwieder and Felix Lobert. - In the TSA submodule, it is now possible to disable fitting a monotonic trend within the harmonic model. The new parameter ``HARMONIC_TREND = TRUE/FALSE`` was added. Default behaviour (as before) is ``TRUE``. - After being deprecated for a while now, the SPLITS method has been removed from the code base (TSA submodule) - The Level 3 submodule now allows to use the internal scores for determining the best available observation as as a weight, i.e., not performing a selection, but a weighted average. This idea is taken from the MAJA WASP approach of Olivier Hagolle. A new parameter ``SELECT = TRUE/FALSE`` was introduced. with ``FALSE`` enabling the weighted average. Note that the ``INF`` and ``SCR`` products will not be produced when using this option. - The Level 3 submodule received the new parameters ``USE_CLOUDY`` and ``USE_HAZY``, which can be used to allow observations with a cloud or haze score of less than 1% in the composite. On one hand, this reduces nodata gaps in the composite. On the other hand, it will include clouds or haze if there is no better observation. - The Level 3 submodule received a new parameter ``COMBINE_SCORES``, which allows you to choose between additive or multiplicative combination of the individual scores into the total score. Additive combination is the previously used method and remains the default. Multiplicative combination can be used, which is much stricter in case one observation is characterized by a very poor suitability in a single score. - a new parameter ``OUTPUT_SUBFOLDERS`` was implemented that allows to output the data into subfolders under the tile folder instead of just dumping everything into the tile folder itself. It is planned that this feature will be expanded in the future. ``OUTPUT_SUBFOLDERS = FALSE`` remains the default. - UDF functions now receive sensor short names instead of the generic ``BLEND`` that was used before. This only applies if the non-interpolated time series is given to the UDF. If the interpolated time series is given to the UDF, ``BLEND`` is still passed into the function. - TSA outputs now carry the whole ``DATE_RANGE`` in their filename to reduce overwriting conflicts. - In the TSA submodule, ``INTERPOLATE = NONE`` has become the new default as it has become evident that the previous default ``INTERPLOATE = RBF`` confused users. - In the TSA submodule, the outlier/inlier detection has been switched off per default. - **FORCE AUX** - new auxilliary program `force-info`, which replaces `force` effective immediately. The new program has the same functionality, but does not hardcode the available tools. Tools that are still missing the ``-i`` option are currently not listed though. This still needs to be done. - new auxilliary program `force-init`. This program will create a new project with reasonably named folders that can be used as a starting point for a new project with some suggestions on organizing things. Unused folders can be deleted and the naming and structure is a mere suggestion and by no means prescriptive or mandatory. This tool is especially meant for beginners. - new auxilliary program `force-datacube-size`. This program prints the size of your datacube, per sensor and in total. - new auxilliary program `force-hist`. This program computes the histogram of image values (can be vrt) and writes a csv table. This is intended to be used in a validation workflow. - new auxilliary program `force-sample-size`. This program computes the required sample size following Olofsson et al. 2013/2014, which should be used to properly validate a classification map. This is intended to be used in a validation workflow. - new auxilliary program `force-stratified-sample`. This program draws a stratified random sample based on a classification map and sample size computations. This is intended to be used in a validation workflow. - new auxilliary program `force-map-accuracy`. This program computes area-adjusted accuracies following Olofsson et al. 2013/2014. This is intended to be used in a validation workflow. - ``force-tabulate-grid`` has been updated to produce properly named output files. The default output file name is ``grid.kml``, created in the current directory, using the ``KML`` format. ``-o`` can override the output file name (including path and extension). ``-f`` can override the file format, use the GDAL vector drive short name - thus grid can now be generated in any GDAL vector format. Both need to be given if not using defaults. - ``force-tile-extent`` has been updated to common Linux usage with options. Small changes for users are necessary, though all the functionality remained the same - Florian Katerndahl fixed the available resampling options in ``force-cube`` as a response to changes in GDAL. Thanks. - The CLI help of `force-tile-finder` has been corrected concerning the separator for the coordinates. - all aux scripts using parallel processing now use the number of available cores as default, not the number of input jobs. - **General** - The Makefile as well as the codebase has been restructured. It is currently under active testing whether it simplifies or hinders development. Some changes may be reverted in the future. Usage is not affected. - Some changes in the Dockerfile have been made. Thanks to Peter A. Jonsson. - Unit test functionality was added, along with a small set of unit tests. This is far from complete at the moment. A new program ``force-unit-testing`` has been added. This is meant for DevOps pipelines only, not for regular users. - a pre-commit hook has been implemented, such that *each* commit in the develop branch is assigned an individual software version. This is a time-stamped version number. It should be communicated when reporting errors, bugs, etc. This is a rather big collection of new featrues and many fixes. I hope I didn't miss anything important in here.