70

MODIStsp

A Tool for Automating Download and Preprocessing of MODIS Land Products Data

Allows automating the creation of time series of rasters derived from MODIS Satellite Land Products data. It performs several typical preprocessing steps such as download, mosaicking, reprojection and resize of data acquired on a specified time period. All processing parameters can be set using a user-friendly GUI. Users can select which layers of the original MODIS HDF files they want to process, which additional Quality Indicators should be extracted from aggregated MODIS Quality Assurance layers and, in the case of Surface Reflectance products , which Spectral Indexes should be computed from the original reflectance bands. For each output layer, outputs are saved as single-band raster files corresponding to each available acquisition date. Virtual files allowing access to the entire time series as a single file are also created. Command-line execution exploiting a previously saved processing options file is also possible, allowing to automatically update time series related to a MODIS product whenever a new image is available.

Travis-CI BuildStatus DOI CoverageStatus

MODIStsp is a “R” package devoted to automatizing the creation of time series of rasters derived from MODIS Land Products data. MODIStsp allows to perform several preprocessing steps (e.g., download, mosaicing, reprojection and resize) on MODIS data available within a given time period. Users have the ability to select which specific layers of the original MODIS HDF files they want to process. They also can select which additional Quality Indicators should be extracted from the aggregated MODIS Quality Assurance layers and, in the case of Surface Reflectance products, which Spectral Indexes should be computed from the original reflectance bands. For each output layer, outputs are saved as single-band raster filescorresponding to each available acquisition date. Virtual files allowing access to the entire time series as a single file can be also created. All processing parameters can be easily selected with a user-friendly GUI, although non-interactive execution exploiting a previously created Options File is possible. Stand-alone execution outside an “R” environment is also possible, allowing to use scheduled execution of MODIStsp to automatically update time series related to a MODIS product and extent whenever a new image is available.

For more information, documentation and examples of use, please see MODIStsp website at ropensci.github.io/MODIStsp

MODIStsp is developed and maintained by Lorenzo Busetto and Luigi Ranghetti, Institute of Remote Sensing of Environment - National Research Council - Italy (CNR-IREA)

Citation

To cite MODIStsp please use:

L. Busetto, L. Ranghetti (2016) MODIStsp: An R package for automatic preprocessing of MODIS Land Products time series, Computers & Geosciences, Volume 97, Pages 40-48, ISSN 0098-3004, http://dx.doi.org/10.1016/j.cageo.2016.08.020, URL: https://github.com/ropensci/MODIStsp.

Important News !

  • 29/11/2018 - We recently discovered a nasty bug in the computation of some custom spectral indices (those including additions / subtractions on reflectance values, such as in b1_NIR+0.1) / b2_Red. See here for further details! The bug is fixed as of version 1.3.7.

  • 07/08/2018 - We are glad to report that MODIStsp is now included in the rOpenSci packages’ ecosystem. We thank reviewers Leah Wasser and Jeffrey Hanson for their valuable reviews, which helped us to further improve the package!

  • 10/07/2018 - MODIStsp v. 1.3.6 is out. Check out the Release Notes for further details !

  • 20/06/2018 - MODIStsp v. 1.3.5 is out. Check out the Release Notes for further details !

  • 11/04/2018 - Due to new NASA Policies the MODIS FTP servers were shut down starting, April 2, 2018. FTP download is therefore no longer working and will be removed in the next MODIStsp version!

  • 11/04/2018 - Decommissioning of MODIS Version 5 Land Data Products. As per NASA notice above, MODIS v005 products are going to be decommisioned, and will soon be no longer available for download. Support for those products will be removed in the next MODIStsp version!.

  • 11/08/2017 - MODIStp 1.3.3 was released today. It provides improvements in processing speed, as well as the usual bug fixes (thanks to all the users that signalled problems !). Check the Release Notes for further details !

  • 25/07/2017 - As of today, **most of the content related to MODIStsp has been moved to our new website at ropensci.github.io/MODIStsp **, which provides a much better user interface and ease of access to MODIStsp-related information. From now on, please consult the new website for detailed and updated information on the package.

  • Also our previous FAQ page on github containing info for solving common installation, downloading and processing problems and issues was discontinued and migrated at ropensci.github.io/MODIStsp/articles/faq.html.

Problems and Issues

System Requirements

MODIStsp requires R v >= 3.2.1 and GDAL (Geospatial Data Abstraction Library) v >= 1.11.1 with support for HDF4 raster format to be installed in your system. Brief instructions for installing R and GDAL can be found HERE.

Installation Instructions

Installing on Windows

You can install the stable version of MODIStsp from CRAN:

install.packages("MODIStsp")

, or the development version (containing the latest improvements and bug fixes) from github:

library(devtools)
install_github("ropensci/MODIStsp")

Note that if the GTK+ library is not already installed on your system, installation may fail. In that case, please install and load the gWidgetsRGtk2 library beforehand:

install.packages("gWidgetsRGtk2")
library(gWidgetsRGtk2)

Upon loading gWidgetsRGtk2, an error window will probably appear. This signals that library “GTK+” is not yet installed on your system or is not on your PATH. To install it press “OK”. A new window dialog window will appear, asking if you. want to install “GTK+”. Select “Install GTK” and then “OK” . Windows will download and install the GTK+ library. When it finishes, the RSession should be restarted and you should be ready to go !

In case RStudio does not automatically restart or continuously asks to install GTK+ again, kill it form “Task Manager” (or restart the R session from RStudio “Session” menu), reload RStudio and the try to reload gWidgetsRGtk2. If it loads correctly, you should be ready to go.

If it still fails, try downloading the GTK+ bundle from:

http://ftp.gnome.org/pub/gnome/binaries/win64/gtk+/2.22/gtk+-bundle_2.22.1-20101229_win64.zip (OR http://ftp.gnome.org/pub/gnome/binaries/win32/gtk+/2.22/gtk+-bundle_2.22.1-20101227_win32.zip if on Win32)

, unzip the archive on a folder of your choice (e.g., C:\\Program Files\\GTK+), then add the path to its “bin” subfolder (e.g., C:\\Program Files\\GTK+\\bin\\ to your system PATH environment variable.

Restart your system and try loading again gWidgetsRGtk2: if it loads ok, you should be ready to install MODIStsp

Installing on Linux Systems

To install MODIStsp on Linux, you have to first install the following required dependencies:

  • Cairo >= 1.0.0, ATK >= 1.10.0, Pango >= 1.10.0, GTK+ >= 2.8.0, GLib >= 2.8.0 (required by package RGtk2)
  • Curl (required by package curl)
  • GDAL >= 1.6.3, PROJ.4 >= 4.4.9 (required by package rgdal)

On Debian and Ubuntu-based systems, to install those packages open a terminal and type:

sudo apt-get install r-cran-cairodevice r-cran-rgtk2 libcairo2-dev libatk1.0-dev libpango1.0-dev 
libgtk2.0-dev libglib2.0-dev libcurl4-openssl-dev libgdal-dev libproj-dev

On rpm-base systems, to install packages open a terminal and type:

sudo yum install libcairo2-devel libatk1.0-devel libpango1.0-devel gtk2 gtk2-devel 
glib2-devel libcurl-devel gdal-devel proj proj-devel proj-epsg proj-nad

Then, you can install the stable version of MODIStsp from CRAN:

install.packages("MODIStsp")

, or the development version (containing the latest improvements and bug fixes) from github;

library(devtools)
install_github("ropensci/MODIStsp")

Installing on Mac

NOTE: The following installation notes should be valid for MODIStsp installation on R 3.4.0 and above with Mac OSX Sierra. They were mainly taken (i.e., blatantly copied…) from: https://zhiyzuo.github.io/installation-rattle/. Thanks to Zhiya Zuo for providing this!

To properly install MODIStsp you will need to first install package RGTk2. This is a somehow difficult operation. The following instructions should help:

**1. Check your Mac OS X version and update if necessary: **

Enter the following command in terminal to check your macOS version. Expected output is as below the dashed line —.

~$ sw_vers  
------------------------  
ProductName:    Mac OS X  
ProductVersion: 10.12.6  
BuildVersion:   16G29

If your system is above 10.11, continue. Otherwise, upgrade it to Sierra.

Install homebrew if you do not have it already installed. homebrew is a very convenient package manager for macOS. To do so, open a terminal, copy the following command in it and hit Enter:

~$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Follow the instructions to get brew ready. When inserting your password, nothing will show up for security reasons. Just hit Enter when you are finished.

When brew is finished, copy the following command in terminal and hit Enter:

~$ touch ~/.bash_profile
~$ echo "export PATH=/usr/local/bin:$PATH
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/lib/pkgconfig/gtk+-2.0.pc:/opt/X11/lib/pkgconfig" >> ~/.bash_profile
~$ source ~/.bash_profile

2. Install the cairo library with x11 support. Enter the following into your terminal:

~$ brew uninstall --force cairo --ignore-dependencies
~$ brew cask install xquartz
~$ brew install --with-x11 cairo

3. Install the gtk+ library:

To do so, you first have to change the way homebrew wants to install gtk+. In an editor, write:

~$ brew edit gtk+

A text editor will open. Look in the file, and find a section that begins with “def install”. Substitute the current args section with the following text:

def install
 args = [
         "--disable-dependency-tracking",
         "--disable-silent-rules",
         "--prefix=#{prefix}",
         "--disable-glibtest",
         "--enable-introspection=yes",
         # "--disable-visibility",
         # "--with-gdktarget=quartz",
         "--with-gdktarget=x11",
         "--enable-x11-backend"
        ]

Save the modified file using ctrl+x ctrl+c, followed by y to quit emacs. Now install the library using:

~$ brew install --build-from-source --verbose gtk+

4. Update your path so that gtk+ is recognized, using:

~$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/lib/pkgconfig/gtk+-2.0.pc:/opt/X11/lib/pkgconfig

5. Install RGtk2 from source:

~$ cd ~/Downloads
~/Downloads$ R CMD INSTALL RGtk2_2.20.33.tar.gz

(Note that the name of the tar.gz file may vary depending on when you downloaded the file).

6. Open R and run:

library(RGtk2)

hopefully, RGtk2 will load without errors! If so, you should be ready to go, and you can:

7. Install MODIStsp from CRAN:

install.packages("MODIStsp")
MODIStsp()

or the development version from GitHub:

library(devtools) 
install_github("ropensci/MODIStsp", ref = "master")
MODIStsp()

Good luck!

Usage

The easiest way to use MODIStsp is to use its powerful GUI (Graphical User Interface) for selection of processing options, and then run the processing.

To open the GUI, load the package and launch the MODIStsp function, with no parameters:

library(MODIStsp)
MODIStsp()

This opens a GUI from which processing options can be specified (and eventually saved or loaded). After specifying all required parameters, clicking on “Start” will start the processing (see HERE for more detailed instructions).

MODIStsp can also be launched in non-interactive mode within an R session or script by setting the optional GUI parameter to FALSE, and the options_file parameter to the path of a previously saved JSON Options file. This allows to exploit MODIStsp functionalities within generic “R” processing scripts.

library(MODIStsp) 
# --> Specify the path to a valid options file saved in advance from MODIStsp GUI 
options_file <- "X:/yourpath/youroptions.json" 
  
# --> Launch the processing
MODIStsp(gui = FALSE, options_file = options_file)

(see HERE for more detailed instructions and examples).

Code of Conduct

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

ropensci_footer

News

MODIStsp 1.3.6

Main changes

Maintenance release to solve CRAN build errors on debian, due to the test_resetindexesx test. The test is now skipped on CRAN. Additionally, the MODIStsp_addindex function was modified to require explicit permission by the user to write on the MODIStsp_previous.json file

Bug fixing

  • Fixed a bug leading to errors if only "Aqua" platform was selected for download #133

MODIStsp 1.3.5

Main changes

Maintenance release to solve CRAN build errors on debian, due to the test_addindex test. The test is now skipped on CRAN. Additionally, the MODIStsp_addindex function was modified to require explicit permission by the user to write on the MODIStsp_previous.json file

Bug fixing

  • Fixed bug leading to errors in processing extent when switching products with different Native projection (4008 vs sinusoidal), the projection string was not properly updated. 77f5693e9

  • Fixed warnings on check for uniqueness in http addresses

MODIStsp 1.3.4

Main changes

Breaking changes

  • Due to improvements and changes in the GUI (see below), MODIStsp .json options files saved with older versions are no longer supported. Users will be informed of this if trying to use an obsolete json file.

  • Removed support for FTP download due to switch-off of NASA servers.

Updates in supported products

  • Removed all v005 and earlier products, due to discontinuation of their distribution by NASA servers

  • Added support for the following products:

    MCD64A1; MCD12C1; MCD18A1; MCD18A2; MCD12Q1; MOD44B; MOD44W; MCD12C1; MCD12Q1; MOD12A2; MOD12A3

Improvements in download functions

  • Improvements in GUI. It is now possible to set the processing extent interactively using the "Select on Map" button. This opens a browser window allowing to select and extent.

  • Use of httr::RETRY to improve behavior while navigating the servers to retrieve available files and while downloading hdf file (when use_aria == FALSE), thus removing dependency to RCurl.

Improvements in processing functions

  • Improved functionality for dealing with NoData for products with multiple fill-values. If "Change NoData" is set to "Yes", then in case a layer has multiple Nodata values all those values are set to NA in the output (see github.com/lbusett/MODIStsp#113)

Extensive code refactoring for submission to ropensci.

  • Long functions were simplified/split into smaller functions to allow for easier maintenance
  • GUI event handlers were moved into dedicated "R" files
  • Extensive code linting to abide to ropensci standards
  • Switch to jsonlite/xml2 to abide to ropensci standards
  • Removal of some less-used dependencies (e.g., hash)

Improvements in documentation and website

  • More detailed documentation for the main functions
  • Improvements in pkgdown articles to include some "worked examples" (e.g., MODIStsp_Extract)
  • New article in pkgdown showing a list of supported products

Improvements in test coverage

  • Several new tests added, bringing coverage above 90%

New functions

  • Added MODIStsp_resetindexes to remove all custom indexes from a MODIStsp json options file and MODIStsp_reset_options to reset MODIStsp options to default.

Bug fixing

MODIStsp 1.3.3.1 Release Notes

This was mostly a maintenance release - fixing a bug in 1.3.3 submission related to a missing import in NAMESPACE

Minor Changes

Improved organization of Virtual Raster files and RData files in the "Time_Series" output subfolder. Now virtual files and RData files are organized by sensor and layer to facilitate access.

MODIStsp 1.3.3 Release Notes

v1.3.3 was released on 10/08/2017

Major Changes

  • Improved speed in computation of spectral indexes, quality indicators and in computation of scaled variables by usind raster::calc() and raster::overlay (commits 0f5d76d, 0f5d76d, e462721)

  • Added functionality for unit testing using testthat and codecov integration. (commit 0c00fc6 and others)

  • Added "testing mode" to allow users to test proper functioning. Now, running MODIStsp(test = X) (with X in (0,6)) runs the processing using default processing parameters (commit 0c00fc6 and others)

Minor Changes

  • Suppression of verbose messages and (useless) warning messages while parsing the NASA servers and downloading data using "ftp" ( 3775d60);

Bug fixing

  • Fixed a bug preventing the "last" choice (or that present in the json file) from correctly showing in the GUI upon launch/restore of a saved json file (commit 633c2dd)

  • Fixed a bug affecting MODIS layers coded as Unsigned Integer - 32 bit (Thanks to Rob Critchlow for signaling this). The bug was due to improper handling of UInt32 data in gdalbuildvrt, causing sometimes an incorrect translation from HDF to output formats (#72).

    M*D09A1 - 500m Reflectance Band Quality (V005 and V006); M*DO9CMG - Coarse Resolution QA (V005 and V006); M*D09CMG - Coarse Resolution Number Mapping (V006); M*D09GA - 500m Reflectance Band Quality (V005 and V006); M*DOCGA - Band quality for MODIS bands 8-15 (V006); M*D11C3 - Days with clear-sky conditions and validated LSTs; Nights with clear-sky conditions and validated LSTs (V005 and V006); MCD43A2 - BRDF_Albedo_Band_Quality (V005 and V006).

  • Fixed a bug affecting creation of time series files (RData and virtual rasters) on all MCD products (#77)

  • Fixed a bug a error on creation of "burn_date" layers for MCD45A1 product (#77)

  • Fixed bugs on specifying spatial extent files on non-interactive execution (#75)

17/04/2017 - MODIStsp is now on CRAN !

MODIStsp was recently accepted on CRAN. From now on, you can install it simply using

install.packages("MODIStsp")

You'll however still be able to install the development version from github, containing the last improvements and bug fixing using:

install_github("lbusett/MODIStsp", ref = "master")

MODIStsp 1.3.2 Release Notes

v1.3.2 was released on 22/03/2017

Major Changes:

  • Added functionality to apply scale and offset coefficients on MODIS original values according with the specifications of single MODIS products.

Details:

  • MODIS hdf datasets are always stored as integer values, with scales factor and/or offsets to apply in order to convert to the indicated measure units reported in the products' documentation.

  • Starting from v1.3.2:

    • Leaving the "Scale output values" option to "No", output files are left as provided by NASA, and additional indices are produced as integer values with a 10000 factor scale;
    • Setting the "Scale output values" option to "Yes", scale factor and offsets are applied if existing (for example, in this case Land Surface Temperature values in the output raster will be in °K), and spectral indices are floating point values (for example, NDVI is between -1 and 1, etcetera).

Minor Changes:

  • Some product names and output layer names were modified to reduce the length of output file names, homogenize the names of the outputs and correct some errors. For compatibility with already created output files (versions up to 1.3.1), the old "XML" file specifying output files format is still available in inst/ExtData/MODIStsp_ProdOpts_old_v1.3.1.xml. To use the old file naming conventions, users have to:

    1. delete inst/ExtData/MODIStsp_ProdOpts.xml and rename MODIStsp_ProdOpts_old_v1.3.1.xml to MODIStsp_ProdOpts.xml.
    2. delete MODIStsp_ProdOpts.RData from the Previous folder within your_R-library_path/MODIStsp/Previous
    3. Restart MODIStsp

  • Timeouts of httr/ftp requests were increased to prevent problems on download on slow connections

Bug fixing:

MODIStsp 1.3.1 Release Notes

v1.3.1 was released on 13/02/2017

Major Changes

  • Added functionality for processing of Snow Cover datasets: MOD10A1, MOD10A2, MOD10C1, MOD10C2, MOD10CM (Issue #55) on devel

  • Added functionality for downloading "partial" years (Issue #54) on devel

  • Added functionality for computing vegetation indexes on MCD43A4 (v5-v6), MCD43B4 (v5), MCD43C4 (v5-v6) (Issue #59) on master/devel

  • Added functionality for accelerating download using aria2c (Issue #55) on devel

Bug fixing

  • Fixed bug on download with aria, throwing an error on partial download on http download with aria (6fbc875)

  • Fixed bug on M*D15A2 processing (Issue #60) on devel/master

  • Fixed bug on MCD12Q1 processing (Issue #58) on devel/master

  • Fixed bug on MOD13C2 processing (Issue #52) on devel/master

  • Fixed bug on insertion of custom projection (Issue #57) on devel/master

  • Fixed bug on selection of custom index (Issue #53) on devel/master

MODIStsp 1.3.0 Release Notes

v1.3.0 was released on 11/05/2016

Major Changes

  • Added functionality for downloading and preprocessing MODIS collection 006 datasets. For products with both 005 and 006 collections, the user can select the version using a new droplist in the GUI.

  • Added functionality for off-line processing. This allows both to i) reprocessing already downloaded data (for example, to create time series for an additional layer) without the need to connect to NASA servers, and ii) process HDF files downloaded outside MODIStsp (e.g., directly from NASA ftp) and stored on the user's PC, without the need of an active internet connection.

  • Improved the way in which options are saved. Much more readable. JSON files are now used instead than .RData. User options are no longer saved alongside products characteristics. This will allow to re-use an "old" options file even if changes are made on the XML file describing the products.

  • Improved the GUI interface for specifying additional Spectral Indexes. Hints are now showed to the user, and multiple indexes can be added in the same session.

Minor Changes

  • General improvements in the GUI interface. Products are now grouped by categories, to allow easier identification and selection.

  • Improvements in the README file and vignettes, providing more instructions on package use.

  • Improved functionality for checking for "complete" download, by comparing the size of the downloaded files with that of files on the server.

  • Added "configure" file for Linux installation.

  • Temporary files necessary for processing (e.g., vrt files) are now created (and destroyed) within the "R" temporary folder.

  • Miscellaneous bug-fixing

MODIStsp 1.2.1 Release Notes

v1.2.1 was released on 11/05/2016

Major Changes

  1. Modified format of "R" output time series from rts objects to RasterStack objects with temporal information added in the "z" attribute via setZ()

  2. Major changes/improvements in MODIStsp_extract function:

    • Use of plain RasterStack with "z" attribute instead than rasterstackts
    • Use of gdal_rasterize (gdalUtils) instead of rasterize (rgdal) to improve speed. Temporary shapes and rasters necessay are saved in "R" temporary folder and removed automatically
    • Fixed bugs on functionality for point/lines shapefiles, according to what specified by the "small" and "small_method" parameters
    • Added functionality for retrieving data for small polygons
    • Added out_format selection - xts or plain data.frame
    • Added possibility to use a shp filename as input directly
    • Added conformity checks on inputs
    • Added functionality to run without specifying start and end dates
    • Added id_field parameter for choosing which column of the input SP object should be used for "naming" the columns of the output

  3. Removed possibility to use "complex" resampling methods when reprojecting (e.g., bilinear, cubic, etc.) to avoid incorrect resampling on categorical variables and "contamination" of good pixels data.

Minor Changes

  • Changed the input method for starting and ending dates selection in the GUI. Now a text field is used
  • Added functionality for writing data ignore value on ENVI files
  • Removed automatic deletion of XML files created by writeRaster to keep metadata information
  • Changed names of products in the GUI for products with both TERRA and AQUA dataset to M*D09A1, M*D13Q1, etc...
  • Modified code syntax to satisfy R code styling guidelines
  • Modified roxygen parameters so that only required functions are imported from dependent packages
  • Updated and corrected the list of dependencies
  • Updated required "R" version to 3.2, and minimum versions for dependent packages to current versions.
  • Added Welcome message
  • Updated links to LPDAAC product description pages
  • Changed all "print" and "cat" calls to show messages/warnings to "message" or "warning" to allow easy disabling MODIStsp verbose messages
  • Using "R" tempfile/tempdir to save vrt files

Bug Fixes

  • Corrected a bug that threw an error in case incorrect bounding box specified

MODIStsp 1.2.0 Release Notes

v1.2.0 was released on 29/07/2015

Major changes

First stable release of advanced implementation of MODIStsp ! We know it should be 1.0.0, but thats'it !

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.
install.packages("MODIStsp")

1.3.7 by Lorenzo Busetto, 3 months ago

https://github.com/ropensci/MODIStsp

Report a bug at https://github.com/ropensci/MODIStsp/issues

Browse source code at https://github.com/cran/MODIStsp

Authors: Lorenzo Busetto [aut, cre] , Luigi Ranghetti [aut] , Leah Wasser [rev] (Leah Wasser reviewed the package for rOpenSci , see https://github.com/ropensci/onboarding/issues/184) , Jeff Hanson [rev] (Jeff Hanson reviewed the package for rOpenSci , see https://github.com/ropensci/onboarding/issues/184)

Documentation:   PDF Manual  

Task views: Hydrological Data and Modeling

GPL-3 license

Imports bitops, data.table, gdalUtils, gWidgets, gWidgetsRGtk2, methods, httr, jsonlite, pacman, parallel, raster, rgdal, rgeos, sp, stringr, xts, xml2

Suggests testthat, spelling, knitr, rmarkdown, png, grid, httptest, sf, leaflet, shiny, mapview, mapedit

System requirements: Cairo >= 1.0.0, ATK (>= 1.10.0), Pango (>= 1.10.0), GTK+ (>= 2.8.0), GLib (>= 2.8.0), Curl, GDAL (>= 1.6.3), PROJ.4 (>= 4.4.9)

See at CRAN

R links

R homepage

Download R

Mailing lists

R documentation

R manuals

R FAQs

The R Journal

CRAN links

CRAN homepage

CRAN repository policy

Submit a package

METACRAN stuff

About METACRAN

At github

Report a bug