MODIStsp
A Tool for Automating Download and Preprocessing of MODIS Land Products Data
Allows automating the creation of time series of rasters derived from MODIS Satellite Land Products data. It performs several typical preprocessing steps such as download, mosaicking, reprojection and resize of data acquired on a specified time period. All processing parameters can be set using a user-friendly GUI. Users can select which layers of the original MODIS HDF files they want to process, which additional Quality Indicators should be extracted from aggregated MODIS Quality Assurance layers and, in the case of Surface Reflectance products , which Spectral Indexes should be computed from the original reflectance bands. For each output layer, outputs are saved as single-band raster files corresponding to each available acquisition date. Virtual files allowing access to the entire time series as a single file are also created. Command-line execution exploiting a previously saved processing options file is also possible, allowing to automatically update time series related to a MODIS product whenever a new image is available.
MODIStsp
MODIStsp is a "R" package devoted to automatizing the creation of time series of rasters derived from MODIS Land Products data. MODIStsp allows to perform several preprocessing steps (e.g., download, mosaicing, reprojection and resize) on MODIS data available within a given time period. Users have the ability to select which specific layers of the original MODIS HDF files they want to process. They also can select which additional Quality Indicators should be extracted from the aggregated MODIS Quality Assurance layers and, in the case of Surface Reflectance products, which Spectral Indexes should be computed from the original reflectance bands. For each output layer, outputs are saved as single-band raster files corresponding to each available acquisition date. Virtual files allowing access to the entire time series as a single file can be also created. All processing parameters can be easily selected with a user-friendly GUI, although non-interactive execution exploiting a previously created Options File is possible. Stand-alone execution outside an "R" environment is also possible, allowing to use scheduled execution of MODIStsp to automatically update time series related to a MODIS product and extent whenever a new image is available.
MODIStsp is developed and maintained by L.Busetto and L.Ranghetti, from the Institute of Remote Sensing of Environment - National Research Council - Italy (CNR-IREA)
Important News ! MODIStsp content migrated to our new website !
-
11/08/2017 - MODIStp 1.3.3 was released today. It provides improvements in processing speed, as well as the usual bug fixes (thanks to all the users that signalled problems !). Check the Release Notes for further details !
-
25/07/2017 - As of today, most of the content related to MODIStsp has been moved to our new website at lbusett.github.io/MODIStsp, which provides a much better user interface and ease of access to MODIStsp-related information. From now on, please consult the new website for detailed and updated information on the package.
-
Also our previous FAQ page on github containing info for solving common installation, downloading and processing problems and issues was discontinued and migrated at lbusett.github.io/MODIStsp/articles/faq.html.
Problems and Issues
- Please report any issues you may encounter in our issues page on github .
System Requirements
MODIStsp requires R v >= 3.2.1 and GDAL (Geospatial Data Abstraction Library) v >= 1.11.1 with support for HDF4 raster format to be installed in your system. Brief instructions for installing R and GDAL can be found HERE.
Installing on Windows
You can install the stable version of MODIStsp from CRAN:
install.packages("MODIStsp")
, or the development version (containing the latest improvements and bug fixes) from github:
library(devtools)install_github("lbusett/MODIStsp")
Note that if the GTK+ library is not already installed on your system, installation may fail. In that case, please install and load the gWidgetsRGtk2 library beforehand:
install.packages("gWidgetsRGtk2")library(gWidgetsRGtk2)
Upon loading gWidgetsRGtk2, an error window will probably appear. This signals that library "GTK+" is not yet installed on your system or is not on your PATH. To install itpress "OK". A new window dialog window will appear, asking if you want to install "GTK+". Select "Install GTK" and then "OK" . Windows will download and install the GTK+ library. When it finishes, the RSession should be restarted and you should be ready to go !
In case RStudio doesn't automatically restart or continuously asks to install GTK+ again, kill it form "Task Manager" (or restart the R session from RStudio "Session" menu), reload RStudio and the try to reload gWidgetsRGtk2. If it loads correctly, you should be ready to go.
If it still fails, try downloading the GTK+ bundle from:
http://ftp.gnome.org/pub/gnome/binaries/win64/gtk+/2.22/gtk+-bundle_2.22.1-20101229_win64.zip (OR http://ftp.gnome.org/pub/gnome/binaries/win32/gtk+/2.22/gtk+-bundle_2.22.1-20101227_win32.zip if on Win32)
, unzip the archive on a folder of your choice (e.g., C:\\Program Files\\GTK+), then add the path to its "bin" subfolder (e.g., C:\\Program Files\\GTK+\\bin\\ to your system PATH environment variable.
Restart your system and try loading again gWidgetsRGtk2: if it loads ok, you should be ready to install MODIStsp
Installing on Linux Systems
To install MODIStsp on Linux, you have to first install the following required dependencies:
Cairo>= 1.0.0,ATK>= 1.10.0,Pango>= 1.10.0,GTK+>= 2.8.0,GLib>= 2.8.0 (required by packageRGtk2)Curl(required by packagecurl)GDAL>= 1.6.3,PROJ.4>= 4.4.9 (required by packagergdal)
On Debian and Ubuntu-based systems, to install those packages open a terminal and type:
sudo apt-get install r-cran-cairodevice r-cran-rgtk2 libcairo2-dev libatk1.0-dev libpango1.0-devlibgtk2.0-dev libglib2.0-dev libcurl4-openssl-dev libgdal-dev libproj-dev
On rpm-base systems, to install packages open a terminal and type:
sudo yum install libcairo2-devel libatk1.0-devel libpango1.0-devel gtk2 gtk2-develglib2-devel libcurl-devel gdal-devel proj proj-devel proj-epsg proj-nad
Then, you can install the stable version of MODIStsp from CRAN:
install.packages("MODIStsp")
, or the development version (containing the latest improvements and bug fixes) from github;
library(devtools)install_github("lbusett/MODIStsp")
Installing on Mac
NOTE: The following installation notes were kindly provided by a MODIStsp user and should be valid for MODIStsp installation on R 3.4.0 and above with Mac OSX Sierra. Since we are not working on Mac we were not able yet to check this, but we hope it may be useful at least to put you in the right direction if you have installation problems)
Installing MODIStsp requires many dependencies:
For installation on MAC OSX sierra, there are three main issues: - As outlined here in the comment by tobybot11 (https://gist.github.com/sebkopf/9405675), Rgtk requires the x11 libraries/headers (gdk/gdkx.h specifically) and doesn't work with the quartz libraries/headers which now are the default for GTK - When installing the dependencies gWidgetsRGtk2 and cairoDevice from CRAN you need to choose the version "OS X Mavericks binaries" and not "Package source" - You have to be sure that gdal is installed with HDF4 support.
1 - Update to R > 3.4 if needed, then update all packages
update.packages()# install the development version of devtools:install.packages(c("devtools"))devtools::install_github("hadley/devtools")
2 - Now, install RGtk2 using Homebrew (https://gist.github.com/sebkopf/9405675). First, ensure you have cairo installed with "--with-x11". Open a terminal and run:
brew uninstall cairo --ignore-dependenciesbrew install --with-x11 cairo
next, edit the configure options for GTK to require x11 rather than Quartz:
brew edit gtk+
in the def install section, remove the reference to quartz and switch to:
--with-gdktarget=x11,
--enable-x11-backend
Now install:
brew install --build-from-source --verbose gtk+export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/lib/pkgconfig/gtk+-2.0.pc:/opt/X11/lib/pkgconfig
3 - next, install RGtk2. Since install.packages("Rgtk2") is not going to work, go here, download RGtk2_2.20.33.tar.gz and, from a terminal run:
R CMD INSTALL RGtk2_2.20.33.tar.gz
Now, open R and run:
library(RGtk2)
hopefully, RGTk2 will load without errors !
4 - Install packages gWidgetsRGtk2 and cairoDevice
Very important !!!! from CRAN, download the "OS X Mavericks binaries" for gWidgetsRGtk2 and cairoDevice( not "Package source"). Save both to Frameworks/R.framework/3.4..., open R and run the code below (This will also install cairoDevice)
install.packages("gWidgetsRGtk2", lib="~/Library/Frameworks/R.framework/Versions/3.4/Resources/library/gWidgetsRGtk2")library(gWidgetsRGtk2)library(cairoDevice)
(This will work for R version 3.4, mac OS X Sierra)
5 - Install GDAL with HDF4 support
Check that gdal is installed with hdf4 support. From a terminal:
if gdal is installed, check what drivers are installed: the list should include hdf4.
If gdal is not yet installed or hdf4 is not supproted, install/reinstall it following these notes
> brew install hdf4# prefer hdf4 links over NetCDF> brew link --overwrite hdf4> brew install gdal --complete --enable-unsupported --with-hdf4# check what drivers are installed, list should now include hdf4:> gdal-config --formats
6 - Install rgdal
since rgdal is not usually looking in "/usr/local/lib"" you must include that with install.packages():
install.packages('rgdal',type = "source",configure.args = c('--with-proj-include=/usr/local/include','--with-proj-lib=/usr/local/lib'))
7 - Finally, install MODIStsp from CRAN or GitHub:
install.packages("MODIStsp")MODIStsp()
OR
library(devtools)install_github("lbusett/MODIStsp", ref = "master")MODIStsp()
News
MODIStsp 1.3.3.1 Release Notes
This is mostly a maintenance release - fixing a bug in 1.3.3 submission related to a missing import in NAMESPACE
Minor Changes
Improved organization of Virtual Raster files and RData files in the "Time_Series" output subfolder. Now virtual files and RData files are organized by sensor and layer to facilitate access.
MODIStsp 1.3.3 Release Notes
v1.3.3 was released on 10/08/2017
Major Changes
-
Improved speed in computation of spectral indexes, quality indicators and in computation of scaled variables by usind
raster::calc()andraster::overlay(commits 0f5d76d, 0f5d76d, e462721) -
Added functionality for unit testing using
testthatand codecov integration. (commit 0c00fc6 and others) -
Added "testing mode" to allow users to test proper functioning. Now, runnning
MODIStsp(test = X)(with X in (0,6)) runs the processing using default processing parameters (commit 0c00fc6 and others)
Minor Changes
- Suppression of verbose messages and (useless) warning messages while parsing the NASA servers and downloading data using "ftp" ( 3775d60);
Bug fixing
-
Fxed a bug preventing the "last" choice (or that present in the json file) from correctly showing in the GUI upon launch/restore of a saved json file (commit 633c2dd)
-
Fixed a bug affecting MODIS layers coded as Unsigned Integer - 32 bit (Thanks to Rob Critchlow for signaling this). The bug was due to improper handling of UInt32 data in
gdalbuildvrt, causing sometimes an incorrect translation from HDF to output formats (#72).M*D09A1 - 500m Reflectance Band Quality (V005 and V006); M*DO9CMG - Coarse Resolution QA (V005 and V006); M*D09CMG - Coarse Resolution Number Mapping (V006); M*D09GA - 500m Reflectance Band Quality (V005 and V006); M*DOCGA - Band quality for MODIS bands 8-15 (V006); M*D11C3 - Days with clear-sky conditions and validated LSTs; Nights with clear-sky conditions and validated LSTs (V005 and V006); MCD43A2 - BRDF_Albedo_Band_Quality (V005 and V006).
-
Fixed a bug affecting creation of time series files (RData and virtual rasters) on all MCD products (#77)
-
Fixed a bug a error on creation of "burn_date" layers for MCD45A1 product (#77)
-
Fixed bugs on specifying spatial extent files on non-interactive execution (#75)
17/04/2017 - MODIStsp is now on CRAN !
MODIStsp was recently accepted on CRAN. From now on, you can install it simply using
install.packages("MODIStsp")
You'll however still be able to install the development version from github,
containing the last improvements and bug fixing using:
install_github("lbusett/MODIStsp", ref = "master")
MODIStsp 1.3.2 Release Notes
v1.3.2 was released on 22/03/2017
Major Changes:
- Added functionality to apply scale and offset coeeficients on MODIS original values according with the specifications of single MODIS products.
Details:
-
MODIS hdf datasets are always stored as integer values, with scales factor and/or offsets to apply in order to convert to the indicated measure units reported in the products' documentation.
-
Starting from v1.3.2:
- Leaving the "Scale output values" option to "No", output files are left as provided by NASA, and additional indices are produced as integer values with a 10000 factor scale;
- Setting the "Scale output values" option to "Yes", scale factor and offsets are applied if existing (for example, in this case Land Surface Temperature values in the output raster will be in °K), and spectral indices are floating point values (for example, NDVI is between -1 and 1, etcetera).
Minor Changes:
-
Some product names and output layer names were modified to reduce the length of output file names, homogenize the names of the outputs and correct some errors. For compatibility with already created output files (versions up to 1.3.1), the old "XML" file specifying output files format is still available in
inst/ExtData/MODIStsp_ProdOpts_old_v1.3.1.xml. To use the old file naming conventions, users have to:- delete
inst/ExtData/MODIStsp_ProdOpts.xmland renameMODIStsp_ProdOpts_old_v1.3.1.xmltoMODIStsp_ProdOpts.xml. - delete
MODIStsp_ProdOpts.RDatafrom thePreviousfolder withinyour_R-library_path/MODIStsp/Previous - Restart
MODIStsp
- delete
-
Timeouts of httr/ftp requests were increased to prevent problems on download on slow connections
Bug fixing:
- Fixed bug on FTP download speed (Issue #65)
- Fixed bug on download of tile 0, preventing download of images with DOY 001 and of all "yearly based" products (e.g., MOD17)(Issue #64)
- Fixed other bugs affecting FTP download (https://github.com/lbusett/MODIStsp/commit/efbf1b469e7518ffc8a7ec6d9922242d6a5c228f, https://github.com/lbusett/MODIStsp/commit/1dc53a5ff5b355965acec86678a3104bd2d27fd9, https://github.com/lbusett/MODIStsp/commit/fa6c7b42eadce516a2f781604c9db28418120f36)
MODIStsp 1.3.1 Release Notes
v1.3.1 was released on 13/02/2017
Major Changes
-
Added functionality for processing of Snow Cover datasets: MOD10A1, MOD10A2, MOD10C1, MOD10C2, MOD10CM (Issue #55) on devel
-
Added functionality for downloading "partial" years(Issue #54) on devel
-
Added functionality for computing vegetation indexes on MCD43A4 (v5-v6), MCD43B4 (v5), MCD43C4 (v5-v6) (Issue #59) on master/devel
-
Added functionality for accelerating download using aria2c (Issue #55) on devel
Bug fixing
-
Fixed bug on download with aria, throwing an error on partial download on http downlaod with aria (6fbc875)
-
Fixed bug on M*D15A2 processing (Issue #60) on devel/master
-
Fixed bug on MCD12Q1 processing (Issue #58) on devel/master
-
Fixed bug on MOD13C2 processing (Issue #52) on devel/master
-
Fixed bug on insertion of custom projection (Issue #57) on devel/master
-
Fixed bug on selection of custom index (Issue #53) on devel/master
MODIStsp 1.3.0 Release Notes
v1.3.0 was released on 11/05/2016
Major Changes
-
Added functionality for downloading and preprocessing MODIS collection 006 datasets. For products with both 005 and 006 collections, the user can select the version using a new droplist in the GUI.
-
Added functionality for off-line processing. This allows both to i) reprocessing already downloaded data (for example, to create time series for an additional layer) without the need to connect to NASA servers, and ii) process HDF files downloaded outside MODIStsp (e.g., directly from NASA ftp) and stored on the user's PC, without the need of an active internet connection.
-
Improved the way in which options are saved. Much more readable .JSON files are now used instead than .RData. User options are no longer saved alongside products characteristics. This will allow to re-use an "old" options file even if changes are made on the XML file descriving the products.
-
Improved the GUI inteface for specifying additional Spectral Indexes. Hints are now showed to the user, and multiple indexes can be added in the same session.
Minor Changes
-
General improvements in the GUI inteface. Products are now grouped by categories, to allow easier identification and selection.
-
Improvements in the README file and vignettes, providing more instructions on package use.
-
Improved functionality for checking for "complete" download, by comparing the size of the downloaded files with that of files on the server.
-
Added "configure" file for Linux installation.
-
Temporary files necessary for processing (e.g., vrt files) are now created (and destroyed) within the "R" temporary folder.
-
Miscellaneous bug-fixing
MODIStsp 1.2.1 Release Notes
v1.2.1 was released on 11/05/2016
Major Changes
-
Modified format of "R" output time series from rts objects to RasterStack objects with temporal information added in the "z" attribute via setZ()
-
Major changes/improvements in MODIStsp_extract function:
- Use of plain rasterstack with "z" attribute instead than rasterstackts
- Use of gdal_rasterize (gdalUtils) instead of rasterize (rgdal) to improve speed. Temporary shapes and rasters necessay are saved in "R" temporary folder and removed automatically
- Fixed bugs on functionality for point/lines shapefiles, according to what specified by the "small" and "small_method" parameters
- Added functionality for retrieving data for small polygons
- Added out_format selection - xts or plain data.frame
- Added possibility to use a shp filename as input directly
- Added conformity checks on inputs
- Added functionaluity to run without specifying start and end dates
- Added id_field parameter for choosing which column of the input SP object should be used for "naming" the columns of the output
-
Removed possibility to use "complex" resampling methods when reprojecting (e.g., bilinear, cubic, etc.) to avoid incorrect resampling on categorical variables and "contamination" of good pixels data.
Minor Changes
- Changed the input method for starting and ending dates selection in the GUI. Now a text field is used
- Added functionaluty for writing data ignore value on ENVI files
- Removed automatic deletion of XML files created by writeRaster to keep metadata information
- Changed names of products in the GUI for products with both TERRA and AQUA dataset to M*D09A1, M*D13Q1, etc...
- Modified code syntax to satisfy R code styling guidelines
- Modified roxygen parameters so that only required functions are imported from dependent packages
- Updated and corrected the list of dependencies
- Updated required "R" version to 3.2, and minimum versions for dependent packages to current versions.
- Added Welcome message
- Updated links to LPDAAC product description pages
- Changed all "print" and "cat" calls to show messages/warnings to "message" or "warning" to allow easy disabling MODIStsp verbose messages
- Using "R" tempfile/tempdir to save vrt files
Bug Fixes
- Corrected a bug that threw an error in case incorrect bounding box specified
MODIStsp 1.2.0 Release Notes
v1.2.0 was released on 29/07/2015
Major changes
First stable release of advanced implementation of MODIStsp ! We know it should be 1.0.0, but thats'it !