Allows automating the creation of time series of rasters derived
from MODIS satellite land products data. It performs several typical
preprocessing steps such as download, mosaicking, reprojecting and resizing
data acquired on a specified time period. All processing parameters
can be set using a user-friendly GUI. Users can select which layers of
the original MODIS HDF files they want to process, which additional
quality indicators should be extracted from aggregated MODIS quality
assurance layers and, in the case of surface reflectance products,
which spectral indexes should be computed from the original reflectance
bands. For each output layer, outputs are saved as single-band raster
files corresponding to each available acquisition date. Virtual files
allowing access to the entire time series as a single file are also created.
Command-line execution exploiting a previously saved processing options
file is also possible, allowing users to automatically update time series
related to a MODIS product whenever a new image is available.
For additional documentation refer to the following article:
Busetto and Ranghetti (2016)
MODIStsp is a “R” package devoted to automatizing the creation of time series of rasters derived from MODIS Land Products data. MODIStsp allows to perform several preprocessing steps (e.g., download, mosaicing, reprojection and resize) on MODIS data available within a given time period. Users have the ability to select which specific layers of the original MODIS HDF files they want to process. They also can select which additional Quality Indicators should be extracted from the aggregated MODIS Quality Assurance layers and, in the case of Surface Reflectance products, which Spectral Indexes should be computed from the original reflectance bands. For each output layer, outputs are saved as single-band raster files corresponding to each available acquisition date. Virtual files allowing access to the entire time series as a single file can be also created. All processing parameters can be easily selected with a user-friendly GUI, although non-interactive execution exploiting a previously created Options File is possible. Stand-alone execution outside an “R” environment is also possible, allowing to use scheduled execution of MODIStsp to automatically update time series related to a MODIS product and extent whenever a new image is available.
For more information, documentation and examples of use, please see MODIStsp website at ropensci.github.io/MODIStsp
MODIStsp
is developed and maintained by Lorenzo Busetto and Luigi
Ranghetti, Institute of Remote Sensing of
Environment - National Research Council -
Italy (CNR-IREA)
To cite MODIStsp
please use:
L. Busetto, L. Ranghetti (2016) MODIStsp: An R package for automatic preprocessing of MODIS Land Products time series, Computers & Geosciences, Volume 97, Pages 40-48, ISSN 0098-3004, http://dx.doi.org/10.1016/j.cageo.2016.08.020, URL: https://github.com/ropensci/MODIStsp.
05/03/2019 - MODIStsp 1.3.8 is out. Fixes an issue causing incorrect application of scale/offset values on GDAL versions > 2.3 (https://github.com/ropensci/MODIStsp/issues/163) and adds support for products MOD21A1D.006 MOD21A1N.006 MOD21A2.006
29/11/2018 - We recently discovered a nasty bug in the computation of some custom spectral indices (those including additions / subtractions on reflectance values, such as in (b1_NIR+0.1) / b2_Red. See here for further details! The bug is fixed as of version 1.3.7.
07/08/2018 - We are glad to report that MODIStsp is now included in the rOpenSci packages’ ecosystem. We thank reviewers Leah Wasser and Jeffrey Hanson for their valuable reviews, which helped us to further improve the package!
10/07/2018 - MODIStsp v. 1.3.6 is out. Check out the Release Notes for further details !
20/06/2018 - MODIStsp v. 1.3.5 is out. Check out the Release Notes for further details !
11/04/2018 - Due to new NASA Policies the MODIS FTP servers were shut down starting, April 2, 2018. FTP download is therefore no longer working and will be removed in the next MODIStsp version!
11/04/2018 - Decommissioning of MODIS Version 5 Land Data Products. As per NASA notice above, MODIS v005 products are going to be decommissioned, and will soon be no longer available for download. Support for those products will be removed in the next MODIStsp version!.
11/08/2017 - MODIStp 1.3.3 was released today. It provides improvements in processing speed, as well as the usual bug fixes (thanks to all the users that signalled problems !). Check the Release Notes for further details !
25/07/2017 - As of today, **most of the content related to MODIStsp has been moved to our new website at ropensci.github.io/MODIStsp **, which provides a much better user interface and ease of access to MODIStsp-related information. From now on, please consult the new website for detailed and updated information on the package.
Also our previous FAQ page on github containing info for solving common installation, downloading and processing problems and issues was discontinued and migrated at ropensci.github.io/MODIStsp/articles/faq.html.
MODIStsp
requires R v >= 3.2.1 and
GDAL (Geospatial Data Abstraction Library) v >=
1.11.1 with support for HDF4 raster format to be installed in your
system. Brief instructions for installing R and GDAL can be found
HERE.
You can install the stable version of MODIStsp
from CRAN:
install.packages("MODIStsp")
, or the development version (containing the latest improvements and bug fixes) from github:
library(devtools)install_github("ropensci/MODIStsp")
Note that if the GTK+
library is not already installed on your
system, installation may fail. In that case, please install and load
the gWidgetsRGtk2
library beforehand:
install.packages("gWidgetsRGtk2")library(gWidgetsRGtk2)
Upon loading gWidgetsRGtk2
, an error window will probably appear. This
signals that library “GTK+” is not yet installed on your system or is
not on your PATH. To install it press “OK”. A new window dialog window
will appear, asking if you. want to install “GTK+”. Select “Install GTK”
and then “OK” . Windows will download and install the GTK+ library. When
it finishes, the RSession should be restarted and you should be ready to
go !
In case RStudio does not automatically restart or continuously asks to
install GTK+ again, kill it form “Task Manager” (or restart the R
session from RStudio “Session” menu), reload RStudio and the try to
reload gWidgetsRGtk2
. If it loads correctly, you should be ready to
go.
If it still fails, try downloading the GTK+ bundle from:
http://ftp.gnome.org/pub/gnome/binaries/win64/gtk+/2.22/gtk+-bundle_2.22.1-20101229_win64.zip (OR http://ftp.gnome.org/pub/gnome/binaries/win32/gtk+/2.22/gtk+-bundle_2.22.1-20101227_win32.zip if on Win32)
, unzip the archive on a folder of your choice (e.g., C:\\Program Files\\GTK+
), then add the path to its “bin” subfolder (e.g.,
C:\\Program Files\\GTK+\\bin\\
to your system PATH environment
variable.
Restart your system and try loading again gWidgetsRGtk2
: if it loads
ok, you should be ready to install
MODIStsp
To install MODIStsp
on Linux, you have to first install the following
required dependencies:
Cairo
>= 1.0.0, ATK
>= 1.10.0, Pango
>= 1.10.0, GTK+
>=
2.8.0, GLib
>= 2.8.0 (required by package RGtk2
)Curl
(required by package curl
)GDAL
>= 1.6.3, PROJ.4
>= 4.4.9 (required by package rgdal
)On Debian and Ubuntu-based systems, to install those packages open a terminal and type:
sudo apt-get install r-cran-cairodevice r-cran-rgtk2 libcairo2-dev libatk1.0-dev libpango1.0-devlibgtk2.0-dev libglib2.0-dev libcurl4-openssl-dev libgdal-dev libproj-dev
On rpm-base systems, to install packages open a terminal and type:
sudo yum install libcairo2-devel libatk1.0-devel libpango1.0-devel gtk2 gtk2-develglib2-devel libcurl-devel gdal-devel proj proj-devel proj-epsg proj-nad
Then, you can install the stable version of MODIStsp from CRAN:
install.packages("MODIStsp")
, or the development version (containing the latest improvements and bug fixes) from github;
library(devtools)install_github("ropensci/MODIStsp")
NOTE: The following installation notes should be valid for MODIStsp installation on R 3.4.0 and above with Mac OSX Sierra. They were mainly taken (i.e., blatantly copied…) from: https://zhiyzuo.github.io/installation-rattle/. Thanks to Zhiya Zuo for providing this!
To properly install MODIStsp
you will need to first install package
RGTk2
. This is a somehow difficult operation. The following
instructions should help:
**1. Check your Mac OS X version and update if necessary: **
Enter the following command in terminal to check your macOS version. Expected output is as below the dashed line —.
~$ sw_vers
------------------------
ProductName: Mac OS X
ProductVersion: 10.12.6
BuildVersion: 16G29
If your system is above 10.11, continue. Otherwise, upgrade it to Sierra.
Install homebrew if you do not have it already installed. homebrew is a very convenient package manager for macOS. To do so, open a terminal, copy the following command in it and hit Enter:
~$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Follow the instructions to get brew ready. When inserting your password, nothing will show up for security reasons. Just hit Enter when you are finished.
When brew is finished, copy the following command in terminal and hit Enter:
~$ touch ~/.bash_profile~$ echo "export PATH=/usr/local/bin:$PATHexport PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/lib/pkgconfig/gtk+-2.0.pc:/opt/X11/lib/pkgconfig" >> ~/.bash_profile~$ source ~/.bash_profile
2. Install the cairo
library with x11 support. Enter the following
into your terminal:
~$ brew uninstall --force cairo --ignore-dependencies~$ brew cask install xquartz~$ brew install --with-x11 cairo
3. Install the gtk+
library:
To do so, you first have to change the way homebrew wants to install gtk+. In an editor, write:
~$ brew edit gtk+
A text editor will open. Look in the file, and find a section that
begins with “def install”. Substitute the current args
section with
the following text:
def installargs = ["--disable-dependency-tracking","--disable-silent-rules","--prefix=#{prefix}","--disable-glibtest","--enable-introspection=yes",# "--disable-visibility",# "--with-gdktarget=quartz","--with-gdktarget=x11","--enable-x11-backend"]
Save the modified file using ctrl+x ctrl+c
, followed by y
to quit
emacs. Now install the library using:
~$ brew install --build-from-source --verbose gtk+
4. Update your path so that gtk+
is recognized,
using:
~$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/lib/pkgconfig/gtk+-2.0.pc:/opt/X11/lib/pkgconfig
5. Install RGtk2
from source:
Download the newest source file for RGtk2 from https://CRAN.R-project.org/package=gWidgetsRGtk2.
Assuming that the path to this file is ~/Downloads. Run the following in terminal (change the path if you did not download in ~/Downloads):
~$ cd ~/Downloads~/Downloads$ R CMD INSTALL RGtk2_2.20.33.tar.gz
(Note that the name of the tar.gz file may vary depending on when you downloaded the file).
6. Open R and run:
library(RGtk2)
hopefully, RGtk2
will load without errors! If so, you should be ready
to go, and you can:
7. Install MODIStsp from CRAN:
install.packages("MODIStsp")MODIStsp()
or the development version from GitHub:
library(devtools)install_github("ropensci/MODIStsp", ref = "master")MODIStsp()
Good luck!
The easiest way to use MODIStsp
is to use its powerful GUI (Graphical
User Interface) for selection of processing options, and then run the
processing.
To open the GUI, load the package and launch the MODIStsp function, with no parameters:
library(MODIStsp)MODIStsp()
This opens a GUI from which processing options can be specified (and eventually saved or loaded). After specifying all required parameters, clicking on “Start” will start the processing (see HERE for more detailed instructions).
MODIStsp
can also be launched in non-interactive mode within an R
session or script by setting the optional GUI
parameter to FALSE, and
the options_file
parameter to the path of a previously saved JSON
Options file. This allows to exploit MODIStsp
functionalities within
generic “R” processing scripts.
library(MODIStsp)# --> Specify the path to a valid options file saved in advance from MODIStsp GUIoptions_file <- "X:/yourpath/youroptions.json"# --> Launch the processingMODIStsp(gui = FALSE, options_file = options_file)
(see HERE for more detailed instructions and examples).
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
Fixed an issue causing incorrect application of scale/offset values on GDAL versions > 2.3 (introduced by change of behaviour of gdal_buildvrt - https://trac.osgeo.org/gdal/ticket/3221#comment:5) - see https://github.com/ropensci/MODIStsp/issues/163
Added support for the following products: MOD21A1D.006 MOD21A1N.006 MOD21A2.006/
Fixed a bug leading to not properly reassigning NoData values when processing full tiles and using change_nodata = TRUE
Fixed inconsistencies in definition of characteristics of products MOD/MYD13C2/C1 and MOD/MYD13A3 (erroneous layers in xml)
Fixed a bug leading to help messages in the select layers portion of the GUI to not render
Updated MOD44B specifications to allow download of v006 version
Maintenance release to solve CRAN build errors on debian, due to the test_resetindexes test. The test is now skipped on CRAN. Additionally, the MODIStsp_addindex function was modified to require explicit permission by the user to write on the MODIStsp_previous.json file
Maintenance release to solve CRAN build errors on debian, due to the test_addindex test. The test is now skipped on CRAN. Additionally, the MODIStsp_addindex function was modified to require explicit permission by the user to write on the MODIStsp_previous.json file
Fixed bug leading to errors in processing extent when switching products with different Native projection (4008 vs sinusoidal), the projection string was not properly updated. 77f5693e9
Fixed warnings on check for uniqueness in http addresses
Due to improvements and changes in the GUI (see below), MODIStsp
.json options
files saved with older versions are no longer supported. Users will be informed of
this if trying to use an obsolete json file.
Removed support for FTP download due to switch-off of NASA servers.
Removed all v005 and earlier products, due to discontinuation of their distribution by NASA servers
Added support for the following products:
MCD64A1; MCD12C1; MCD18A1; MCD18A2; MCD12Q1; MOD44B; MOD44W; MCD12C1; MCD12Q1; MOD12A2; MOD12A3
Improvements in GUI. It is now possible to set the processing extent interactively using the "Select on Map" button. This opens a browser window allowing to select and extent.
Use of httr::RETRY
to improve behavior while navigating the servers to
retrieve available files and while downloading hdf file (when use_aria == FALSE),
thus removing dependency to RCurl.
MODIStsp_resetindexes
to remove all custom indexes from a MODIStsp
json options file and MODIStsp_reset_options
to reset MODIStsp options to
default.Fixed bug affecting extent selection when working with non-tiled (MCD) products https://github.com/ropensci/MODIStsp/issues/122
Fixed bugs affecting the "Seasonal" time series download
This was mostly a maintenance release - fixing a bug in 1.3.3 submission related to a missing import in NAMESPACE
Improved organization of Virtual Raster files and RData files in the "Time_Series" output subfolder. Now virtual files and RData files are organized by sensor and layer to facilitate access.
v1.3.3 was released on 10/08/2017
Improved speed in computation of spectral indexes, quality indicators and in
computation of scaled variables by using raster::calc()
and raster::overlay
(commits 0f5d76d, 0f5d76d, e462721)
Added functionality for unit testing using testthat
and codecov integration.
(commit 0c00fc6
and others)
Added "testing mode" to allow users to test proper functioning. Now, running
MODIStsp(test = X)
(with X in (0,6)) runs the processing using default processing
parameters (commit 0c00fc6 and others)
Fixed a bug preventing the "last" choice (or that present in the json file) from correctly showing in the GUI upon launch/restore of a saved json file (commit 633c2dd)
Fixed a bug affecting MODIS layers coded as Unsigned Integer - 32 bit (Thanks
to Rob Critchlow for signaling this). The bug was due to improper handling of
UInt32 data in gdalbuildvrt
, causing sometimes an incorrect translation from
HDF to output formats (#72).
M*D09A1 - 500m Reflectance Band Quality (V005 and V006); M*DO9CMG - Coarse Resolution QA (V005 and V006); M*D09CMG - Coarse Resolution Number Mapping (V006); M*D09GA - 500m Reflectance Band Quality (V005 and V006); M*DOCGA - Band quality for MODIS bands 8-15 (V006); M*D11C3 - Days with clear-sky conditions and validated LSTs; Nights with clear-sky conditions and validated LSTs (V005 and V006); MCD43A2 - BRDF_Albedo_Band_Quality (V005 and V006).
Fixed a bug affecting creation of time series files (RData and virtual rasters) on all MCD products (#77)
Fixed a bug a error on creation of "burn_date" layers for MCD45A1 product (#77)
Fixed bugs on specifying spatial extent files on non-interactive execution (#75)
MODIStsp was recently accepted on CRAN. From now on, you can install it simply using
install.packages("MODIStsp")
You'll however still be able to install the development
version from github,
containing the last improvements and bug fixing using:
install_github("ropensci/MODIStsp", ref = "master")
v1.3.2 was released on 22/03/2017
MODIS hdf datasets are always stored as integer values, with scales factor and/or offsets to apply in order to convert to the indicated measure units reported in the products' documentation.
Starting from v1.3.2:
Some product names and output layer names were modified to reduce the length of
output file names, homogenize the names of the outputs and correct some errors.
For compatibility with already created output files (versions up to 1.3.1),
the old "XML" file specifying output files format is still available in
inst/ExtData/MODIStsp_ProdOpts_old_v1.3.1.xml
. To use the old file naming
conventions, users have to:
inst/ExtData/MODIStsp_ProdOpts.xml
and rename MODIStsp_ProdOpts_old_v1.3.1.xml
to MODIStsp_ProdOpts.xml
.MODIStsp_ProdOpts.RData
from the Previous
folder within
your_R-library_path/MODIStsp/Previous
MODIStsp
Timeouts of httr/ftp requests were increased to prevent problems on download on slow connections
v1.3.1 was released on 13/02/2017
Added functionality for processing of Snow Cover datasets: MOD10A1, MOD10A2, MOD10C1, MOD10C2, MOD10CM (Issue #55) on devel
Added functionality for downloading "partial" years (Issue #54) on devel
Added functionality for computing vegetation indexes on MCD43A4 (v5-v6), MCD43B4 (v5), MCD43C4 (v5-v6) (Issue #59) on master/devel
Added functionality for accelerating download using aria2c (Issue #55) on devel
Fixed bug on download with aria, throwing an error on partial download on http download with aria (6fbc875)
Fixed bug on M*D15A2 processing (Issue #60) on devel/master
Fixed bug on MCD12Q1 processing (Issue #58) on devel/master
Fixed bug on MOD13C2 processing (Issue #52) on devel/master
Fixed bug on insertion of custom projection (Issue #57) on devel/master
Fixed bug on selection of custom index (Issue #53) on devel/master
v1.3.0 was released on 11/05/2016
Added functionality for downloading and preprocessing MODIS collection 006 datasets. For products with both 005 and 006 collections, the user can select the version using a new droplist in the GUI.
Added functionality for off-line processing. This allows both to i) reprocessing already downloaded data (for example, to create time series for an additional layer) without the need to connect to NASA servers, and ii) process HDF files downloaded outside MODIStsp (e.g., directly from NASA ftp) and stored on the user's PC, without the need of an active internet connection.
Improved the way in which options are saved. Much more readable. JSON files are now used instead than .RData. User options are no longer saved alongside products characteristics. This will allow to re-use an "old" options file even if changes are made on the XML file describing the products.
Improved the GUI interface for specifying additional Spectral Indexes. Hints are now showed to the user, and multiple indexes can be added in the same session.
General improvements in the GUI interface. Products are now grouped by categories, to allow easier identification and selection.
Improvements in the README file and vignettes, providing more instructions on package use.
Improved functionality for checking for "complete" download, by comparing the size of the downloaded files with that of files on the server.
Added "configure" file for Linux installation.
Temporary files necessary for processing (e.g., vrt files) are now created (and destroyed) within the "R" temporary folder.
Miscellaneous bug-fixing
v1.2.1 was released on 11/05/2016
Modified format of "R" output time series from rts objects to RasterStack objects with temporal information added in the "z" attribute via setZ()
Major changes/improvements in MODIStsp_extract function:
RasterStack
with "z" attribute instead than rasterstackts
Removed possibility to use "complex" resampling methods when reprojecting (e.g., bilinear, cubic, etc.) to avoid incorrect resampling on categorical variables and "contamination" of good pixels data.
v1.2.0 was released on 29/07/2015
First stable release of advanced implementation of MODIStsp ! We know it should be 1.0.0, but that's it !