Web-Processing of Large Gridded Datasets

Processes gridded datasets found on the U.S. Geological Survey Geo Data Portal web application or elsewhere, using a web-enabled workflow that eliminates the need to download and store large datasets that are reliably hosted on the Internet. The package provides access to several data subset and summarization algorithms that are available on remote web processing servers.


Tools for geo-web processing of gridded data via the Geo Data Portal. geoknife slices up gridded data according to overlap with irregular features, such as watersheds, lakes, points, etc. The result is subsetted data in plain text, NetCDF, geotiff or other formats.

To install the geoknife from CRAN:

install.packages("geoknife")

To install the stable version of geoknife package with dependencies:

install.packages("geoknife", 
    repos = c("https://owi.usgs.gov/R","https://cran.rstudio.com/"),
    dependencies = TRUE)

Or to install the current development version of the package:

install.packages("devtools")
devtools::install_github('USGS-R/geoknife')

The geoknife package was created to support web-based geoprocessing of large gridded datasets according to their overlap with landscape (or aquatic/ocean) features that are often irregularly shaped. geoknife creates data access and subsequent geoprocessing requests for the USGS's Geo Data Portal to carry out on a web server. The results of these requests are available for download after the processes have been completed. This type of workflow has three main advantages: 1) it allows the user to avoid downloading large datasets, 2) it avoids reinventing the wheel for the creation and optimization of complex geoprocessing algorithms, and 3) computing resources are dedicated elsewhere, so geoknife operations do not have much of an impact on a local computer.

geoknife interacts with a remote server to figure out what types of processing capabilities are available, in addition to seeing what types of geospatial features are already available to be used as an area of interest (commonly, these are user-uploaded shapefiles). Because communication with web resources are central to geoknife operations, users must have an active internet connection.

The main elements of setting up and carrying out a geoknife 'job' (geojob) include defining the feature of interest (the stencil argument in the geoknife function), the gridded web dataset to be processed (the fabric argument in the geoknife function), and the the processing algorithm parameters (the knife argument in the geoknife function). The status of the geojob can be checked with check, and output can be loaded into a data.frame with result.

library(geoknife)
# from a single point
stencil <- simplegeom(c(-89, 46.23))
   # -- or --
# from a collection of named points
stencil <- simplegeom(data.frame(
              'point1' = c(-89, 46), 
              'point2' = c(-88.6, 45.2)))
   # -- or --
#for a state from a web available dataset
stencil <- webgeom('state::New Hampshire')
stencil <- webgeom('state::New Hampshire,Wisconsin,Alabama')
   # -- or --
#for HUC8s from a web available dataset
stencil <- webgeom('HUC8::09020306,14060009')
# from the prism dataset:
fabric <- webdata('prism')
   # -- or --
# explicitly define webdata from a list:
fabric <- webdata(list(
            times = as.POSIXct(c('1895-01-01','1899-01-01')),
            url = 'https://cida.usgs.gov/thredds/dodsC/prism_v2',
            variables = 'ppt'))
# modify the times field:
times(fabric) <- as.POSIXct(c('2003-01-01','2005-01-01'))
job <- geoknife(stencil, fabric, wait = TRUE)
 
# use existing convienence functions to check on the job:
check(job)
## $status
## [1] "Process successful"
## 
## $URL
## [1] "https://cida.usgs.gov:443/gdp/process/RetrieveResultServlet?id=056d8e89-6d18-418f-b62c-af76ded6b83dOUTPUT"
## 
## $statusType
## [1] "ProcessSucceeded"

see also:

running(job)
error(job)
successful(job)
data <- result(job)
plot(data[,1:2], ylab = variables(fabric))

job <- geoknife(webgeom('state::New Hampshire'), fabric = 'prism', email = 'fake.email@gmail.com')
FunctionTitle
geoknifeslice up gridded data according to overlap with feature(s)
gconfigset or query package settings for geoknife processing defaults
algorithmthe algorithm of a webprocess
attributethe attribute of an webgeom
checkcheck status of geojob
downloaddownload the results of a geojob
errorconvenience function for state of geojob
runningconvenience function for state of geojob
successfulconvenience function for state of geojob
startstart a geojob
cancelcancel a geojob
geomthe geom of a webgeom
inputsthe inputs of a webprocess
idthe process id of a geojob
valuesthe values of a webgeom
resultload the output of a completed geojob into data.frame
variablesthe variables for a webdata object
waitwait for a geojob to complete processing
timesthe times of a webdata object
urlthe url of a webdata, webgeom, geojob, or webprocess
versionthe version of a webgeom or webdata
xmlthe xml of a geojob
queryquery datasets or variables
ClassTitle
simplegeoma simple geometric class. Extends sp::SpatialPolygons
webgeoma web feature service geometry
webprocessa web processing service
webdataweb data
geojoba geo data portal processing job
datagroupa simple class that contains data lists that can be webdata

This version requires httr, sp, and XML. All of these packages are available on CRAN, and will be installed automatically when using the install.packages() instructions above.

This software is in the public domain because it contains materials that originally came from the U.S. Geological Survey, an agency of the United States Department of Interior. For more information, see the official USGS copyright policy

Although this software program has been used by the U.S. Geological Survey (USGS), no warranty, expressed or implied, is made by the USGS or the U.S. Government as to the accuracy and functioning of the program and related program material nor shall the fact of distribution constitute any such warranty, and no responsibility is assumed by the USGS in connection therewith.

This software is provided "AS IS."

News

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("geoknife")

1.5.4 by Jordan Read, a month ago


https://github.com/USGS-R/geoknife


Report a bug at https://github.com/USGS-R/geoknife/issues


Browse source code at https://github.com/cran/geoknife


Authors: Jordan Read [aut, cre], Jordan Walker [aut], Alison Appling [aut], David Blodgett [aut], Emily Read [aut], Luke Winslow [aut], Lindsay Carr [aut], David Watkins [aut]


Documentation:   PDF Manual  


CC0 license


Imports XML, methods, httr, curl, sp, utils

Suggests testthat, xtable, knitr, rmarkdown, ggmap, dplyr, rasterVis, ggplot2, rgdal, maps


See at CRAN