Select and Download Climate Data from 'DWD' (German Weather Service)

Handle climate data from the 'DWD' ('Deutscher Wetterdienst', see < https://www.dwd.de/EN/climate_environment/cdc/cdc.html> for more information). Choose files with 'selectDWD()', download and process data sets with 'dataDWD()' and 'readDWD()'.

rdwd is an R package to select, download and read climate data from the German Weather Service (Deutscher Wetterdienst, DWD). They provide over 228 thousand datasets with weather observations online at ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate.

rdwd is available on CRAN:

It has been presented at FOSDEM 2017 and UseR!2017 in Brussels, featured in Rstudios data package list and written about in OSOR.

Usage of the package usually looks something like the following:

# download and install the rdwd package (only needed once):
install.packages("rdwd")
berryFunctions::instGit("brry/rdwd") # latest development version (if wanted)
 
# load the package into library (needed in every R session):
library(rdwd)
 
# select a dataset (e.g. last year's daily climate data from Potsdam City):
link <- selectDWD("Potsdam", res="daily", var="kl", per="recent")
 
# Actually download that dataset, returning the local storage file name:
file <- dataDWD(link, read=FALSE)
 
# Read the file from the zip folder:
clim <- readDWD(file)
 
# Inspect the data.frame:
str(clim)

You can also select datasets with the interactive map.
Installation instructions and more examples are available in the package vignette.

vignette("mapDWD") # interactive map, likely faster than CRAN link above
vignette("rdwd")   # package instructions and examples

A real-life usage example of the package can be found at https://github.com/brry/prectemp/blob/master/Code_analysis.R

help

I'm looking for someone to help implement multiple downloads in dataDWD via e.g. curl or wget. The requirements are as follows:

works cross-platform
is called from R
has as few dependencies as possible
optimally enables a progress bar

News

rdwd ToDo-list / wishlist / issues

dataDWD: potentially use curl to download data without sleep need
expand + automate localtests, e.g. with all non-cran-run examples
selectDWD: if res/var/per is NA or "", give a selection popup
for per="monthly", make readDWD recognize MESS_DATUM_ENDE and _BEGINN, potentially create new column MESS_DATUM yyyy-mm-15
readDWD: figure out reason for fread trouble (reported in emails by some users)
readVars: test for all FTP folders (one file each), automate column name addition
allow abbreviations of res hourly/daily/monthly

rdwd: DWD Climate Data access package by Berry Boessenkool, [email protected] Any feedback is welcome, per email or on github.com/brry/rdwd

rdwd Version history

Version 0.11.0 (2018-11-26)

dataDWD: argument force can now be NA to download files older than 24 hours
hourly/solar timestamp now processed automatically
use cases vignette added
vignettes precompiled + CRAN tests removed to reduce automated load on DWD Server
indexFTP: got much smarter in discerning files from folders
indexes expanded for new DWD data New function: readVars, along with parameter_abreviations

Version 0.10.0 (2018-03-26)

created new index with sub-hourly data, reflected in data overview at selectDWD
dataDWD,readDWD,selectDWD: file ending checks more elegant
createIndex: updated to handle the new files
some minor improvements, see https://github.com/brry/rdwd/commits

Version 0.9.0 (2017-11-03)

readDWD: readDWD.data and readDWD.meta are now separate (non-exported) functions
readDWD: fread default is now FALSE
dataDWD: fread and overwrite arguments added
references dutch meteo package and useR!2017 rdwd presentation
importFrom entries completed
DESCRIPTION: BugReports entry added

Version 0.8.0 (2017-06-09) reflects DWD FTP update June 1

indexFTP: ftp blockage chance reduced, data loss avoided, progress bar added
errors/warnings/messages now often include a traceback
metaInfo tells about non-public files
createIndex + readDWD read meta files in German locale to handle Umlaute
Index documentation and creation unified
Vignettes and readme improved
various minor improvements and fixes, see https://github.com/brry/rdwd/commits renamed: indexDWD -> indexFTP added: lldist, maxlldist, nearbyStations no longer exported in NAMESPACE: rowDisplay, dirDWD removed: fileDWD, geoIndexAll

Version 0.7.0 (2017-02-03)

interactive map now colored by availability of recent file
stations with slightly varying coordinates (<900 m apart) aggregated in geoIndex
documentation corrections, improved messages in fileDWD and dirDWD
map vignette expanded, package vignette abbreviated slightly
metaInfo printout greatly improved
duplicate index entries removed New function: rowDisplay New object: geoIndexAll Removed object: mapDWD (to reduce package size)

Version 0.6.1 (2017-01-24) Initial release of the package, development can be seen at https://github.com/brry/rdwd/compare/[email protected]{2016-10-19}[email protected]{2017-01-24}

'selectDWD' uses index information to find files matching a path or station criteria. With the returned path/file names, 'dataDWD' and 'readDWD' download and read data. 'dirDWD' and 'fileDWD' control that no file will be overwritten and path messages are useful. 'indexDWD' lists all the files on the FTP server with DWD data. 'createIndex' uses that list to create 'fileIndex', 'metaIndex' and 'geoIndex'. 'mapDWD' is an interactive leaflet map also useful to explore datasets.

Note: 'dataDWD' and 'readDWD' were started in June 2016 within my misc package berryFunctions, from which they will be deleted after rdwd is on CRAN