Select and Download Climate Data from 'DWD' (German Weather Service)

Handle climate data from the 'DWD' ('Deutscher Wetterdienst', see < https://www.dwd.de/EN/climate_environment/cdc/cdc.html> for more information). Choose files with 'selectDWD()', download and process data sets with 'dataDWD()' and 'readDWD()'.

rdwd is an R package to select, download and read climate data from the German Weather Service (Deutscher Wetterdienst, DWD). They provide over 228 thousand datasets with weather observations online at ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate.

rdwd is available on CRAN:

It has been presented at FOSDEM 2017 and UseR!2017 in Brussels, featured in Rstudios data package list and written about in OSOR.

Usage of the package usually looks something like the following:

# download and install the rdwd package (only needed once):
install.packages("rdwd")
berryFunctions::instGit("brry/rdwd") # latest development version (if wanted)
 
# load the package into library (needed in every R session):
library(rdwd)
 
# select a dataset (e.g. last year's daily climate data from Potsdam City):
link <- selectDWD("Potsdam", res="daily", var="kl", per="recent")
 
# Actually download that dataset, returning the local storage file name:
file <- dataDWD(link, read=FALSE)
 
# Read the file from the zip folder:
clim <- readDWD(file)
 
# Inspect the data.frame:
str(clim)

You can also select datasets with the interactive map.
Installation instructions and more examples are available in the package vignette.

vignette("mapDWD") # interactive map, likely faster than CRAN link above
vignette("rdwd")   # package instructions and examples

A real-life usage example of the package can be found at https://github.com/brry/prectemp/blob/master/Code_analysis.R

help

I'm looking for someone to help implement multiple downloads in dataDWD via e.g. curl or wget. The requirements are as follows:

works cross-platform
is called from R
has as few dependencies as possible
optimally enables a progress bar

News

rdwd ToDo-list / wishlist / issues

dataDWD: potentially use curl to download data without sleep need
readDWD: figure out reason for fread trouble (reported in emails by some users)

rdwd: DWD Climate Data access package by Berry Boessenkool, [email protected] Any feedback is welcome, per email or on github.com/brry/rdwd

rdwd Version history

Version 1.0.0 (2019-03-17) First major version increase! Tests are now systematic and comprehensive (albeit run locally and manually to reduce CRAN load and enable local file writing). selectDWD got a very noticable performance boost and an interactive res/var/per selection. Handling raster files is now supported in rdwd. Examples and vignettes have been refined. Detailed changes can be found on https://github.com/brry/rdwd/commits, here's the main overview:

dataDWD: dots now passed to readDWD, download errors checked and returned as informative warning
readDWD: reading functions added for binary, raster and multi_annual files, MESS_DATUM POSIXct column added in monthly data, list output now named
indexFTP: now removes duplicates from file list, sped up in default 1min/prec usage
rdwd now works fine with multi_annual and subdaily data
readVars: parameter abbreviations completed, output now visible
selectDWD: order of arguments changed, interactive res/var/per selection enabled, computing speed extremely increased
localtests.R heavily expanded, examples reduced and refined
indexes + vignettes updated, use cases expanded with nearbyStations example New functions and objects: readMeta, dwdbase (central package base url)

Version 0.11.0 (2018-11-26)

dataDWD: argument force can now be NA to download files older than 24 hours
hourly/solar timestamp now processed automatically
use cases vignette added
vignettes precompiled + CRAN tests removed to reduce automated load on DWD Server
indexFTP: got much smarter in discerning files from folders
indexes expanded for new DWD data New function: readVars, along with parameter_abreviations

Version 0.10.0 (2018-03-26)

created new index with sub-hourly data, reflected in data overview at selectDWD
dataDWD,readDWD,selectDWD: file ending checks more elegant
createIndex: updated to handle the new files
some minor improvements, see https://github.com/brry/rdwd/commits

Version 0.9.0 (2017-11-03)

readDWD: readDWD.data and readDWD.meta are now separate (non-exported) functions
readDWD: fread default is now FALSE
dataDWD: fread and overwrite arguments added
references dutch meteo package and useR!2017 rdwd presentation
importFrom entries completed
DESCRIPTION: BugReports entry added

Version 0.8.0 (2017-06-09) reflects DWD FTP update June 1

indexFTP: ftp blockage chance reduced, data loss avoided, progress bar added
errors/warnings/messages now often include a traceback
metaInfo tells about non-public files
createIndex + readDWD read meta files in German locale to handle Umlaute
Index documentation and creation unified
Vignettes and readme improved
various minor improvements and fixes, see https://github.com/brry/rdwd/commits renamed: indexDWD -> indexFTP added: lldist, maxlldist, nearbyStations no longer exported in NAMESPACE: rowDisplay, dirDWD removed: fileDWD, geoIndexAll

Version 0.7.0 (2017-02-03)

interactive map now colored by availability of recent file
stations with slightly varying coordinates (<900 m apart) aggregated in geoIndex
documentation corrections, improved messages in fileDWD and dirDWD
map vignette expanded, package vignette abbreviated slightly
metaInfo printout greatly improved
duplicate index entries removed New function: rowDisplay New object: geoIndexAll Removed object: mapDWD (to reduce package size)

Version 0.6.1 (2017-01-24) Initial release of the package, development can be seen at https://github.com/brry/rdwd/compare/[email protected]{2016-10-19}[email protected]{2017-01-24}

'selectDWD' uses index information to find files matching a path or station criteria. With the returned path/file names, 'dataDWD' and 'readDWD' download and read data. 'dirDWD' and 'fileDWD' control that no file will be overwritten and path messages are useful. 'indexDWD' lists all the files on the FTP server with DWD data. 'createIndex' uses that list to create 'fileIndex', 'metaIndex' and 'geoIndex'. 'mapDWD' is an interactive leaflet map also useful to explore datasets.

Note: 'dataDWD' and 'readDWD' were started in June 2016 within my misc package berryFunctions, from which they will be deleted after rdwd is on CRAN