An API Wrapper for the Bureau of Labor Statistics (BLS)

Scrapes various data from < https://www.bls.gov/>. The U.S. Bureau of Labor Statistics is the statistical branch of the United States Department of Labor. The package has additional functions to help parse, analyze and visualize the data.

Designed to be a tidy API wrapper for the Bureau of Labor Statistics (BLS.) The package has additional functions to help parse, analyze and visualize the data. The package utalizes "tidyverse" concepts for internal functionality and encourages the use of those concepts with the output data.

Install

Stable version from CRAN:

install.packages("blscrapeR")

The latest development version from GitHub:

devtools::install_github("keberwein/blscrapeR")

Before getting started, you’ll probably want to head over to the BLS and get set up with an API key. While an API key is not required to use the package, the query limits are much higher if you have a key and you’ll have access to more data. Plus, it’s free (as in beer), so why not?

Basic Usage

For “quick and dirty” type of analysis, the package has some quick functions that will pull metrics from the API without series numbers. These quick functions include unemployment, employment, and civilian labor force on a national level.

library(blscrapeR)
df <- quick_unemp_rate()
head(df, 5)
#> # A tibble: 5 x 6
#>    year    period periodName value footnotes  seriesID
#>   <dbl>    <list>     <list> <dbl>    <list>    <list>
#> 1  2017 <chr [1]>  <chr [1]>   4.1 <chr [1]> <chr [1]>
#> 2  2017 <chr [1]>  <chr [1]>   4.2 <chr [1]> <chr [1]>
#> 3  2017 <chr [1]>  <chr [1]>   4.4 <chr [1]> <chr [1]>
#> 4  2017 <chr [1]>  <chr [1]>   4.3 <chr [1]> <chr [1]>
#> 5  2017 <chr [1]>  <chr [1]>   4.4 <chr [1]> <chr [1]>

Search BLS IDs

Some knowledge of BLS ids are needed to query the API. The package includes a "fuzzy search" function to help find these ids. There are currenlty more than 75,000 series ids in the package's internal dataset, series_ids. While these aren't all the series ids the BLS offers, it contains the most popular. The BLS Data Finder is another good resource for finding series ids, that may not be in the internal dataset.

library(blscrapeR)
# Find series ids relating to the total labor force in LA.
ids <- search_ids(keyword = c("Labor Force", "Los Angeles"))
head(ids)
#> # A tibble: 6 x 4
#>                                                                  series_title
#>                                                                         <chr>
#> 1 Labor Force: Balance Of California, State Less Los Angeles-Long Beach-Glend
#> 2  Labor Force: Los Angeles-Long Beach-Glendale, Ca Metropolitan Division (S)
#> 3 Labor Force: Balance Of California, State Less Los Angeles-Long Beach-Glend
#> 4       Labor Force: Los Angeles-Long Beach, Ca Combined Statistical Area (U)
#> 5                                     Labor Force: Los Angeles County, Ca (U)
#> 6                                       Labor Force: Los Angeles City, Ca (U)
#> # ... with 3 more variables: series_id <chr>, seasonal <chr>,
#> #   periodicity_code <chr>

library(blscrapeR)
# Find series ids relating to median weekly earnings of women software developers.
ids <- search_ids(keyword = c("Earnings", "Software", "Women"))
head(ids)
#> # A tibble: 1 x 4
#>                                                                  series_title
#>                                                                         <chr>
#> 1 (Unadj)- Median Usual Weekly Earnings (Second Quartile), Employed Full Time
#> # ... with 3 more variables: series_id <chr>, seasonal <chr>,
#> #   periodicity_code <chr>

API Keys

You should consider getting an API key form the BLS. The package has a function to install your key in your .Renviron so you’ll only have to worry about it once. Plus, it will add extra security by not having your key hard-coded in your scripts for all the world to see.

From the BLS:

Service	Version 2.0 (Registered)	Version 1.0 (Unregistered)
Daily query limit	500	25
Series per query limit	50	25
Years per query limit	20	10
Net/Percent Changes	Yes	No
Optional annual averages	Yes	No
Series descriptions	Yes	No

Key Install

library(blscrapeR)
set_bls_key("YOUR_KEY_IN_QUOTATIONS")
# First time, reload your enviornment so you can use the key without restarting R.
readRenviron("~/.Renviron")
# You can check it with:
Sys.getenv("BLS_KEY")

Note: The above script will add a line to your .Renviron file to be re-used when ever you are in the package. If you are not comfortable with that, you can add the following line to your .Renviron file manually to produce the same result.

BLS_KEY='YOUR_KEY_IN_SINGLE_QUOTES'

Advanced Usage

Now that you have an API key installed, you can call your key in the package’s function arguments with "BLS_KEY". Don't forget the quotes! If you just HAVE to have your key hard-coded in your scripts, you can also pass they key as a string.

Download Multiple BLS Series at Once

library(blscrapeR)
 
# Grab several data sets from the BLS at onece.
# NOTE on series IDs: 
# EMPLOYMENT LEVEL - Civilian labor force - LNS12000000
# UNEMPLOYMENT LEVEL - Civilian labor force - LNS13000000
# UNEMPLOYMENT RATE - Civilian labor force - LNS14000000
df <- bls_api(c("LNS12000000", "LNS13000000", "LNS14000000"),
              startyear = 2008, endyear = 2017, Sys.getenv("BLS_KEY")) %>%
    # Add time-series dates
    dateCast()

# Plot employment level
library(ggplot2)
gg1200 <- subset(df, seriesID=="LNS12000000")
library(ggplot2)
ggplot(gg1200, aes(x=date, y=value)) +
    geom_line() +
    labs(title = "Employment Level - Civ. Labor Force")

Median Weekly Earnings

library(blscrapeR)
library(tidyverse)
# Median Usual Weekly Earnings by Occupation, Unadjusted Second Quartile.
# In current dollars
df <- bls_api(c("LEU0254530800", "LEU0254530600"), startyear = 2000, endyear = 2016, registrationKey = Sys.getenv("BLS_KEY")) %>%
    spread(seriesID, value) %>% dateCast()

# A little help from ggplot2!
library(ggplot2)
ggplot(data = df, aes(x = date)) + 
    geom_line(aes(y = LEU0254530800, color = "Database Admins.")) +
    geom_line(aes(y = LEU0254530600, color = "Software Devs.")) + 
    labs(title = "Median Weekly Earnings by Occupation") + ylab("value") +
    theme(legend.position="top", plot.title = element_text(hjust = 0.5))

For more advanced usage, please see the package vignettes.

Basic Mapping

Like the the “quick functions” for requesting API data, there are two "quick" map functions, bls_map_county() and bls_map_state(). These functions are designed to work with two helper functions get_bls_county() and get get_bls_state(). Each helper function downloads recent data for unemployment rate, unemployment level, employment rate, employment level and civilian labor force. These functions do not pull data from the API, rather they pull data from text files and do not count against daily query limits.

Note: Even though the get_bls functions return data in the correct formats, the bls_map functions can be used with any data set that includes FIPS codes.

The example below maps the current unemployment rate by county. Alaska and Hawaii have to re-located to save space.

library(blscrapeR)
# Grap the data in a pre-formatted data frame.
# If no argument is passed to the function it will load the most recent month's data.
df <- get_bls_county()
 
#Use map function with arguments.
map_bls(map_data = df, fill_rate = "unemployed_rate", 
               labtitle = "Unemployment Rate by County")

Advanced Mapping

What's R mapping without some interactivity? Below we’re using two additional packages, leaflet, which is popular for creating interactive maps and tigris, which allows us to download the exact shape files we need for these data and includes a few other nice tools.

# Leaflet map
library(blscrapeR)
library(tigris)
library(leaflet)
map.shape <- counties(cb = TRUE, year = 2015)
df <- get_bls_county()
 
# Slice the df down to only the variables we need and rename "fips" colunm
# so I can get a cleaner merge later.
df <- df[, c("unemployed_rate", "fips")]
colnames(df) <- c("unemployed_rate", "GEOID")
 
# Merge df with spatial object.
leafmap <- geo_join(map.shape, df, by="GEOID")
 
# Format popup data for leaflet map.
popup_dat <- paste0("<strong>County: </strong>", 
                    leafmap$NAME, 
                    "<br><strong>Value: </strong>", 
                    leafmap$unemployed_rate)
 
pal <- colorQuantile("YlOrRd", NULL, n = 20)
# Render final map in leaflet.
leaflet(data = leafmap) %>% addTiles() %>%
    addPolygons(fillColor = ~pal(unemployed_rate), 
                fillOpacity = 0.8, 
                color = "#BDBDC3", 
                weight = 1,
                popup = popup_dat)

Note: This is just a static image since the full map would not be as portable for the purpose of documentation.

News

blscrapeR 3.1.6

Testing

Nightly API tests on various platforms caused error due to reaching daily query limit. Removed automated test runs in favor of testing manually.

blscrapeR 3.1.5

Documentation

Converted BLS hyperlinks to https.
Fixed various dead hyperlinks.

blscrapeR 3.1.4

Bug fixes

Fixed area in tesing module where qcew_api() function was pulling inproper dates.

Enhancements

Added logic to qcew_api() that throws a warning if useer attempts to query dates outside of the API's range.

Updated NIACS and FIPS codes for 2018.

blscrapeR 3.1.3

Bug fixes

Fixed case issue in qcew annual data.

blscrapeR 3.1.2

Enhancements

Updated NIACS and FIPS codes for 2018.

Bug fixes

The API sometimes adds an extra column for the most current month, which was causing irregular column shifts. Added logic to the map loop to automaticlly adjust for such situations.

blscrapeR 3.1.1

Bug fixes

The bls_api() function now returns the catalog data with a catalog=TRUE argument.

Enhancements

The bls_api() function now returns verbose error message if the api request fails.
Added BLS series_ids internal dataset.
Added search_ids() function to search the internal series_ids data set.

blscrapeR 3.0.1

Bug Fixes

Added an functionality and documentation for an "annual" argument for qcew_api().

Vignettes

Added new feature examples to QCEW_API.Rmd.
Removed qcew.Rmd, since it was duplicated.

blscrapeR 3.0

New Features

Added the map_bls() function. This function replaces the deprecated bls_map_county() bls_map_state() functions and is not backward compatible.
Updated internal data to 2016 FIPS standards.

Enhancements

Updated internal functions to use tidyverse methods. This includes the addition of a few dependencies such as purrr, tibble, and dplyr.
All data are now returned as tibbles.

Documentation

Updated all existing vignettes to use tidyverse functions.
Edited mapping vignette to include new map_bls() function.
Added tidyverse style code to the readme and vignettes.

blscrapeR 2.1.5

Document and vignette fixes

blscrapeR 2.1.4

New Features

Added total non-farm employment as a "quick function."
Added a dt_format argument to the dateCast() function.

blscrapeR 2.1.3

New Features

Added the compound pipe operator from the magrittr package to exported functions.

Enhancements

Added urlExists as a utility function for more robust error handling.
Added try/catch to url downloads.

blscrapeR 2.1.2

New Features

The ability to select custom choropleth colors with all map functions.
Automatic end dates set for bls_api() function.
Removed dependencies for xts and zoo packages. Changed to base R.

Bug fixes

Fixed error when rendering national maps with county-wise data.
Alaska and Hawaii moved further apart in map_data.
Date column changed to optional for bls_api() function.

blscrapeR 2.1.1

New Features

Added a map projection argument to the bls_map_county() function. Choices are either Lambert or Mercator. Mercator is the default for single states, and Lambert the default for national views.
Enhanced error handling for the get_bls_state() function.

Data

Added Mercator projection data set.

Bug Fixes

Fixed date parsing with the get_bls_state() function.

blscrapeR 2.0.0

Major Release

The BLS changed their servers from http to https. API protocols changed in accordance.

blscrapeR 1.0.2.9000

Bug Fixes

FIPS fix for the get_bls_state() function.

blscrapeR 1.0.2

Bug Fixes

Updated URLs for the get_bls_state() function.

Tets

Updated tests for the get_bls_state() function.

blscrapeR 1.0.1

Major Release

The BLS changed the location of their county level employment data on the server side. Changes were made in the get_bls_county() function that were not backwards compatible.

New Features

Added the qcew_api() function to gather data from the BLS Quarterly Census of Employment and Wages. This API is owned by the QCEW and is separate from the main BLS API.
Added 404 error handling to the main bls_api() function.

Data

Added three data sets; niacs, size_titles, and area_titles. These data sets act as helpers for the QCEW API and provide industry and area codes that the API regularly uses.

Bug Fixes

Updated URLs for the get_bls_county() function.

Documentation

Added documentation for the qcew_api() function.
Added documentation for the three new data sets; niacs, size_titles, and area_titles.
Added a vignette for the qcew_api() function.

Tets

Added a test for the qcew_api() function.

blscrapeR 0.4.2

New Features

Added stateName argument to get_bls_county() that allows user to specify a state or list of states.
Added stateName argument to map_bls_county() that allows user to specify state(s) to map.

Documentation

Added mapping vignette.
Added manual pages for the county_map_data and state_map_data internal data sets.

Data

Added tigris and broom packages to prep_maps.R in data-raw folder.

Bug Fixes

Fixed date argument in get_bls_county() and get_bls_state() to return the most recent date if argument is NULL.
Added error handling to map_bls_state() and map_bls_county().
Added tests for map_bls_state() and map_bls_county() to the /testthat directory.

blscrapeR 0.4.1

New Features

Added quick functions to pull popular data sets from the API without the need of the user inputting a series id.
Added data argument to get_bls_county. If NULL the function will return the most recent month.
Added the inflation_adjust() function to help users with calculating inflation from the CPI. Since the API will only allow twenty years of data, the inflation function pulls data from a text file instead that allows the user to get CPI data back to 1947.
Added more error handling to bls_api() function.

Documentation

Added documentation to get_bls_county() to explain the new date argument.
Added package vignettes.

blscrapeR 0.4.0

Major Changes

Added the set_bls_key() function to be used with the bls_api() function. The new function writes the user's api key to the Renviron. The change is backward compatible since the user is sill able to enter their api key as a string. However, for security purposes, the stored key method is preferred and should be promoted.
Added testthat directory and added test_bls_api.R. The tests won't affect anything in the package functionality, but will be useful for future testing.

Deprecated Features

Truncated the bls_state_data() function and added those features to the get_bls_state() function.
Removed dplyr from imports since it's not necessary anymore. Added leaflet and cron to package recommends.

Documentation

Renamed blscrape.R to bls_api.R since the bls_api() function was the only function in that file.
Added testthat to recommends.

blscrapeR 0.3.2

New Features

Revised the map prep in data-raw to render smaller data frames, thereby making the total package size much smaller.
Added a label title argument to map functions bls_map_state() and bls_map_county()
Standardized colnames in all returned data frames throughout the package.

Deprecated Features

Truncated the inflation() function for now. The API seems to return adjusted dollar-values.

Documentation

Made documentation for helper functions invisible in the manual.
Added News.Rd

blscrapeR 0.3.1

New Features

Changed name of county_map() to avoid conflicts with other packages. I didn't mark this as a major release because no other functions in the package are dependent on this.
Various fixes in map functions including removing unused themes and general code tidyness.

Documentation

Various fixes to examples.
Made descriptions more robust.

blscrapeR 0.3.0

Major Changes

Changed name of get_data() to bls_api() to avoid any problems in the global environment. Bumped the version to 0.3.0 because this is the primary workhorse function of the package.

blscrapeR 0.2.1

New Features

Added custom maps rendered from shape files

Major Changes

Removed shape files from package data to cut down on size. This fundamentally changes the way the mapping functions work. Opted instead, for spacial poly data frames rendered from the shapefiles.

blscrapeR 0.2.0

Major Changes

Complete re-write of old get_data() function to pull from API only.
Added arguments to get_data() to pass the BLS API perams.

blscrapeR 0.1.0

New features

Added bls_state_data() function to pull most recent sate-level employment data for mapping.
Added arguments to get_data() to pass the BLS API perams.
Added documentation for data sets.

Bug Fixes

Fixed zoo::index() error.

Reference manual

install.packages("blscrapeR")

3.2.0 by Kris Eberwein, a year ago

https://github.com/keberwein/blscrapeR

Report a bug at https://github.com/keberwein/blscrapeR/issues

Browse source code at https://github.com/cran/blscrapeR

Authors: Kris Eberwein [aut, cre]

Documentation: PDF Manual

MIT + file LICENSE license

Imports httr, jsonlite, ggplot2, magrittr, utils, stats, grDevices, dplyr, purrr, tibble, stringr

Suggests testthat, knitr, rmarkdown

See at CRAN