Stubbing and Setting Expectations on 'HTTP' Requests

Stubbing and setting expectations on 'HTTP' requests. Includes tools for stubbing 'HTTP' requests, including expected request conditions and response conditions. Match on 'HTTP' method, query parameters, request body, headers and more. Can be used for unit tests or outside of a testing context.

R library for stubbing and setting expectations on HTTP requests.

Port of the Ruby gem webmock

How it works in detail

The very very short version is: webmockr helps you stub HTTP requests so you don't have to repeat yourself.

More details

You tell webmockr what HTTP request you want to match against and if it sees a request matching your criteria it doesn't actually do the HTTP request. Instead, it gives back the same object you would have gotten back with a real request, but only with the bits it knows about. For example, we can't give back the actual data you'd get from a real HTTP request as the request wasn't performed.

In addition, if you set an expectation of what webmockr should return, we return that. For example, if you expect a request to return a 418 error (I'm a Teapot), then that's what you'll get.

What you can match against

HTTP method (required)

Plus any single or combination of the following:

URI
- Right now, we can match directly against URI's, and with regex URI patterns. Eventually, we will support RFC 6570 URI templates.
- We normalize URI paths so that URL encoded things match URL un-encoded things (e.g. hello world to hello%20world)
Query parameters
- We normalize query parameter values so that URL encoded things match URL un-encoded things (e.g. message = hello world to message = hello%20world)
Request headers
- We normalize headers and treat all forms of same headers as equal. For example, the following two sets of headers are equal:
  - list(H1 = "value1", content_length = 123, X_CuStOm_hEAder = "foo")
  - list(h1 = "value1", "Content-Length" = 123, "x-cuSTOM-HeAder" = "foo")
Request body

Real HTTP requests

There's a few scenarios to think about when using webmockr:

After doing

library(webmockr)

webmockr is loaded but not turned on. At this point webmockr doesn't change anythning.

Once you turn on webmockr like

webmockr::enable()

webmockr will now by default not allow real HTTP requests from the http libraries that adapters are loaded for (right now only crul).

You can optionally allow real requests via webmockr_allow_net_connect(), and disallow real requests via webmockr_disable_net_connect(). You can check whether you are allowing real requests with webmockr_net_connect_allowed().

Certain kinds of real HTTP requests allowed: We don't suppoprt this yet, but you can allow localhost HTTP requests with the allow_localhost parameter in the webmockr_configure() function.

Storing actual HTTP responses

webmockr doesn't do that. Check out vcr

Stubbing HTTP requests at low http client lib level
Setting and verifying expectations on HTTP requests
Matching requests based on method, URI, headers and body
Support for testthat via vcr
Can be used for testing or outside of a testing context

Supported HTTP libraries

crul
httr

Install

from cran

install.packages("webmockr")

Dev version

devtools::install_github("ropensci/webmockr")

library(webmockr)

Enable webmockr

webmockr::enable()
#> CrulAdapter enabled!
#> HttrAdapter enabled!

Inside a test framework

library(crul)
library(testthat)
 
# make a stub
stub_request("get", "https://httpbin.org/get") %>%
   to_return(body = "success!", status = 200)
#> <webmockr stub> 
#>   method: get
#>   uri: https://httpbin.org/get
#>   with: 
#>     query: 
#>     body: 
#>     request_headers: 
#>   to_return: 
#>     status: 200
#>     body: success!
#>     response_headers: 
#>   should_timeout: FALSE
#>   should_raise: FALSE
 
# check that it's in the stub registry
stub_registry()
#> <webmockr stub registry> 
#>  Registered Stubs
#>    GET: https://httpbin.org/get   | to_return:  with body "success!"  with status 200
 
# make the request
z <- crul::HttpClient$new(url = "https://httpbin.org")$get("get")
 
# run tests (nothing returned means it passed)
expect_is(z, "HttpResponse")
expect_equal(z$status_code, 200)
expect_equal(z$parse("UTF-8"), "success!")

Outside a test framework

library(crul)

Stubbed request based on uri only and with the default response

stub_request("get", "https://httpbin.org/get")
#> <webmockr stub> 
#>   method: get
#>   uri: https://httpbin.org/get
#>   with: 
#>     query: 
#>     body: 
#>     request_headers: 
#>   to_return: 
#>     status: 
#>     body: 
#>     response_headers: 
#>   should_timeout: FALSE
#>   should_raise: FALSE

x <- HttpClient$new(url = "https://httpbin.org")
x$get('get')
#> <crul response> 
#>   url: https://httpbin.org/get
#>   request_headers: 
#>     User-Agent: libcurl/7.54.0 r-curl/3.3 crul/0.7.0.9100
#>     Accept-Encoding: gzip, deflate
#>     Accept: application/json, text/xml, application/xml, */*
#>   response_headers: 
#>   status: 200

set return objects

stub_request("get", "https://httpbin.org/get") %>%
  wi_th(
    query = list(hello = "world")) %>%
    to_return(status = 418)
#> <webmockr stub> 
#>   method: get
#>   uri: https://httpbin.org/get
#>   with: 
#>     query: hello=world
#>     body: 
#>     request_headers: 
#>   to_return: 
#>     status: 418
#>     body: 
#>     response_headers: 
#>   should_timeout: FALSE
#>   should_raise: FALSE

x$get('get', query = list(hello = "world"))
#> <crul response> 
#>   url: https://httpbin.org/get?hello=world
#>   request_headers: 
#>     User-Agent: libcurl/7.54.0 r-curl/3.3 crul/0.7.0.9100
#>     Accept-Encoding: gzip, deflate
#>     Accept: application/json, text/xml, application/xml, */*
#>   response_headers: 
#>   params: 
#>     hello: world
#>   status: 418

Stubbing requests based on method, uri and query params

stub_request("get", "https://httpbin.org/get") %>%
  wi_th(query = list(hello = "world"), 
        headers = list('User-Agent' = 'libcurl/7.51.0 r-curl/2.6 crul/0.3.6', 
                       'Accept-Encoding' = "gzip, deflate"))
#> <webmockr stub> 
#>   method: get
#>   uri: https://httpbin.org/get
#>   with: 
#>     query: hello=world
#>     body: 
#>     request_headers: User-Agent=libcurl/7.51.0 r-cur..., Accept-Encoding=gzip, deflate
#>   to_return: 
#>     status: 
#>     body: 
#>     response_headers: 
#>   should_timeout: FALSE
#>   should_raise: FALSE

stub_registry()
#> <webmockr stub registry> 
#>  Registered Stubs
#>    GET: https://httpbin.org/get 
#>    GET: https://httpbin.org/get?hello=world   | to_return:   with status 418 
#>    GET: https://httpbin.org/get?hello=world   with headers {"User-Agent":"libcurl/7.51.0 r-curl/2.6 crul/0.3.6","Accept-Encoding":"gzip, deflate"}

x <- HttpClient$new(url = "https://httpbin.org")
x$get('get', query = list(hello = "world"))
#> <crul response> 
#>   url: https://httpbin.org/get?hello=world
#>   request_headers: 
#>     User-Agent: libcurl/7.54.0 r-curl/3.3 crul/0.7.0.9100
#>     Accept-Encoding: gzip, deflate
#>     Accept: application/json, text/xml, application/xml, */*
#>   response_headers: 
#>   params: 
#>     hello: world
#>   status: 418

Stubbing requests and set expectation of a timeout

stub_request("post", "https://httpbin.org/post") %>% to_timeout()
#> <webmockr stub> 
#>   method: post
#>   uri: https://httpbin.org/post
#>   with: 
#>     query: 
#>     body: 
#>     request_headers: 
#>   to_return: 
#>     status: 
#>     body: 
#>     response_headers: 
#>   should_timeout: TRUE
#>   should_raise: FALSE
x <- HttpClient$new(url = "https://httpbin.org")
x$post('post')
#> Error: Request Timeout (HTTP 408).
#>  - The client did not produce a request within the time that the server was prepared to wait. The client MAY repeat the request without modifications at any later time.

Stubbing requests and set HTTP error expectation

library(fauxpas)
stub_request("get", "https://httpbin.org/get?a=b") %>% to_raise(HTTPBadRequest)
#> <webmockr stub> 
#>   method: get
#>   uri: https://httpbin.org/get?a=b
#>   with: 
#>     query: 
#>     body: 
#>     request_headers: 
#>   to_return: 
#>     status: 
#>     body: 
#>     response_headers: 
#>   should_timeout: FALSE
#>   should_raise: HTTPBadRequest
x <- HttpClient$new(url = "https://httpbin.org")
x$get('get', query = list(a = "b"))
#> Error: Bad Request (HTTP 400).
#>  - The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.

httr integration

library(webmockr)
library(httr)
#> 
#> Attaching package: 'httr'
#> The following object is masked from 'package:crul':
#> 
#>     handle
 
# turn on httr mocking
httr_mock()

# no stub found
GET("https://httpbin.org/get")
#> Error: Real HTTP connections are disabled.
#> Unregistered request:
#>   GET https://httpbin.org/get   with headers {Accept: application/json, text/xml, application/xml, */*}
#> 
#> You can stub this request with the following snippet:
#> 
#>    stub_request('get', uri = 'https://httpbin.org/get') %>%
#>      wi_th(
#>        headers = list('Accept' = 'application/json, text/xml, application/xml, */*')
#>      )
#> ============================================================

make a stub

stub_request('get', uri = 'https://httpbin.org/get') %>%
  wi_th(
    headers = list('Accept' = 'application/json, text/xml, application/xml, */*')
  ) %>%
  to_return(status = 418, body = "I'm a teapot!!!", headers = list(im_a = "teapot"))
#> <webmockr stub> 
#>   method: get
#>   uri: https://httpbin.org/get
#>   with: 
#>     query: 
#>     body: 
#>     request_headers: Accept=application/json, te...
#>   to_return: 
#>     status: 418
#>     body: I'm a teapot!!!
#>     response_headers: im_a=teapot
#>   should_timeout: FALSE
#>   should_raise: FALSE

now returns mocked response

(res <- GET("https://httpbin.org/get"))
res$status_code
#> [1] 418
res$response_headers
#> $im_a
#> [1] "teapot"

News

webmockr 0.3.4

DEFUNCT

underscore methods to_return_() and wi_th_() are defunct (#60) (#64)

NEW FEATURES

to_return() gains parameter .list (#60) (#64)

MINOR IMPROVEMENTS

typo fixes (#62) thanks @Bisaloo !
improved the print method for stubs, found in StubbedRequest, to have better behavior for very long strings such as in headers and bodies (#63)

BUG FIXES

fix date in mocked httr response object to match the date format that httr uses in real HTTP requests (#58) (#61) via https://github.com/ropensci/vcr/issues/91
fix response headers in mocked httr response objects. httr makes the list of headers insensitive to case, so we now use that function from the package (#59) (#61)
to_return() and wi_th() drop use of the lazyeval package and fall back to using the simple list(...) - fixes problem where creating stubs was failing within test_that() blocks due to some weird lazy eval conflicts (i think) (#60) (#64) thanks @karawoo !

webmockr 0.3.0

MINOR IMPROVEMENTS

returned mocked response headers were retaining case that the user gave - whereas they should be all lowercased to match the output in crul and httr. now fixed. (#49) thanks @hlapp
returned mocked response headers were not all of character class, but depended on what class was given by the user on creating the stub. this is now fixed, returning all character class values for response headers (#48) thanks @hlapp
skip tests that require vcr if vcr is not available (#53)
internal change to crul adapter to produce the same http response as a new version of crul returns - adds a response_headers_all slot (#51) (#54)

webmockr 0.2.9

MINOR IMPROVEMENTS

make request_registry() and stub_registry() print methods more similar to avoid confusion for users (#35)
update docs for enable/disable to indicate that crul and httr supported (#46) (related to #45)
wrap httr adapter examples in requireNamespace so only run when httr available
clean up .onLoad call, removing commented out code, and add note about creating adapter objects does not load crul and httr packages

BUG FIXES

fix to enable() and disable() methods. even though httr is in Suggests, we were loading all adapters (crul, httr) with stop when the package was not found. We now give a message and skip when a package not installed. In addition, we enable() and disable() gain an adapter parameter to indicate which package you want to enable or disable. If adapter not given we attempt all adapters. Note that this bug shouldn't have affected vcr users as httr is in Imports in that package, so you'd have to have httr installed (#45) thanks to @maelle for uncovering the problem

webmockr 0.2.8

NEW FEATURES

Added support for integration with package httr; see HttrAdapter for the details; webmockr now integrates with two HTTP R packages: crul and httr (#43) (#44)
Along with httr integration is a new method httr_mock() to turn on mocking for httr; and two methods build_httr_response and build_httr_request meant for internal use

webmockr 0.2.6

NEW FEATURES

Added support for integration with package vcr (now on CRAN) for doing HTTP request caching

webmockr 0.2.4

NEW FEATURES

New function enabled() to ask if webmockr is enabled, gives a boolean
wi_th() gains new parameter .list as an escape hatch to avoid NSE. examples added in the wi_th man file to clarify its use

MINOR IMPROVEMENTS

matching by request body was not supported, it now is; added examples of matching on request body, see ?stub_request (#36)
make sure that the adapter for crul handles all types of matches (#29)
removed all internal usage of pipes in the package. still exporting pipe for users (#30)
fixed internals to give vcr error when vcr loaded - for future release with vcr support (#34)
require newest crul version

BUG FIXES

Error messages with the suggest stub were not giving bodies. They now give bodies if needed along with method, uri, headers, query (#37)
Fixed Response class that was not dealing with capitalization correctly

webmockr 0.2.0

NEW FEATURES

New function to_raise() to say that a matched response should return a certain exception, currently to_raise accepts error classes from the fauxpas package (#9)
New function to_timeout() to say that a matched response should return a timeout. This is a special case of to_raise to easily do a timeout expectation (#11)
New function request_registry() to list requests in the request registry (#23)
package crul moved to Imports from Suggests as it's the only http client supported for now. will move back to Suggests once we support at least one other http client
webmockr_configure() changes: turn_on has been removed; allow_net_connect and allow_localhost were ignored before, but are now used and are now set to FALSE by default; fixed usage of allow which now accepts character vector of URLs instead of a boolean; the following correctly marked as being ignored for now until fixed net_http_connect_on_start, show_stubbing_instructions, query_values_notation, show_body_diff (#19) (#21)
webmockr_disable_net_connect() now accepts an allow parameter to disable all other connections except those URLs given in allow
webmockr_net_connect_allowed() now accepts a uri parameter to test if a URI/URL is allowed

MINOR IMPROVEMENTS

Fixed printed stub statement when printed to the console - we weren't including headers accurately (#18)
Added examples to the stub_registry() and stub_registry_clea() manual files (#24)
internal methods build_crul_request and build_crul_response moved outside of the CrulAdapter class so that they can be accesed like webmockr:: in other packages
enable() and disable() now return booleans invisibly
General improvements to documentation throughout
Added linting of user inputs to the to_return() method, and docs details on what to input to the method
Added linting of user inputs to the wi_th() method, and docs details on what to input to the method

BUG FIXES

Fixed option allow_localhost, which wasn't actually workin before (#25)

DEPRECATED AND DEFUNCT

webmockr_enable() and webmockr_disable are now defunct. Use webmockr::enable() and webmockr::disable() instead

webmockr 0.1.0

NEW FEATURES

Released to CRAN.

Reference manual

install.packages("webmockr")

0.5.0 by Scott Chamberlain, 3 months ago

https://github.com/ropensci/webmockr (devel) https://books.ropensci.org/http-testing (user manual)

Report a bug at https://github.com/ropensci/webmockr/issues

Browse source code at https://github.com/cran/webmockr

Authors: Scott Chamberlain [aut, cre] , rOpenSci [fnd] (https://ropensci.org)

Documentation: PDF Manual

Task views: Web Technologies and Services

MIT + file LICENSE license

Imports curl, jsonlite, magrittr, R6, urltools, fauxpas, crul

Suggests roxygen2, testthat, xml2, vcr, httr

Imported by vcr, zoltr.

Suggested by RTD, crul, rdatacite, ritis, rplos.

See at CRAN

webmockr

Stubbing and Setting Expectations on 'HTTP' Requests

Supported HTTP libraries

Install

Enable webmockr

Inside a test framework

Outside a test framework

Stubbed request based on uri only and with the default response

Stubbing requests based on method, uri and query params

Stubbing requests and set expectation of a timeout

Stubbing requests and set HTTP error expectation

httr integration

Meta

News

webmockr 0.3.4

DEFUNCT

NEW FEATURES

MINOR IMPROVEMENTS

BUG FIXES

webmockr 0.3.0

MINOR IMPROVEMENTS

webmockr 0.2.9

MINOR IMPROVEMENTS

BUG FIXES

webmockr 0.2.8

NEW FEATURES

webmockr 0.2.6

NEW FEATURES

webmockr 0.2.4

NEW FEATURES

MINOR IMPROVEMENTS

BUG FIXES

webmockr 0.2.0

NEW FEATURES

MINOR IMPROVEMENTS

BUG FIXES

DEPRECATED AND DEFUNCT

webmockr 0.1.0

NEW FEATURES

Reference manual

R links

R homepage

Download R

Mailing lists

R documentation

R manuals

R FAQs

The R Journal

CRAN links

CRAN homepage

CRAN repository policy

Submit a package

METACRAN stuff

About METACRAN

At github

Report a bug