A programmatic interface to the Web Service methods provided by Bold Systems (< http://www.boldsystems.org/>) for genetic 'barcode' data. Functions include methods for searching by sequences by taxonomic names, ids, collectors, and institutions; as well as a function for searching for specimens, and downloading trace files.
bold
accesses BOLD barcode data.
The Barcode of Life Data Systems (BOLD) is designed to support the generation and application of DNA barcode data. The platform consists of four main modules: a data portal, a database of barcode clusters, an educational portal, and a data collection workbench.
This package retrieves data from the BOLD database of barcode clusters, and allows for searching of over 1.7M public records using multiple search criteria including sequence data, specimen data, specimen plus sequence data, as well as trace files.
Documentation for the BOLD API.
See also the taxize book for more options for taxonomic workflows with BOLD: https://ropensci.github.io/taxize-book/
Installation instructions
Stable Version
install.packages("bold")
Development Version
Install sangerseqR
first
source("http://bioconductor.org/biocLite.R")biocLite("sangerseqR")
Then bold
devtools::install_github("ropensci/bold")
library("bold")
Default is to get a list back
bold_seq(taxon='Coelioxys')[[1]]#> $id#> [1] "ABEE013-17"#> #> $name#> [1] "Coelioxys afra"#> #> $gene#> [1] "ABEE013-17"#> #> $sequence#> [1] "AATATTATATATAATTTTTGCAATTTGATCAGGTATAATTGGATCTTCATTAAGAATAATTATTCGAATAGAATTAAGAACTCCAGGAAGATGAATCAACAACGATCAAATTTATAATTCTTTTATTACAGCTCATGCATTTTTAATAATTTTTTTTTTAGTAATACCATTTTTAATTGGAGGATTTGGAAATTGATTAGTACCTTTAATACTAGGAGCCCCCGATATAGCTTTTCCACGAATAAATAATGTAAGATTTTGACTATTACCTCCCTCAATTTTCTTATTATTATCAAGAACCCTAATTAACCCAAGAGCTGGTACTGGATGAACTGTATATCCTCCTTTATCCTTATATACATTTCATGCCTCACCTTCCGTTGATTTAGCAATTTTTTCACTTCATTTATCAGGAATTTCATCAATTATTGGATCAATAAATTTTATTGTTACAATCTTAATAATAAAAAATTTTTCTTTAAATTATAGACAAATACCATTATTTTCATGATCAGTTTTAATTACTACAATTTTACTTTTATTATCACTACCAATTTTAGCTGGAGCAATTACTATACTCCTATTTGATCGAAATTTAAATACCTCATTCTTTGACCCAATAGGAGGAGGAGATCCAATTTTATATCAACATTTATTT-----------------"
You can optionally get back the crul
response object
res <- bold_seq(taxon='Coelioxys', response=TRUE)res$response_headers#> $status#> [1] "HTTP/1.1 200 OK"#> #> $date#> [1] "Wed, 14 Nov 2018 00:49:40 GMT"#> #> $server#> [1] "Apache/2.2.15 (Red Hat)"#> #> $`x-powered-by`#> [1] "PHP/5.3.15"#> #> $`content-disposition`#> [1] "attachment; filename=fasta.fas"#> #> $connection#> [1] "close"#> #> $`transfer-encoding`#> [1] "chunked"#> #> $`content-type`#> [1] "application/x-download"
By default you download tsv
format data, which is given back to you as a data.frame
res <- bold_specimens(taxon='Osmia')head(res[,1:8])#> processid sampleid recordID catalognum#> 1 ABEE151-17 NHMW-HYM 773 8362250 #> 2 ABEE184-17 NHMW-HYM 2142 8362283 NHMW-HYM 2142#> 3 ABEE185-17 NHMW-HYM 2144 8362284 NHMW-HYM 2144#> 4 ABEE188-17 NHMW-HYM 2242 8362287 NHMW-HYM 2242#> 5 ABEE190-17 NHMW-HYM 2259 8362289 NHMW-HYM 2259#> 6 GBAH3878-08 EU726629 856409 EU726629#> fieldnum#> 1 #> 2 NBH2 Zimmermann 2017.04.03 Oesterreich WienAUTWienZimmermann#> 3 NBH2 Zimmermann 2017.04.03 Oesterreich WienAUTWienZimmermann#> 4 NHB3 Schoder 2017.05.30 Oesterreich WienAUTWienSchoder#> 5 NBH9 Schoder 2017.06.15 Oesterreich WienAUTWienSchoder#> 6 #> institution_storing collection_code bin_uri#> 1 Naturhistorisches Museum Wien NA BOLD:AAE5409#> 2 Naturhistorisches Museum Wien NA BOLD:AAE5409#> 3 Naturhistorisches Museum Wien NA BOLD:ADJ1069#> 4 Naturhistorisches Museum Wien NA BOLD:AAF2155#> 5 Naturhistorisches Museum Wien NA BOLD:AAD0313#> 6 Mined from GenBank, NCBI NA BOLD:AAA4494
By default you download tsv
format data, which is given back to you as a data.frame
res <- bold_seqspec(taxon='Osmia', sepfasta=TRUE)res$fasta[1:2]#> $`ABEE151-17`#> [1] "----------------------------------------------------------TTTTTGCTATATGATCAGGTACAGTAGGTTCAGCTATAAGAATTATTATTCGAATAGAACTTAGAGTTCCAGGATCATGAATTTCTAATGACCAAATTTATAATACTTTAGTAACTGCTCATGCTTTTTTAATAATTTTCTTTCTTGTAATACCATTTCTAATTGGAGGATTTGGAAATTGATTAATTCCTTTAATATTAGGAATTCCAGATATAGCCTTTCCACGAATAAATAATATTAGATTTTGACTTTTACCACCTTCTTTAATATTATTAATATTAAGAAATTTTATAAATCCAAGTCCAGGAACTGGATGAACTGTTTATCCTCCTCTTTCATCTTATATATTTCATTCTTCCCCATCAGTAGATTTAGCAATTTTTTCATTACATATTTCCGGATTATCCTCTATTATAGGTTCATTAAATTTTATTGTCACAATTATTATAATAAAAAATATTTCATTAAAACATACTCAATTACCCTTATTTTCTTGATCTGTATTTATTACTACTATTTTATTACTTTTCTCTCTCCCAGTTTTAGCTGGAGCTATTACTATACTTTTATTTGATCGAAATTTTAACACCTCATTTTTTGACCCGACGGGAGGTGGAGATCCAATTTTATACCAACATTTATTTTGATTTTTTGGACAT-----------------------"#> #> $`ABEE184-17`#> [1] "-------CTCACTATAGGGATTCAACCAATCATAAAGATATTGGAATTCTTTATATAATTTTTGCTATATGATCAGGTACAGTAGGTTCAGCTATAAGAATTATTATTCGAATAGAACTTAGAGTTCCAGGATCATGAATTTCTAATGACCAAATTTATAATACTTTAGTAACTGCTCATGCTTTTTTAATAATTTTCTTTCTTGTAATACCATTTCTAATTGGAGGATTTGGAAATTGATTAATTCCTTTAATATTAGGAATTCCAGATATAGCCTTTCCACGAATAAATAATATTAGATTTTGACTTTTACCACCTTCTTTAATATTATTAATATTAAGAAATTTTATAAATCCAAGTCCAGGAACTGGATGAACTGTTTATCCTCCTCTTTCATCTTATATATTTCATTCTTCCCCATCAGTAGATTTAGCAATTTTTTCATTACATATTTCCGGATTATCCTCTATTATAGGTTCATTAAATTTTATTGTCACAATTATTATAATAAAAAATATTTCATTAAAACATACTCAATTACCCTTATTTTCTTGATCTGTATTTATTACTACTATTTTATTACTTTTCTCTCTCCCAGTTTTAGCTGGAGCTATTACTATACTTTTATTTGATCGAAATTTTAACACCTCATTTTTTGACC-------------------------------------------------------------------------------"
Or you can index to a specific sequence like
res$fasta['GBAH0293-06']#> $`GBAH0293-06`#> [1] "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TTAATGTTAGGGATTCCAGATATAGCTTTTCCACGAATAAATAATATTAGATTTTGACTGTTACCTCCATCTTTAATATTATTACTTTTAAGAAATTTTTTAAATCCAAGTCCTGGAACAGGATGAACAGTTTATCCTCCTTTATCATCAAATTTATTTCATTCTTCTCCTTCAGTTGATTTAGCAATTTTTTCTTTACATATTTCAGGTTTATCTTCTATTATAGGTTCATTAAATTTTATTGTTACAATTATTATAATAAAAAATATTTCTTTAAAATATATTCAATTACCTTTATTTTCTTGATCTGTATTTATTACTACTATTCTTTTATTATTTTCTTTACCTGTATTAGCTGGAGCTATTACTATATTATTATTTGATCGAAATTTTAATACATCTTTTTTTGATCCAACAGGAGGGGGAGATCCAATTCTTTATCAACATTTATTTTGATTTTTTGGTCATCCTGAAGTTTATATTTTAATTTTACCTGGATTTGGATTAATTTCTCAAATTATTTCTAATGAAAGAGGAAAAAAAGAAACTTTTGGAAATATTGGTATAATTTATGCTATATTAAGAATTGGACTTTTAGGTTTTATTGTT---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------"
This function downloads files to your machine - it does not load them into your R session - but prints out where the files are for your information.
x <- bold_trace(ids = 'ACRJP618-11', progress = FALSE)read_trace(x$ab1)#> Number of datapoints: 8877#> Number of basecalls: 685#> #> Primary Basecalls: NNNNNNNNNNNNNNNNNNGNNNTTGAGCAGGNATAGTAGGANCTTCTCTTAGTCTTATTATTCGAACAGAATTAGGAAATCCAGGATTTTTAATTGGAGATGATCAAATCTACAATACTATTGTTACGGCTCATGCTTTTATTATAATTTTTTTTATAGTTATACCTATTATAATTGGAGGATTTGGTAATTGATTAGTTCCCCTTATACTAGGAGCCCCAGATATAGCTTTCCCTCGAATAAACAATATAAGTTTTTGGCTTCTTCCCCCTTCACTATTACTTTTAATTTCCAGAAGAATTGTTGAAAATGGAGCTGGAACTGGATGAACAGTTTATCCCCCACTGTCATCTAATATTGCCCATAGAGGTACATCAGTAGATTTAGCTATTTTTTCTTTACATTTAGCAGGTATTTCCTCTATTTTAGGAGCGATTAATTTTATTACTACAATTATTAATATACGAATTAACAGTATAAATTATGATCAAATACCACTATTTGTGTGATCAGTAGGAATTACTGCTTTACTCTTATTACTTTCTCTTCCAGTATTAGCAGGTGCTATCACTATATTATTAACGGATCGAAATTTAAATACATCATTTTTTGATCCTGCAGGAGGAGGAGATCCAATTTTATATCAACATTTATTTTGATTTTTTGGACNTCNNNNAAGTTTAAN#> #> Secondary Basecalls:
Sometimes with bold_seq()
you request a lot of data, which can cause problems due
to BOLD's servers.
An example is the taxonomic name Arthropoda. When you send a request like
bold_seq(taxon = "Arthropoda")
BOLD attempts to give you back sequences
for all records under Arthropoda. This, as you can imagine, is a lot of
sequences.
library("taxize")
Using taxize::downstream
get children of Arthropoda
x <- downstream("Arthropoda", db = "ncbi", downto = "class")nms <- x$Arthropoda$childtaxa_name
Optionally, check that the name exists in BOLD's data. Any that are not in BOLD will give back a row of NAs
checks <- bold_tax_name(nms)# all is goodchecks[,1:5]#> taxid taxon tax_rank tax_division parentid#> 1 26059 Pycnogonida class Animals 20#> 2 63 Arachnida class Animals 20#> 3 74 Merostomata class Animals 20#> 4 493944 Pauropoda class Animals 20#> 5 80390 Symphyla class Animals 20#> 6 85 Diplopoda class Animals 20#> 7 75 Chilopoda class Animals 20#> 8 82 Insecta class Animals 20#> 9 372 Collembola class Animals 20#> 10 734357 Protura class Animals 20#> 11 84 Remipedia class Animals 20#> 12 73 Cephalocarida class Animals 20#> 13 68 Branchiopoda class Animals 20#> 14 765970 Hexanauplia class Animals 20#> 15 69 Malacostraca class Animals 20#> 16 889450 Ichthyostraca class Animals 20#> 17 80 Ostracoda class Animals 20
Then pass those names to bold_seq()
. You could pass all names in at once,
but we're trying to avoid the large data request problem here, so run each
one separately with lapply
or a for loop like request.
out <- lapply(nms, bold_seq)
Get citation information for bold
in R by running: citation(package = 'bold')
bold
in R doing citation(package = 'bold')
vcr
to cache responses, speeds up tests significantly, and no longer relies on an internet connection (#55) (#56)bold_seq()
: sometimes on large requests, the BOLD servers time out, and give back partial output but don't indicate that there was an error. We catch this kind of error now, throw a message for the user, and the function gives back the partial output given by the server. Also added to the documentation for bold_seq()
and in the README that if you run into this problem try to do many queries that will result in smaller set of results instead of one or fewer larger queries (#52) (#53)bold_seq()
: remove return characters (\r
and \n
) from sequences (#54)bold_identify_parents()
gains many new parameters (taxid
, taxon
, tax_rank
, tax_division
, parentid
, parentname
, taxonrep
, specimenrecords
) to filter parents based on any of a number of fields - should solve problem where multiple parents found for a single taxon, often in different kingdoms (#50)bold_identify()
that the function uses lapply
internally, so queries with lots of sequences can take a long timebold_specimens()
: use rawToChar()
on raw bytes instead of parse()
from crul
(#47)crul
for HTTP requests. Only really affects users in that
specifying curl options works slightly differenlty (#42)marker
parameter in bold_seqspec
was and maybe still is not working,
in the sense that using the parameter doesn't always limit results to the
marker you specify. Not really fixed - watch out for it, and filter after you
get results back to get markers you want. (#25)bold_identify_parents
- was failing when no match for a
parent name. (#41) thx @VascoElbrechttsv
results were erroring in bold_specimens
and other fxns (#46) - fixed
by switching to new BOLD v4 API (#30)stats
and utils
- replaced
is
with inherits
(#39)bold_identify_parents()
to add taxonomic information
to the output of bold_identif()
. We take the taxon names from bold_identify
output, and use bold_tax_name
to get the taxonomic ID, passing it to
bold_tax_id
to get the parent names, then attaches those to the input data.
There are two options given what you put for the wide
parameter. If TRUE
you get data.frames of the same dimensions with parent rank name and ID
as new columns (for each name going up the hierarchy) - while if FALSE
you get a long data.frame. thanks @dougwyu for inspiring this (#36)xml2::xml_find_one
with xml2::xml_find_first
(#33)db
options in bold_identify
man file -
COX1 and COX1_SPECIES were switched (#37) thanks for pointing that out
@dougwyubold_tax_id
for when some elements returned from the BOLD
API were empty/NULL
(#32) thanks @fmichonneau !!xml2
from XML
as the XML parser for this package (#26)bold_trace()
to create dir and tar file when it doesn't
already existcontent(x, "text")
, so now using rawToChar(content(x))
,
which works (#24)sangerseqR
package now in Suggests for reading trace files, and is only used in bold_trace()
function.bold_trace()
gains two new parameters: overwrite
to choose whether to overwrite an existing
file of the same name or not, progress
to show a progress bar for downloading or not.bold_trace()
gains a print method to show a tidy summary of the trace file downloaded.bold_tax_name()
(#17) and bold_tax_id()
(#18) in which species that were missing from the BOLD database returned empty arrays but 200 status codes. Parsing those as failed attempts now. Also fixes problem in taxize in bold_search()
that use these two functions.bold_tax_name()
and bold_tax_id()
, which search for taxonomic data from BOLD using either names or BOLD identifiers, respectively. (#11)jsonlite
and reshape
.callopts
parameter changed to ...
throughout the package, so that passing on options to httr::GET
is done via named parameters, e.g., config=verbose()
. (#13)httr
(v0.4) (#9), and added a few more tests (#7)