bold
Interface to Bold Systems API
A programmatic interface to the Web Service methods provided by Bold Systems (< http://www.boldsystems.org/>) for genetic 'barcode' data. Functions include methods for searching by sequences by taxonomic names, ids, collectors, and institutions; as well as a function for searching for specimens, and downloading trace files.
bold accesses BOLD barcode data.
The Barcode of Life Data Systems (BOLD) is designed to support the generation and application of DNA barcode data. The platform consists of four main modules: a data portal, a database of barcode clusters, an educational portal, and a data collection workbench.
This package retrieves data from the BOLD database of barcode clusters, and allows for searching of over 1.7M public records using multiple search criteria including sequence data, specimen data, specimen plus sequence data, as well as trace files.
Documentation for the BOLD API.
See also the taxize book for more options for taxonomic workflows with BOLD: https://ropensci.github.io/taxize-book/
Installation instructions
Stable Version
install.packages("bold")Development Version
Install sangerseqR first
source("http://bioconductor.org/biocLite.R")biocLite("sangerseqR")Then bold
devtools::install_github("ropensci/bold")Usage
library("bold")Search for sequence data only
Default is to get a list back
bold_seq(taxon='Coelioxys')[[1]]#> $id#> [1] "ABEE013-17"#> #> $name#> [1] "Coelioxys afra"#> #> $gene#> [1] "ABEE013-17"#> #> $sequence#> [1] "AATATTATATATAATTTTTGCAATTTGATCAGGTATAATTGGATCTTCATTAAGAATAATTATTCGAATAGAATTAAGAACTCCAGGAAGATGAATCAACAACGATCAAATTTATAATTCTTTTATTACAGCTCATGCATTTTTAATAATTTTTTTTTTAGTAATACCATTTTTAATTGGAGGATTTGGAAATTGATTAGTACCTTTAATACTAGGAGCCCCCGATATAGCTTTTCCACGAATAAATAATGTAAGATTTTGACTATTACCTCCCTCAATTTTCTTATTATTATCAAGAACCCTAATTAACCCAAGAGCTGGTACTGGATGAACTGTATATCCTCCTTTATCCTTATATACATTTCATGCCTCACCTTCCGTTGATTTAGCAATTTTTTCACTTCATTTATCAGGAATTTCATCAATTATTGGATCAATAAATTTTATTGTTACAATCTTAATAATAAAAAATTTTTCTTTAAATTATAGACAAATACCATTATTTTCATGATCAGTTTTAATTACTACAATTTTACTTTTATTATCACTACCAATTTTAGCTGGAGCAATTACTATACTCCTATTTGATCGAAATTTAAATACCTCATTCTTTGACCCAATAGGAGGAGGAGATCCAATTTTATATCAACATTTATTT-----------------"You can optionally get back the crul response object
res <- bold_seq(taxon='Coelioxys', response=TRUE)res$response_headers#> $status#> [1] "HTTP/1.1 200 OK"#> #> $date#> [1] "Wed, 14 Nov 2018 00:49:40 GMT"#> #> $server#> [1] "Apache/2.2.15 (Red Hat)"#> #> $`x-powered-by`#> [1] "PHP/5.3.15"#> #> $`content-disposition`#> [1] "attachment; filename=fasta.fas"#> #> $connection#> [1] "close"#> #> $`transfer-encoding`#> [1] "chunked"#> #> $`content-type`#> [1] "application/x-download"Search for specimen data only
By default you download tsv format data, which is given back to you as a data.frame
res <- bold_specimens(taxon='Osmia')head(res[,1:8])#> processid sampleid recordID catalognum#> 1 ABEE151-17 NHMW-HYM 773 8362250 #> 2 ABEE184-17 NHMW-HYM 2142 8362283 NHMW-HYM 2142#> 3 ABEE185-17 NHMW-HYM 2144 8362284 NHMW-HYM 2144#> 4 ABEE188-17 NHMW-HYM 2242 8362287 NHMW-HYM 2242#> 5 ABEE190-17 NHMW-HYM 2259 8362289 NHMW-HYM 2259#> 6 GBAH3878-08 EU726629 856409 EU726629#> fieldnum#> 1 #> 2 NBH2 Zimmermann 2017.04.03 Oesterreich WienAUTWienZimmermann#> 3 NBH2 Zimmermann 2017.04.03 Oesterreich WienAUTWienZimmermann#> 4 NHB3 Schoder 2017.05.30 Oesterreich WienAUTWienSchoder#> 5 NBH9 Schoder 2017.06.15 Oesterreich WienAUTWienSchoder#> 6 #> institution_storing collection_code bin_uri#> 1 Naturhistorisches Museum Wien NA BOLD:AAE5409#> 2 Naturhistorisches Museum Wien NA BOLD:AAE5409#> 3 Naturhistorisches Museum Wien NA BOLD:ADJ1069#> 4 Naturhistorisches Museum Wien NA BOLD:AAF2155#> 5 Naturhistorisches Museum Wien NA BOLD:AAD0313#> 6 Mined from GenBank, NCBI NA BOLD:AAA4494Search for specimen plus sequence data
By default you download tsv format data, which is given back to you as a data.frame
res <- bold_seqspec(taxon='Osmia', sepfasta=TRUE)res$fasta[1:2]#> $`ABEE151-17`#> [1] "----------------------------------------------------------TTTTTGCTATATGATCAGGTACAGTAGGTTCAGCTATAAGAATTATTATTCGAATAGAACTTAGAGTTCCAGGATCATGAATTTCTAATGACCAAATTTATAATACTTTAGTAACTGCTCATGCTTTTTTAATAATTTTCTTTCTTGTAATACCATTTCTAATTGGAGGATTTGGAAATTGATTAATTCCTTTAATATTAGGAATTCCAGATATAGCCTTTCCACGAATAAATAATATTAGATTTTGACTTTTACCACCTTCTTTAATATTATTAATATTAAGAAATTTTATAAATCCAAGTCCAGGAACTGGATGAACTGTTTATCCTCCTCTTTCATCTTATATATTTCATTCTTCCCCATCAGTAGATTTAGCAATTTTTTCATTACATATTTCCGGATTATCCTCTATTATAGGTTCATTAAATTTTATTGTCACAATTATTATAATAAAAAATATTTCATTAAAACATACTCAATTACCCTTATTTTCTTGATCTGTATTTATTACTACTATTTTATTACTTTTCTCTCTCCCAGTTTTAGCTGGAGCTATTACTATACTTTTATTTGATCGAAATTTTAACACCTCATTTTTTGACCCGACGGGAGGTGGAGATCCAATTTTATACCAACATTTATTTTGATTTTTTGGACAT-----------------------"#> #> $`ABEE184-17`#> [1] "-------CTCACTATAGGGATTCAACCAATCATAAAGATATTGGAATTCTTTATATAATTTTTGCTATATGATCAGGTACAGTAGGTTCAGCTATAAGAATTATTATTCGAATAGAACTTAGAGTTCCAGGATCATGAATTTCTAATGACCAAATTTATAATACTTTAGTAACTGCTCATGCTTTTTTAATAATTTTCTTTCTTGTAATACCATTTCTAATTGGAGGATTTGGAAATTGATTAATTCCTTTAATATTAGGAATTCCAGATATAGCCTTTCCACGAATAAATAATATTAGATTTTGACTTTTACCACCTTCTTTAATATTATTAATATTAAGAAATTTTATAAATCCAAGTCCAGGAACTGGATGAACTGTTTATCCTCCTCTTTCATCTTATATATTTCATTCTTCCCCATCAGTAGATTTAGCAATTTTTTCATTACATATTTCCGGATTATCCTCTATTATAGGTTCATTAAATTTTATTGTCACAATTATTATAATAAAAAATATTTCATTAAAACATACTCAATTACCCTTATTTTCTTGATCTGTATTTATTACTACTATTTTATTACTTTTCTCTCTCCCAGTTTTAGCTGGAGCTATTACTATACTTTTATTTGATCGAAATTTTAACACCTCATTTTTTGACC-------------------------------------------------------------------------------"Or you can index to a specific sequence like
res$fasta['GBAH0293-06']#> $`GBAH0293-06`#> [1] "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TTAATGTTAGGGATTCCAGATATAGCTTTTCCACGAATAAATAATATTAGATTTTGACTGTTACCTCCATCTTTAATATTATTACTTTTAAGAAATTTTTTAAATCCAAGTCCTGGAACAGGATGAACAGTTTATCCTCCTTTATCATCAAATTTATTTCATTCTTCTCCTTCAGTTGATTTAGCAATTTTTTCTTTACATATTTCAGGTTTATCTTCTATTATAGGTTCATTAAATTTTATTGTTACAATTATTATAATAAAAAATATTTCTTTAAAATATATTCAATTACCTTTATTTTCTTGATCTGTATTTATTACTACTATTCTTTTATTATTTTCTTTACCTGTATTAGCTGGAGCTATTACTATATTATTATTTGATCGAAATTTTAATACATCTTTTTTTGATCCAACAGGAGGGGGAGATCCAATTCTTTATCAACATTTATTTTGATTTTTTGGTCATCCTGAAGTTTATATTTTAATTTTACCTGGATTTGGATTAATTTCTCAAATTATTTCTAATGAAAGAGGAAAAAAAGAAACTTTTGGAAATATTGGTATAATTTATGCTATATTAAGAATTGGACTTTTAGGTTTTATTGTT---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------"Get trace files
This function downloads files to your machine - it does not load them into your R session - but prints out where the files are for your information.
x <- bold_trace(ids = 'ACRJP618-11', progress = FALSE)read_trace(x$ab1)#> Number of datapoints: 8877#> Number of basecalls: 685#> #> Primary Basecalls: NNNNNNNNNNNNNNNNNNGNNNTTGAGCAGGNATAGTAGGANCTTCTCTTAGTCTTATTATTCGAACAGAATTAGGAAATCCAGGATTTTTAATTGGAGATGATCAAATCTACAATACTATTGTTACGGCTCATGCTTTTATTATAATTTTTTTTATAGTTATACCTATTATAATTGGAGGATTTGGTAATTGATTAGTTCCCCTTATACTAGGAGCCCCAGATATAGCTTTCCCTCGAATAAACAATATAAGTTTTTGGCTTCTTCCCCCTTCACTATTACTTTTAATTTCCAGAAGAATTGTTGAAAATGGAGCTGGAACTGGATGAACAGTTTATCCCCCACTGTCATCTAATATTGCCCATAGAGGTACATCAGTAGATTTAGCTATTTTTTCTTTACATTTAGCAGGTATTTCCTCTATTTTAGGAGCGATTAATTTTATTACTACAATTATTAATATACGAATTAACAGTATAAATTATGATCAAATACCACTATTTGTGTGATCAGTAGGAATTACTGCTTTACTCTTATTACTTTCTCTTCCAGTATTAGCAGGTGCTATCACTATATTATTAACGGATCGAAATTTAAATACATCATTTTTTGATCCTGCAGGAGGAGGAGATCCAATTTTATATCAACATTTATTTTGATTTTTTGGACNTCNNNNAAGTTTAAN#> #> Secondary Basecalls:Large data
Sometimes with bold_seq() you request a lot of data, which can cause problems due
to BOLD's servers.
An example is the taxonomic name Arthropoda. When you send a request like
bold_seq(taxon = "Arthropoda") BOLD attempts to give you back sequences
for all records under Arthropoda. This, as you can imagine, is a lot of
sequences.
library("taxize")Using taxize::downstream get children of Arthropoda
x <- downstream("Arthropoda", db = "ncbi", downto = "class")nms <- x$Arthropoda$childtaxa_nameOptionally, check that the name exists in BOLD's data. Any that are not in BOLD will give back a row of NAs
checks <- bold_tax_name(nms)# all is goodchecks[,1:5]#> taxid taxon tax_rank tax_division parentid#> 1 26059 Pycnogonida class Animals 20#> 2 63 Arachnida class Animals 20#> 3 74 Merostomata class Animals 20#> 4 493944 Pauropoda class Animals 20#> 5 80390 Symphyla class Animals 20#> 6 85 Diplopoda class Animals 20#> 7 75 Chilopoda class Animals 20#> 8 82 Insecta class Animals 20#> 9 372 Collembola class Animals 20#> 10 734357 Protura class Animals 20#> 11 84 Remipedia class Animals 20#> 12 73 Cephalocarida class Animals 20#> 13 68 Branchiopoda class Animals 20#> 14 765970 Hexanauplia class Animals 20#> 15 69 Malacostraca class Animals 20#> 16 889450 Ichthyostraca class Animals 20#> 17 80 Ostracoda class Animals 20Then pass those names to bold_seq(). You could pass all names in at once,
but we're trying to avoid the large data request problem here, so run each
one separately with lapply or a for loop like request.
out <- lapply(nms, bold_seq)Citation
Get citation information for bold in R by running: citation(package = 'bold')
Meta
- Please report any issues or bugs
- License: MIT
- Get citation information for
boldin R doingcitation(package = 'bold') - Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
News
bold 0.8.6
MINOR IMPROVEMENTS
- tests that make HTTP requests now use package
vcrto cache responses, speeds up tests significantly, and no longer relies on an internet connection (#55) (#56) bold_seq(): sometimes on large requests, the BOLD servers time out, and give back partial output but don't indicate that there was an error. We catch this kind of error now, throw a message for the user, and the function gives back the partial output given by the server. Also added to the documentation forbold_seq()and in the README that if you run into this problem try to do many queries that will result in smaller set of results instead of one or fewer larger queries (#52) (#53)bold_seq(): remove return characters (\rand\n) from sequences (#54)
bold 0.8.0
MINOR IMPROVEMENTS
- link to taxize bookdown book in readme and vignette (#51)
bold_identify_parents()gains many new parameters (taxid,taxon,tax_rank,tax_division,parentid,parentname,taxonrep,specimenrecords) to filter parents based on any of a number of fields - should solve problem where multiple parents found for a single taxon, often in different kingdoms (#50)- add note in docs of
bold_identify()that the function useslapplyinternally, so queries with lots of sequences can take a long time
BUG FIXES
- fix
bold_specimens(): userawToChar()on raw bytes instead ofparse()fromcrul(#47)
bold 0.5.0
NEW FEATURES
- Now using BOLD's v4 API throughout the package. This was essentially just a change of the BASE URL for each request (#30)
- Now using
crulfor HTTP requests. Only really affects users in that specifying curl options works slightly differenlty (#42)
BUG FIXES
markerparameter inbold_seqspecwas and maybe still is not working, in the sense that using the parameter doesn't always limit results to the marker you specify. Not really fixed - watch out for it, and filter after you get results back to get markers you want. (#25)- Fixed bug in
bold_identify_parents- was failing when no match for a parent name. (#41) thx @VascoElbrecht tsvresults were erroring inbold_specimensand other fxns (#46) - fixed by switching to new BOLD v4 API (#30)
MINOR IMPROVEMENTS
- Namespace calls to base pkgs for
statsandutils- replacediswithinherits(#39)
bold 0.4.0
NEW FEATURES
- New function
bold_identify_parents()to add taxonomic information to the output ofbold_identif(). We take the taxon names frombold_identifyoutput, and usebold_tax_nameto get the taxonomic ID, passing it tobold_tax_idto get the parent names, then attaches those to the input data. There are two options given what you put for thewideparameter. IfTRUEyou get data.frames of the same dimensions with parent rank name and ID as new columns (for each name going up the hierarchy) - while ifFALSEyou get a long data.frame. thanks @dougwyu for inspiring this (#36)
MINOR IMPROVEMENTS
- replace
xml2::xml_find_onewithxml2::xml_find_first(#33) - Fix description of
dboptions inbold_identifyman file - COX1 and COX1_SPECIES were switched (#37) thanks for pointing that out @dougwyu
BUG FIXES
- Fix to
bold_tax_idfor when some elements returned from the BOLD API were empty/NULL(#32) thanks @fmichonneau !!
bold 0.3.5
MINOR IMPROVEMENTS
- Added more tests to the test suite (#28)
BUG FIXES
- Fixed a bug in an internal data parser (#27)
bold 0.3.4
NEW FEATURES
- Added a code of conduct
MINOR IMPROVEMENTS
- Switched to
xml2fromXMLas the XML parser for this package (#26) - Fixes to
bold_trace()to create dir and tar file when it doesn't already exist
BUG FIXES
- Fixed odd problem where sometimes resulting data from HTTP request
was garbled on
content(x, "text"), so now usingrawToChar(content(x)), which works (#24)
bold 0.3.0
MINOR IMPROVEMENTS
- Explicitly import non-base R functions (#22)
- Better package level manual file
bold 0.2.6
MINOR IMPROVEMENTS
sangerseqRpackage now in Suggests for reading trace files, and is only used inbold_trace()function.- General code tidying, reduction of code duplication.
bold_trace()gains two new parameters:overwriteto choose whether to overwrite an existing file of the same name or not,progressto show a progress bar for downloading or not.bold_trace()gains a print method to show a tidy summary of the trace file downloaded.
BUG FIXES
- Fixed similar bugs in
bold_tax_name()(#17) andbold_tax_id()(#18) in which species that were missing from the BOLD database returned empty arrays but 200 status codes. Parsing those as failed attempts now. Also fixes problem in taxize inbold_search()that use these two functions.
bold 0.2.0
NEW FEATURES
- Package gains two new functions for working with the BOLD taxonomy APIs:
bold_tax_name()andbold_tax_id(), which search for taxonomic data from BOLD using either names or BOLD identifiers, respectively. (#11) - Two new packages in Imports:
jsonliteandreshape.
MINOR IMPROVEMENTS
- Added new taxonomy API functions to the vignette (#14)
- Added reference URLS to all function doc files to allow easy reference for the appropriate API docs.
calloptsparameter changed to...throughout the package, so that passing on options tohttr::GETis done via named parameters, e.g.,config=verbose(). (#13)- Added examples of doing curl debugging throughout man pages.
bold 0.1.2
MINOR IMPROVEMENTS
- Improved the vignette (#8)
- Added small function to print helpful message when user inputs no parameters or zero length parameter values.
BUG FIXES
- Fixed some broken tests with the new
httr(v0.4) (#9), and added a few more tests (#7)
bold 0.1.0
NEW FEATURES
- released to CRAN