56

biomartr

Genomic Data Retrieval

Perform metagenomic data retrieval and functional annotation retrieval. In detail, this package aims to provide users with a standardized way to automate genome, proteome, coding sequence ('CDS'), 'GFF', and metagenome retrieval from 'NCBI' and 'ENSEMBL' databases. Furthermore, an interface to the 'BioMart' database (Smedley et al. (2009) ) allows users to retrieve functional annotation for genomic loci. Users can download entire databases such as 'NCBI RefSeq' (Pruitt et al. (2007) ), 'NCBI nr', 'NCBI nt' and 'NCBI Genbank' (Benson et al. (2013) ) as well as 'ENSEMBL' and 'ENSEMBLGENOMES' with only one command.

Travis-CI Build Status rpackages.io rank

Motivation:

This package is born out of my own frustration to automate the genomic data retrieval process to create computationally reproducible scripts for large-scale genomics studies. Since I couldn't find easy-to-use and fully reproducible software libraries that would allow others and me to write transparent and easy to reproduce code, I sat down and tried to implement a framework that would enable anyone to automate the genomic data retrieval process. Personally, I strongly support and believe in reproducible research, and I truly hope that this package might be useful to others as well and that it helps to promote reproducible research in genomics studies.

I happily welcome anyone who wishes to contribute to this project :) Just drop me an email.

Short package description:

The vastly growing number of sequenced genomes allows us to perform a new type of biological research. Using a comparative approach these genomes provide us with new insights on how biological information is encoded on the molecular level and how this information changes over evolutionary time.

The first step, however, of any genome based study is to retrieve genomes from databases. To automate the retrieval process on a meta-genomic scale, the biomartr package provides useful interface functions for genomic sequence retrieval and functional annotation retrieval. The major aim of biomartr is to facilitate computational reproducibility and large-scale handling of genomic data for (meta-)genomic analyses.

In detail, biomartr aims to provide users with an easy to use framework to obtain genome, proteome, CDS, GFF (annotation), genome assembly quality, and metagenome project data. Furthermore, an interface to the Ensembl Biomart database allows users to retrieve functional annotation for genomic loci. Users can download entire databases such as NCBI RefSeq, NCBI nr, NCBI nt, NCBI Genbank, etc. as well as ENSEMBL and ENSEMBLGENOMES with only one command.

Hence, the biomartr package is designed to achieve the highest degree of computational reproducibility in genomics research.

Citation

Please cite the following paper when using biomartr for your own research. This will allow me to continue working on this software tool and will motivate me to extend its functionality and usability in the next years. Many thanks in advance :)

Platforms

Find biomartr also at OmicTools.

Frequently Asked Questions (FAQs)

Please find all FAQs here.

Discussions and Bug Reports

I would be very happy to learn more about potential improvements of the concepts and functions provided in this package.

Furthermore, in case you find some bugs or need additional (more flexible) functionality of parts of this package, please let me know:

twitter: HajkDrost or email

For Bug Reports: Please send me an issue.

Tutorials

Getting Started with biomartr:

Users can also read the tutorials within (RStudio) :

# source the biomartr package
library(biomartr)
 
# look for all tutorials (vignettes) available in the biomartr package
# this will open your web browser
browseVignettes("biomartr")

Installation

# install biomartr 0.5.2
source("http://bioconductor.org/biocLite.R")
biocLite('biomartr')

Install Developer Version

Some bug fixes or new functionality will not be available on CRAN yet, but in the developer version here on GitHub. To download and install the most recent version of biomartr run:

# install the current version of biomartr on your system
source("http://bioconductor.org/biocLite.R")
biocLite("HajkD/biomartr")

NEWS

The current status of the package as well as a detailed history of the functionality of each version of biomartr can be found in the NEWS section.

Genomic Data Retrieval

Meta-Genome Retrieval

  • meta.retrieval() : Perform Meta-Genome Retieval from NCBI of species belonging to the same kingdom of life or to the same taxonomic subgroup
  • meta.retrieval.all() : Perform Meta-Genome Retieval from NCBI of the entire kingdom of life
  • getMetaGenomes() : Retrieve metagenomes from NCBI Genbank
  • getMetaGenomeAnnotations() : Retrieve annotation *.gff files for metagenomes from NCBI Genbank
  • listMetaGenomes() : List available metagenomes on NCBI Genbank
  • getMetaGenomeSummary() : Helper function to retrieve the assembly_summary.txt file from NCBI genbank metagenomes

Genome Retrieval

  • listGenomes() : List all genomes available on NCBI and ENSEMBL servers
  • listKingdoms() : list the number of available species per kingdom of life on NCBI and ENSEMBL servers
  • listGroups() : list the number of available species per group on NCBI and ENSEMBL servers
  • getKingdoms() : Retrieve available kingdoms of life
  • getGroups() : Retrieve available groups for a kingdom of life
  • is.genome.available() : Check Genome Availability NCBI and ENSEMBL servers
  • getGenome() : Download a specific genome stored on NCBI and ENSEMBL servers
  • getProteome() : Download a specific proteome stored on NCBI and ENSEMBL servers
  • getCDS() : Download a specific CDS file (genome) stored on NCBI and ENSEMBL servers
  • getRNA() : Download a specific RNA file stored on NCBI and ENSEMBL servers
  • getGFF() : Genome Annotation Retrieval from NCBI (*.gff) and ENSEMBL (*.gff3) servers
  • getGTF() : Genome Annotation Retrieval (*.gtf) from ENSEMBL servers
  • getRepeatMasker() : Repeat Masker TE Annotation Retrieval
  • getAssemblyStats() : Genome Assembly Stats Retrieval from NCBI
  • getKingdomAssemblySummary() : Helper function to retrieve the assembly_summary.txt files from NCBI for all kingdoms
  • getMetaGenomeSummary() : Helper function to retrieve the assembly_summary.txt files from NCBI genbank metagenomes
  • getSummaryFile() : Helper function to retrieve the assembly_summary.txt file from NCBI for a specific kingdom
  • getENSEMBLInfo() : Retrieve ENSEMBL info file
  • getGENOMEREPORT() : Retrieve GENOME_REPORTS file from NCBI

Import Downloaded Files

  • read_genome() : Import genomes as Biostrings or data.table object
  • read_proteome() : Import proteome as Biostrings or data.table object
  • read_cds() : Import CDS as Biostrings or data.table object
  • read_gff() : Import GFF file
  • read_rna() : Import RNA file
  • read_rm() : Import Repeat Masker output file
  • read_assemblystats() : Import Genome Assembly Stats File

Database Retrieval

  • listNCBIDatabases() : Retrieve a List of Available NCBI Databases for Download
  • download.database() : Download a NCBI database to your local hard drive
  • download.database.all() : Download a complete NCBI Database such as e.g. NCBI nr to your local hard drive

BioMart Queries

  • biomart() : Main function to query the BioMart database
  • getMarts() : Retrieve All Available BioMart Databases
  • getDatasets() : Retrieve All Available Datasets for a BioMart Database
  • getAttributes() : Retrieve All Available Attributes for a Specific Dataset
  • getFilters() : Retrieve All Available Filters for a Specific Dataset
  • organismBM() : Function for organism specific retrieval of available BioMart marts and datasets
  • organismAttributes() : Function for organism specific retrieval of available BioMart attributes
  • organismFilters() : Function for organism specific retrieval of available BioMart filters

Performing Gene Ontology queries

Gene Ontology

  • getGO() : Function to retrieve GO terms for a given set of genes

Download Developer Version On Windows Systems

# On Windows, this won't work - see ?build_github_devtools
install_github("HajkD/biomartr", build_vignettes = TRUE, dependencies = TRUE)
 
# When working with Windows, first you need to install the
# R package: rtools -> install.packages("rtools")
 
# Afterwards you can install devtools -> install.packages("devtools")
# and then you can run:
 
devtools::install_github("HajkD/biomartr", build_vignettes = TRUE, dependencies = TRUE)
 
# and then call it from the library
library("biomartr", lib.loc = "C:/Program Files/R/R-3.1.1/library")

Troubleshooting on Windows Machines

  • Install biomartr on a Win 8 laptop: solution ( Thanks to Andres Romanowski )

Code of conduct

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

News

biomartr 0.6.0

Function changes

  • the function meta.retrieval() will now pick up the download at the organism where it left off and will report which species have already been retrieved

  • all get*() functions and the meta.retrieval() function receive a new argument reference which allows users to retrieve not-reference or not-representative genome versions when downloading from NCBI RefSeq or NCBI Genbank

  • the argument order in meta.retrieval() changed from meta.retrieval(kingdom, group, db, ...) to meta.retrieval(db,kingdom, group, ...) to make the argument order more consistent with the get*() functions

  • the argument order in getGroups() changed from getGroups(kingdom, db) to getGroups(db, kingdom) to make the argument order more consistent with the get*() and meta.retrieval() functions

New Functions

  • new internal functions existingOrganisms() and existingOrganisms_ensembl() which check the organisms that have already been downloaded

biomartr 0.5.2

Bug fixes

  • fixing bug (https://github.com/ropensci/biomartr/issues/6) that caused incorrect filtering condition when more than one entry for an organism is present in the assemblysummary.txt file at NCBI (Thanks to @kalmeshv)

biomartr 0.5.1

Bug fixes

  • fixing a bug in exists.ftp.file() and getENSEMBLGENOMES.Seq() that caused bacterial genome, proteome, etc retrieval to fail due to the wrong construction of a query ftp request https://github.com/HajkD/biomartr/issues/7 (Many thanks to @dbsseven)

  • fix a major bug in which organisms having no representative genome would generate NULL paths that subsequently crashed the meta.retrieval() function when it tried to print out the result paths.

New Functions

  • new function getRepeatMasker() for retrieval of Repeat Masker output files

  • new function getGTF() for genome annotation retrieval from ensembl and ensemblgenomes in gtf format (Thanks for suggesting it Ge Tan)

  • new function getRNA() to perform RNA Sequence Retrieval from NCBI and ENSEMBL databases (Thanks for suggesting it @carlo-berg)

  • new function read_rna() for importing Repeat Masker output files downloaded with getRepeatMasker()

  • new function read_rm() for importing RNA downloaded with getRNA() as Biostrings or data.table object

  • new helper function custom_download() that aims to make the download process more robust and stable -> In detail, the download process is now adapting to the operating system, e.g. using either curl (macOS), wget (Linux), or wininet (Windows)

Function changes

  • function name listDatabases() has been renamed listNCBIDatabases(). In biomartr version 0.6.0 the function name listDatabases() will be depreciated

  • meta.retieval() and meta.retieval.all() now allow the bulk retrieval of GTF files for type = 'ensembl' and type = 'esnemblgenomes' via type = "gtf". See getGTF() for more details.

  • meta.retieval() and meta.retieval.all() now allow the bulk retrieval of RNA files via type = "rna". See getRNA() for more details.

  • meta.retieval() and meta.retieval.all() now allow the bulk retrieval of Repeat Masker output files via type = "rm". See getRepeatMasker() for more details.

  • all get*() retrieval functions now skip the download of a particular file if it already exists in the specified file path

  • download.database() and download.database.all() now internally perform md5 check sum checks to make sure that the file download was successful

  • download.database() and download.database.all() now return the file paths of the downloaded file so that it is easier to use these functions when constructing pipelines, e.g. download.database() %>% ... or download.database.all() %>% ....

  • meta.retrieval() and meta.retrieval.all() now return the file paths of the downloaded file so that it is easier to use these functions when constructing pipelines, e.g. meta.retrieval() %>% ... or meta.retrieval() %>% ....

  • getGenome(), getProteome(), getCDS(), getRNA(), getGFF(), and getAssemblyStats() now internally perform md5 checksum tests to make sure that files are retrieved intact.

biomartr 0.4.0

Bug fixes

  • fixing a major bug https://github.com/HajkD/biomartr/issues/6 that caused that in all get*() (genome, proteome, gff, etc.) and meta.retrieval*() functions the meta retrieval process errored and terminated whenever NCBI or ENSEMBL didn't store all types of sequences for a particular organism: genome, proteome, cds, etc. This has been fixed now and function calls such as meta.retrieval(kingdom = "bacteria", db = "genbank", type = "proteome") should work properly now (Thanks to @ARamesh123 for making me aware if this bug). Hence, this bug affected all attempts to download all proteome sequences e.g. for bacteria and viruses, because NCBI does not store genome AND proteome information for all bacterial or viral species.

New Functions

Function changes

  • meta.retrieval() and meta.retrieval.all() can now also download genome assembly stats for all selected species

  • meta.retrieval() receives a new argument group that allows users to retrieve species belonging to a subgroup instead of the entire kingdom. Available groups can be retrieved with getGroups().

  • functions getSubgroups() and listSubgroups() have been removed and their initial functionality has been merged and integrated into getGroups() and listGroups()

  • listGroups() receives a new argument details that allows users to retrieve the organism names that belong to the corresponding subgroups

  • getGroups() is now based on listGroups()

  • internal function getGENOMESREPORT() is now exported and available to the user

  • all organism*() functions now also support Ensembl Plants, Ensembl Metazoa, Ensembl Protist, and Ensembl Fungi (Thanks for pointing out Alex Gabel)

  • getMarts() and getDatasets() now also support Ensembl Plants, Ensembl Metazoa, Ensembl Protist, and Ensembl Fungi (Thanks for pointing out Alex Gabel)

Vignette updates

  • Vignette Meta-Genome Retrieval has more examples how to download genomes of species that belong to the same subgroup

biomartr 0.3.0

Bug fixes

  • Fixing a bug https://github.com/HajkD/biomartr/issues/2 based on the readr package that affected the getSummaryFile(), getKingdomAssemblySummary(), getMetaGenomeSummary(), getENSEMBL.Seq() and getENSEMBLGENOMES.Seq() functions causing quoted lines in the assembly_summary.txt to be omitted when reading these files. This artefact caused that e.g. instead of information of 80,000 Bacteria genomes only 40,000 (which non-quotations) were read (Thanks to Xin Wu).

biomartr 0.2.1

In this version of biomartr the organism*() functions were adapted to the new ENSEMBL 87 release in which organism name specification in the Biomart description column was changed from a scientific name convention to a mix of common name and scientific name convention.

  • all organism*() functions have been adapted to the new ENSEMBL 87 release organism name notation that is used in the Biomart description

  • fixing error handling bug that caused commands such as download.database(db = "nr.27.tar.gz") to not execute properly

biomartr 0.2.0

In this version, biomartr was extended to now retrieve genome, proteome, CDS, GFF and meta-genome data also from ENSEMBL and ENSEMLGENOMES. Furthermore, all NCBI retrieval functions were updated to the new server folder structure standards of NCBI.

New Functions

  • new meta-retrieval function meta.retrieval.all() allows users to download all individual genomes of all kingdoms of life with one command

  • new metagenome retrieval function getMetaGenomes() allows users to retrieve metagenome projects from NCBI Genbank

  • new metagenome retrieval function getMetaGenomeAnnotations() allows users to retrieve annotation files for genomes belonging to a metagenome project stored at NCBI Genbank

  • new retrieval function getGFF() allows users to retrieve annotation (*.gff) files for specific genomes from NCBI and ENSEMBL databases

  • new import function read_gff() allowing users to import GFF files downloaded with getGFF()

  • new internal functions to check for availability of ENSEMBL or ENSEMBLGENOMES databases

  • new database retrieval function download.database.all() allows users to download entire NCBI databases with one command

  • new function listMetaGenomes() allowing users to list available metagenomes on NCBI Genbank

  • new external helper function getSummaryFile() to retrieve the assembly_summary.txt file from NCBI

  • new external helper function getKingdomAssemblySummary() to retrieve the assembly_summary.txt files from NCBI for all kingdoms and combine them into one big data.frame

  • new function listKingdoms() allows users to list the number of available species per kingdom of life

  • new function listGroups() allows users to list the number of available species per group

  • new function listSubgroups() allows users to list the number of available species per subgroup

  • new function getGroups() allows users to retrieve available groups for a kingdom of life

  • new function getSubgroups() allows users to retrieve available subgroups for a kingdom of life

  • new external helper function getMetaGenomeSummary() to retrieve the assembly_summary.txt files from NCBI genbank metagenomes

  • new internal helper function getENSEMBL.Seq() acting as main interface function to communicate with the ENSEMBL database API for sequence retrieval

  • new internal helper function getENSEMBLGENOMES.Seq() acting as main interface function to communicate with the ENSEMBL database API for sequence retrieval

  • new internal helper function getENSEMBL.Annotation() acting as main interface function to communicate with the ENSEMBL database API for GFF retrieval

  • new internal helper function getENSEMBLGENOMES.Annotation() acting as main interface function to communicate with the ENSEMBL database API for GFF retrieval

  • new internal helper function get.ensemblgenome.info() to retrieve general organism information from ENSEMBLGENOMES

  • new internal helper function get.ensembl.info() to retrieve general organism information from ENSEMBL

  • new internal helper function getGENOMEREPORT() to retrieve the genome reports file from ftp://ftp.ncbi.nlm.nih.gov/genomes/GENOME_REPORTS/overview.txt

  • new internal helper function connected.to.internet() enabling internet connection check

Function changes

  • functions getGenome(), getProteome(), and getCDS() now can also in addition to NCBI retrieve genomes, proteomes or CDS from ENSEMBL and ENSEMLGENOMES

  • the functions getGenome(), getProteome(), and getCDS() were completely re-written and now use the assembly_summary.txt files provided by NCBI to retrieve the download path to the corresponding genome. Furthermore, these functions now lost the kingdom argument. Users now only need to specify the organism name and not the kingdom anymore. Furthermore, all get* functions now return the path to the downloaded genome so that this path can be used as input to all read_* functions.

  • download_databases() has been renamed to download.databases() to be more consistent with other function notation

  • the argument db_format was removed from listDatabases() and download.database() because it was misleading

  • the command listDatabases("all") now returns all available NCBI databases that can be retrieved with download.database()

  • download.database() now internally checks if input database specified by the user is actually available on NCBI servers

  • the documentary file generated by getGenome(), getProteome(), and getCDS() is now extended to store more details about the downloaded genome

  • argument database in is.genome.available() and listGenomes() has been renamed to db to be consistent with all other sequence retrieval functions

  • is.genome.available() now also checks availability of organisms in ENSEMBL. See db = "ensembl"

  • the argument db_name in listDatabases() has been renamed db to be more consistent with the notation in other functions

  • the argument name in download.database() has been renamed db to be more consistent with the notation in other functions

  • getKingdoms() now retrieves also kingdom information for ENSEMBL and ENSEMBLGENOMES

  • getKingdoms() received new argument db to specify from which database (e.g. refseq, genbank, ensembl or ensemblgenomes) kingdom information shall be retrieved

  • getKingdoms(db = "refseq") received one more member: "viral", allowing the genome retrieval of all viruses

  • argument out.folder in meta.retrieval() has been renamed to path to be more consistent with other retrieval functions

  • all read_* functions now received a new argument obj.type allowing users to choose between storing input genomes as Biostrings object or data.table object

  • all read_* functions now have format = "fasta" as default

  • the kingdom argument in the listGenomes() function was renamed to type, now allowing users to specify not only specify kingdoms, but also groups and subgroups. Use: listGenomes(type = "kingdom") or listGenomes(type = "group") or listGenomes(type = "subgroup")

  • the listGenomes() function receives a new argument subset to specify a subset of the selected type argument. E.g. subset = "Eukaryota" when specifying type = "kingdom"

Vignette updates

  • new Vignette Meta-Genome Retrieval
  • Update examples and extend Introduction Vignette
  • Update examples and extend Database Retrieval Vignette
  • Update examples and extend Sequence Retrieval Vignette
  • Update examples and extend Functional Annotation Vignette

biomartr 0.1.0

  • fixing a parsing error of the file ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/vertebrate_mammalian/assembly_summary.txt The problem was that comment lines were introduced and columns couldn't be parsed correctly anymore. This caused that genomes, proteomes, and CDS files could not be downloaded properly. This has been fixed now.

  • genomes, proteome, and CDS as well as meta-genomes can now be retrieved from RefSeq and Genbank (not only RefSeq); only getCDS() does not have genebank access, becasue genbank does not provide CDS sequences

  • adding new function meta.retrieval() to mass retrieve genomes for entire kingdoms of life

  • fixed a major bug in organismBM() causing the function to fail. The failure of this function affected all downstream organism*() functions. Bug is now fixed and everything works properly

  • updated Vignettes

biomartr 0.0.3

  • updating unit tests for new API

  • fixing API problems that caused all BioMart related functions to fail

  • fixing retrieval problems in getCDS(), getProteome(), and getGenome()

  • the listDatabases() function now has a new option db_name = "all" allowing users to list all available databases stored on NCBI

Vignettes

  • adding new vignette: Database Retrieval
  • update the vignettes: Phylotranscriptomics, Sequence Retrieval, and Functional Annotation

biomartr 0.0.2

Vignettes

  • adding vignettes: Introduction, Functional Annotation, Phylotranscriptomics, and Sequence Retrieval

biomartr 0.0.1

Release Version

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.
install.packages("biomartr")

0.7.0 by Hajk-Georg Drost, 5 months ago

https://github.com/ropensci/biomartr

Report a bug at https://github.com/ropensci/biomartr/issues

Browse source code at https://github.com/cran/biomartr

Authors: Hajk-Georg Drost [aut, cre] (http://orcid.org/0000-0002-1567-306X)

Documentation:   PDF Manual  

GPL-2 license

Imports biomaRt, Biostrings, curl, tibble, jsonlite, data.table, dplyr, readr, downloader, RCurl, XML, httr, stringr, purrr

Suggests knitr, rmarkdown, devtools, testthat, seqinr

Imported by myTAI.

See at CRAN

R links

R homepage

Download R

Mailing lists

R documentation

R manuals

R FAQs

The R Journal

CRAN links

CRAN homepage

CRAN repository policy

Submit a package

METACRAN stuff

About METACRAN

At github

Report a bug