Critical Assessment of Metagenome Interpretation

A benchmark of metagenomics software

Alexander Sczyrba, Peter Hofmann, Peter Belmann, David Koslicki, Stefan Jannsen, Johannes Dröge, Ivan Gregor, Stephan Majda, Jessika Fiedler, Eik Dahms, Andreas Bremges, Adrian Fritz, Ruben Garrido-Oter, Tue Sparholt Jørgensen

Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

Resumé

Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.
OriginalsprogEngelsk
TidsskriftNature Methods
Vol/bind14
Udgave nummer11
Sider (fra-til)1063-1071
Antal sider9
ISSN1548-7091
DOI
StatusUdgivet - 2 okt. 2017

Citer dette

Sczyrba, A., Hofmann, P., Belmann, P., Koslicki, D., Jannsen, S., Dröge, J., ... Jørgensen, T. S. (2017). Critical Assessment of Metagenome Interpretation: A benchmark of metagenomics software. Nature Methods, 14(11), 1063-1071. https://doi.org/10.1038/nmeth.4458
Sczyrba, Alexander ; Hofmann, Peter ; Belmann, Peter ; Koslicki, David ; Jannsen, Stefan ; Dröge, Johannes ; Gregor, Ivan ; Majda, Stephan ; Fiedler, Jessika ; Dahms, Eik ; Bremges, Andreas ; Fritz, Adrian ; Garrido-Oter, Ruben ; Jørgensen, Tue Sparholt. / Critical Assessment of Metagenome Interpretation : A benchmark of metagenomics software. I: Nature Methods. 2017 ; Bind 14, Nr. 11. s. 1063-1071.
@article{e309cfb2b1c047aabc93729481c2597f,
title = "Critical Assessment of Metagenome Interpretation: A benchmark of metagenomics software",
abstract = "Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.",
keywords = "Classification and taxonomy, Computational biology, Metagenomics, Bioinformatics",
author = "Alexander Sczyrba and Peter Hofmann and Peter Belmann and David Koslicki and Stefan Jannsen and Johannes Dr{\"o}ge and Ivan Gregor and Stephan Majda and Jessika Fiedler and Eik Dahms and Andreas Bremges and Adrian Fritz and Ruben Garrido-Oter and J{\o}rgensen, {Tue Sparholt}",
year = "2017",
month = "10",
day = "2",
doi = "10.1038/nmeth.4458",
language = "English",
volume = "14",
pages = "1063--1071",
journal = "Nature Methods",
issn = "1548-7091",
publisher = "Nature Publishing Group",
number = "11",

}

Sczyrba, A, Hofmann, P, Belmann, P, Koslicki, D, Jannsen, S, Dröge, J, Gregor, I, Majda, S, Fiedler, J, Dahms, E, Bremges, A, Fritz, A, Garrido-Oter, R & Jørgensen, TS 2017, 'Critical Assessment of Metagenome Interpretation: A benchmark of metagenomics software', Nature Methods, bind 14, nr. 11, s. 1063-1071. https://doi.org/10.1038/nmeth.4458

Critical Assessment of Metagenome Interpretation : A benchmark of metagenomics software. / Sczyrba, Alexander; Hofmann, Peter; Belmann, Peter; Koslicki, David; Jannsen, Stefan; Dröge, Johannes; Gregor, Ivan; Majda, Stephan; Fiedler, Jessika; Dahms, Eik; Bremges, Andreas; Fritz, Adrian; Garrido-Oter, Ruben; Jørgensen, Tue Sparholt.

I: Nature Methods, Bind 14, Nr. 11, 02.10.2017, s. 1063-1071.

Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

TY - JOUR

T1 - Critical Assessment of Metagenome Interpretation

T2 - A benchmark of metagenomics software

AU - Sczyrba, Alexander

AU - Hofmann, Peter

AU - Belmann, Peter

AU - Koslicki, David

AU - Jannsen, Stefan

AU - Dröge, Johannes

AU - Gregor, Ivan

AU - Majda, Stephan

AU - Fiedler, Jessika

AU - Dahms, Eik

AU - Bremges, Andreas

AU - Fritz, Adrian

AU - Garrido-Oter, Ruben

AU - Jørgensen, Tue Sparholt

PY - 2017/10/2

Y1 - 2017/10/2

N2 - Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

AB - Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

KW - Classification and taxonomy

KW - Computational biology

KW - Metagenomics

KW - Bioinformatics

U2 - 10.1038/nmeth.4458

DO - 10.1038/nmeth.4458

M3 - Journal article

VL - 14

SP - 1063

EP - 1071

JO - Nature Methods

JF - Nature Methods

SN - 1548-7091

IS - 11

ER -

Sczyrba A, Hofmann P, Belmann P, Koslicki D, Jannsen S, Dröge J et al. Critical Assessment of Metagenome Interpretation: A benchmark of metagenomics software. Nature Methods. 2017 okt 2;14(11):1063-1071. https://doi.org/10.1038/nmeth.4458