Fisher RA. The Design of Experiments. New York: Hafner Press; 1935.
Google Scholar
Popper KR. The logic of scientific discovery. London: Routledge; 1959.
Google Scholar
Peng RD. Reproducible research in computational science. Science. 2011;334:1226–7.
Article
CAS
PubMed
PubMed Central
Google Scholar
Russell JF. If a job is worth doing, it is worth doing twice. Nature. 2013;496:7.
Article
CAS
PubMed
Google Scholar
Feynman RP. Six Easy Pieces: Essentials of Physics Explained by Its Most Brilliant Teacher. Boston, MA: Addison-Wesley; 1995. p. 34–5.
Murray-Rust P, Murray-Rust D. Reproducible Physical Science and the Declaratron. In: Stodden VC, Leisch F, Peng RD, editors. Implementing Reproducible Research. Boca Raton, FL: CRC Press; 2014. p. 113.
Hey AJG, Tansley S, Tolle KM, Others. The fourth paradigm: data-intensive scientific discovery. Redmond, WA: Microsoft Research Redmond, WA; 2009.
Millman KJ, Pérez F. Developing Open-Source Scientific Practice. Implementing Reproducible Research. Boca Raton, FL: CRC Press; 2014;149.
Wilson G, Aruliah DA, Brown CT, Chue Hong NP, Davis M, Guy RT, et al. Best practices for scientific computing. PLoS Biol. 2014;12:e1001745.
Article
PubMed
PubMed Central
Google Scholar
Software with impact. Nat Methods. 2014;11:211.
Hong NC. We are the 92% [Internet]. Figshare; 2014. Available from: http://dx.doi.org/10.6084/M9.FIGSHARE.1243288. Accessed 1 March 2016.
Sacks J, Welch WJ, Mitchell TJ, Wynn HP. Design and analysis of computer experiments. Stat Sci. 1989;4:409–23.
Article
Google Scholar
Garijo D, Kinnings S, Xie L, Xie L, Zhang Y, Bourne PE, et al. Quantifying reproducibility in computational biology: the case of the tuberculosis drugome. PLoS One. 2013;8:e80278.
Article
PubMed
PubMed Central
Google Scholar
Error prone. Nature. 2012;487:406.
Vandewalle P, Barrenetxea G, Jovanovic I, Ridolfi A, Vetterli M. Experiences with reproducible research in various facets of signal processing research. IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP’07. IEEE. 2007;2007:IV-1253–6.
Google Scholar
Cassey P, Cassey P, Blackburn T, Blackburn T. Reproducibility and repeatability in ecology. Bioscience. 2006;56:958–9.
Article
Google Scholar
Murphy JM, Sexton DMH, Barnett DN, Jones GS, Webb MJ, Collins M, et al. Quantification of modelling uncertainties in a large ensemble of climate change simulations. Nature. 2004;430:768–72.
Article
CAS
PubMed
Google Scholar
McCarthy DJ, Humburg P, Kanapin A, Rivas MA, Gaulton K, Cazier J-B, et al. Choice of transcripts and software has a large effect on variant annotation. Genome Med. 2014;6:26.
Article
PubMed
PubMed Central
Google Scholar
Neuman JA, Isakov O, Shomron N. Analysis of insertion-deletion from deep-sequencing data: Software evaluation for optimal detection. Brief Bioinform. 2013;14:46–55.
Article
PubMed
Google Scholar
Bradnam KR, Fass JN, Alexandrov A, Baranay P, Bechner M, Birol I, et al. Assemblathon 2: evaluating de novo methods of genome assembly in three vertebrate species. Gigascience. 2013;2:10.
Article
PubMed
PubMed Central
Google Scholar
Bilal E, Dutkowski J, Guinney J, Jang IS, Logsdon BA, Pandey G, et al. Improving breast cancer survival analysis through competition-based multidimensional modeling. PLoS Comput Biol. 2013;9:e1003047.
Article
CAS
PubMed
PubMed Central
Google Scholar
Gronenschild EHBM, Habets P, Jacobs HIL, Mengelers R, Rozendaal N, van Os J, et al. The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements. PLoS One. 2012;7:e38234.
Article
CAS
PubMed
PubMed Central
Google Scholar
Moskvin OV, McIlwain S, Ong IM. CAMDA 2014: Making sense of RNA-Seq data: From low-level processing to functional analysis. Systems Biomedicine. 2014;2:31–40.
Article
Google Scholar
Reducing our irreproducibility. Nature. 2013;496:398–398.
Michael CM, Nass SJ, Omenn GS, editors. Evolution of Translational Omics: Lessons Learned and the Path Forward. Washington, D.C: The National Academies Press; 2012.
Google Scholar
Collins FS, Tabak L a. Policy: NIH plans to enhance reproducibility. Nature. 2014;505:612–3.
Article
PubMed
PubMed Central
Google Scholar
Chambers JM. S as a Programming Environment for Data Analysis and Graphics. Problem Solving Environments for Scientific Computing, Proceedings 17th Symposium on the Interface of Statistics and Computing North Holland; 1985. p. 211–4.
LeVeque RJ, Mitchell IM, Stodden V. Reproducible research for scientific computing: Tools and strategies for changing the culture. Comput Sci Eng. 2012;14:13.
Article
Google Scholar
Stodden V, Guo P, Ma Z. Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals. PLoS One. 2013;8:2–9.
Article
Google Scholar
Morin A, Urban J, Adams PD, Foster I, Sali A, Baker D, et al. Research priorities. Shining light into black boxes. Science. 2012;336:159–60.
Article
CAS
PubMed
PubMed Central
Google Scholar
Rebooting review. Nat Biotechnol. 2015;33:319.
Ioannidis JP a, Allison DB, Ball C a, Coulibaly I, Cui X, Culhane AC, et al. Repeatability of published microarray gene expression analyses. Nat Genet. 2009;41:149–55.
Article
CAS
PubMed
Google Scholar
Nekrutenko A, Taylor J. Next-generation sequencing data interpretation: enhancing reproducibility and accessibility. Nat Rev Genet. 2012;13:667–72.
Article
CAS
PubMed
Google Scholar
Baggerly K a, Coombes KR. Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology. Ann Appl Stat. 2009;3:1309–34.
Article
Google Scholar
Decullier E, Huot L, Samson G, Maisonneuve H. Visibility of retractions: a cross-sectional one-year study. BMC Res Notes. 2013;6:238.
Article
PubMed
PubMed Central
Google Scholar
Claerbout JF, Karrenbach M. Electronic Documents Give Reproducible Research a New Meaning. Meeting of the Society of Exploration Geophysics. New Orleans, LA; 1992.
Stodden V, Miguez S. Best practices for computational science: software infrastructure and environments for reproducible and extensible research. J Open Res Softw. 2014;2:21.
Article
Google Scholar
Ravel J, Wommack KE. All hail reproducibility in microbiome research. Microbiome. 2014;2:8.
Article
PubMed
PubMed Central
Google Scholar
Stodden V. 2014: What scientific idea is ready for retirement? [Internet]. http://edge.org/response-detail/25340. 2014. Available from: http://edge.org/response-detail/25340. Accessed 1 March 2016.
Birney E, Hudson TJ, Green ED, Gunter C, Eddy S, Rogers J, et al. Prepublication data sharing. Nature. 2009;461:168–70.
Article
PubMed
Google Scholar
Hothorn T, Leisch F. Case studies in reproducibility. Brief Bioinform. 2011;12:288–300.
Article
PubMed
Google Scholar
Schofield PN, Bubela T, Weaver T, Portilla L, Brown SD, Hancock JM, et al. Post-publication sharing of data and tools. Nature. 2009;461:171–3.
Article
CAS
PubMed
Google Scholar
Piwowar H a., Day RS, Fridsma DB. Sharing detailed research data is associated with increased citation rate. PLoS One. 2007;2.
Johnson VE. Revised standards for statistical evidence. Proc Natl Acad Sci U S A. 2013;110:19313–7.
Article
CAS
PubMed
PubMed Central
Google Scholar
Halsey LG, Curran-everett D, Vowler SL, Drummond GB. The fickle P value generates irreproducible results. Nat Methods. 2015;12:179–85.
Article
CAS
PubMed
Google Scholar
Wilson G. Software Carpentry: lessons learned. F1000Res. 2016;3:62.
PubMed Central
Google Scholar
Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten simple rules for reproducible computational research. PLoS Comput Biol. 2013;9:1–4.
Article
Google Scholar
GNU Make [Internet]. 2016. Available from https://www.gnu.org/software/make. Accessed 1 March 2016.
Make for Windows [Internet]. 2016. Available from http://gnuwin32.sourceforge.net/packages/make.htm. Accessed 1 March 2016.
Puppet [Internet]. 2016. Available from https://puppetlabs.com. Accessed 1 March 2016.
Code share. Nature. 2014;514:536.
Blischak JD, Davenport ER, Wilson G. A quick introduction to version control with Git and GitHub. PLoS Comput Biol. 2016;12:e1004668.
Article
PubMed
PubMed Central
Google Scholar
Loeliger J, McCullough M. Version Control with Git: Powerful Tools and Techniques for Collaborative Software Development. Sebastopol, California: “O’Reilly Media, Inc.”; 2012. p. 456.
Huber W, Carey VJ, Gentleman R, Anders S, Carlson M, Carvalho BS, et al. Orchestrating high-throughput genomic analysis with Bioconductor. Nat Methods. 2015;12:115–21.
Article
CAS
PubMed
PubMed Central
Google Scholar
R Core Team. R: A Language and Environment for Statistical Computing [Internet]. Vienna, Austria: R Foundation for Statistical Computing; 2014. Available from: http://www.r-project.org. Accessed 1 March 2016.
Tóth G, Sokolov IV, Gombosi TI, Chesney DR, Clauer CR, De Zeeuw DL, et al. Space weather modeling framework: a new tool for the space science community. J Geophys Res. 2005;110:A12226.
Article
Google Scholar
Tan E, Choi E, Thoutireddy P, Gurnis M, Aivazis M. GeoFramework: Coupling multiple models of mantle convection within a computational framework. Geochem Geophys Geosyst. [Internet]. 2006;7. Available from: http://doi.wiley.com/10.1029/2005GC001155
Heisen B, Boukhelef D, Esenov S, Hauf S, Kozlova I, Maia L, et al. Karabo: An Integrated Software Framework Combining Control, Data Management, and Scientific Computing Tasks. 14th International Conference on Accelerator & Large Experimental Physics Control Systems, ICALEPCS2013. San Francisco, CA; 2013.
Schneider CA, Rasband WS, Eliceiri KW. NIH Image to ImageJ: 25 years of image analysis. Nat Methods. 2012;9:671–5.
Article
CAS
PubMed
Google Scholar
Schindelin J, Arganda-Carreras I, Frise E, Kaynig V, Longair M, Pietzsch T, et al. Fiji: an open-source platform for biological-image analysis. Nat Methods. 2012;9:676–82.
Article
CAS
PubMed
Google Scholar
Biasini M, Schmidt T, Bienert S, Mariani V, Studer G, Haas J, et al. OpenStructure: an integrated software framework for computational structural biology. Acta Crystallogr D Biol Crystallogr. 2013;69:701–9.
Article
CAS
PubMed
PubMed Central
Google Scholar
Ivy, the agile dependency manager [Internet]. 2016. Available from http://ant.apache.org/ivy. Accessed 1 March 2016.
aRchive: Enabling reproducibility of Bioconductor package versions (for Galaxy) [Internet]. 2016. Available from http://bioarchive.github.io. Accessed 1 March 2016.
Martin RC. Clean code: a handbook of agile software craftsmanship. Pearson Education. 2009.
Google Scholar
Knuth DE. Literate programming. Comput J. 1984;27:97–111.
Article
Google Scholar
Pérez F, Granger BE. IPython: a system for interactive scientific computing. Comput Sci Eng. 2007;9:21–9.
Article
Google Scholar
Shen H. Interactive notebooks: Sharing the code. Nature. 2014;515:151–2.
Article
CAS
PubMed
Google Scholar
Xie Y. Dynamic Documents with R and knitr. Boca Raton, FL: CRC Press; 2013. p. 216.
RStudio Team. RStudio: Integrated Development for R [Internet]. [cited 2015 Nov 20]. Available from: http://www.rstudio.com. Accessed 1 March 2016.
Gross AM, Orosco RK, Shen JP, Egloff AM, Carter H, Hofree M, et al. Multi-tiered genomic analysis of head and neck cancer ties TP53 mutation to 3p loss. Nat Genet. 2014;46:1–7.
Article
Google Scholar
Ding T, Schloss PD. Dynamics and associations of microbial community types across the human body. Nature. 2014;509:357–60.
Article
CAS
PubMed
PubMed Central
Google Scholar
Ram Y, Hadany L. The probability of improvement in Fisher’s geometric model: A probabilistic approach. Theor Popul Biol. 2015;99:1–6.
Article
PubMed
Google Scholar
Meadow JF, Altrichter AE, Kembel SW, Moriyama M, O’Connor TK, Womack AM, et al. Bacterial communities on classroom surfaces vary with human contact. Microbiome. 2014;2:7.
Article
PubMed
PubMed Central
Google Scholar
White E. Programming for Biologists [Internet]. Available from: http://www.programmingforbiologists.org. Accessed 1 March 2016.
Peng RD, Leek J, Caffo B. Coursera course: Exploratory Data Analysis [Internet]. Available from: https://www.coursera.org/learn/exploratory-data-analysis.
Bioconductor - Courses and Conferences [Internet]. [cited 2015 Nov 20]. Available from: http://master.bioconductor.org/help/course-materials. Accessed 1 March 2016.
Gil Y, Deelman E, Ellisman M, Fahringer T, Fox G, Gannon D, et al. Examining the challenges of scientific workflows. Computer. 2007;40:24–32.
Article
Google Scholar
Giardine B, Riemer C, Hardison RC, Burhans R, Elnitski L, Shah P, et al. Galaxy: a platform for interactive large-scale genome analysis. Genome Res. 2005;15:1451–5.
Article
CAS
PubMed
PubMed Central
Google Scholar
Goecks J, Nekrutenko A, Taylor J. Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biol. 2010;11:R86.
Article
PubMed
PubMed Central
Google Scholar
Afgan E, Baker D, Coraor N, Goto H, Paul IM, Makova KD, et al. Harnessing cloud computing with Galaxy Cloud. Nat Biotechnol. 2011;29:972–4.
Article
CAS
PubMed
PubMed Central
Google Scholar
Callahan SP, Freire J, Santos E, Scheidegger CE, Silva CT, Vo HT. VisTrails: Visualization Meets Data Management. Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data. New York, NY, USA: ACM; 2006. p. 745–7.
Davidson SB, Freire J. Provenance and scientific workflows. Proceedings of the 2008 ACM SIGMOD international conference on Management of data - SIGMOD’08. 2008. p. 1345.
Lazarus R, Kaspi A, Ziemann M. Creating re-usable tools from scripts: The Galaxy Tool Factory. Bioinformatics. 2012;28:3139–40.
Article
CAS
PubMed
PubMed Central
Google Scholar
Dudley JT, Butte AJ. In silico research in the era of cloud computing. Nat Biotechnol. 2010;28:1181–5.
Article
CAS
PubMed
PubMed Central
Google Scholar
Hurley DG, Budden DM, Crampin EJ. Virtual Reference Environments: a simple way to make research reproducible. Brief Bioinform. 2015;16(5)901–903.
Article
PubMed
Google Scholar
Gent IP. The Recomputation Manifesto. arXiv [Internet]. 2013; Available from: http://arxiv.org/abs/1304.3674. Accessed 1 March 2016.
Howe B. Virtual appliances, cloud computing, and reproducible research. Comput Sci Eng. 2012;14:36–41.
Article
Google Scholar
Brown CT. Virtual machines considered harmful for reproducibility [Internet]. 2012. Available from: http://ivory.idyll.org/blog/vms-considered-harmful.html. Accessed 1 March 2016.
Piccolo SR. Building portable analytical environments to improve sustainability of computational-analysis pipelines in the sciences [Internet]. 2014. Available from: http://dx.doi.org/10.6084/m9.figshare.1112571. Accessed 1 March 2016.
Krampis K, Booth T, Chapman B, Tiwari B, Bicak M, Field D, et al. Cloud BioLinux: pre-configured and on-demand bioinformatics computing for the genomics community. BMC Bioinformatics. 2012;13:42.
Article
PubMed
PubMed Central
Google Scholar
CloudBioLinux: configure virtual (or real) machines with tools for biological analyses [Internet]. 2016. Available from https://github.com/chapmanb/cloudbiolinux. Accessed 1 March 2016.
Felter W, Ferreira A, Rajamony R, Rubio J. An Updated Performance Comparison of Virtual Machines and Linux Containers [Internet]. IBM Research Division; 2014. Available from: http://domino.research.ibm.com/library/CyberDig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf. Accessed 1 March 2016.
Eglen SJ, Weeks M, Jessop M, Simonotto J, Jackson T, Sernagor E. A data repository and analysis framework for spontaneous neural activity recordings in developing retina. Gigascience. 2014;3:3.
Article
PubMed
PubMed Central
Google Scholar
Eglen SJ. Bivariate spatial point patterns in the retina: a reproducible review. Journal de la Société Française de Statistique. 2016;157:33–48.
Google Scholar
Bremges A, Maus I, Belmann P, Eikmeyer F, Winkler A, Albersmeier A, et al. Deeply sequenced metagenome and metatranscriptome of a biogas-producing microbial community from an agricultural production-scale biogas plant. Gigascience. 2015;4:33.
Article
PubMed
PubMed Central
Google Scholar
Belmann P, Dröge J, Bremges A, McHardy AC, Sczyrba A, Barton MD. Bioboxes: standardised containers for interchangeable bioinformatics software. Gigascience. 2015;4:47.
Article
PubMed
PubMed Central
Google Scholar
Barton M. nucleotides · genome assembler benchmarking [Internet]. [cited 2015 Nov 20]. Available from: http://nucleotid.es. Accessed 1 March 2016.
Hones MJ. Reproducibility as a Methodological Imperative in Experimental Research. PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association. Philosophy of Science Association. 1990. p. 585–99.
Google Scholar
Rosenberg DM, Horn CC. Neurophysiological analytics for all! Free open-source software tools for documenting, analyzing, visualizing, and sharing using electronic notebooks. J Neurophysiol American Physiological Society; Apr2016;jn.00137.2016.
everware [Internet]. 2016. Available from https://github.com/everware/everware. Accessed 1 March 2016.
Crick T. “Share and Enjoy”: Publishing Useful and Usable Scientific Models. Available from: http://arxiv.org/abs/1409.0367v2. Accessed 1 March 2016.
Donoho DL. An invitation to reproducible computational research. Biostatistics. 2010;11:385–8.
Article
PubMed
Google Scholar
Goldberg D. What every computer scientist should know about floating-point arithmetic. ACM Comput Surv. 1991;23:5–48.
Article
Google Scholar
Shirts M, Pande VS. COMPUTING: screen savers of the world unite! Science. 2000;290:1903–4.
Article
CAS
PubMed
Google Scholar
Bird I. Computing for the large hadron Collider. Annu Rev Nucl Part Sci. 2011;61:99–118.
Article
CAS
Google Scholar
Anderson DP. BOINC: A System for Public Resource Computing and Storage. Proceedings of the Fifth IEEE/ACM International Workshop on Grid Computing (GRID’04). 2004.
Ransohoff DF. Bias as a threat to the validity of cancer molecular-marker research. Nat Rev Cancer. 2005;5:142–9.
Article
CAS
PubMed
Google Scholar
Bild AH, Chang JT, Johnson WE, Piccolo SR. A field guide to genomics research. PLoS Biol. 2014;12:e1001744.
Article
PubMed
PubMed Central
Google Scholar
Köster J, Rahmann S. Snakemake—a scalable bioinformatics workflow engine. Bioinformatics. 2012;28:2520–2.
Article
PubMed
Google Scholar
Sadedin SP, Pope B, Oshlack A. Bpipe : a tool for running and managing bioinformatics pipelines. Bioinformatics. 2012;28:1525–6.
Article
CAS
PubMed
Google Scholar
Tange O. GNU Parallel - The Command-Line Power Tool.;login: The USENIX Magazine. Frederiksberg, Denmark; 2011;36:42–7
Albrecht M, Donnelly P, Bui P, Thain D. Makeflow: A portable abstraction for data intensive computing on clusters, clouds, and grids. Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies. 2012.
Knight S, Austin C, Crain C, Leblanc S, Roach A. Scons software construction tool [Internet]. 2011. Available from: http://www.scons.org. Accessed 1 March 2016.
Altintas I, Berkley C, Jaeger E, Jones M, Ludascher B, Mock S. Kepler: an extensible system for design and execution of scientific workflows. Proceedings. 16th International Conference on Scientific and Statistical Database Management, 2004. IEEE; 2004. p. 423–4.
Goff SA, Vaughn M, McKay S, Lyons E, Stapleton AE, Gessler D, et al. The iPlant collaborative: cyberinfrastructure for plant biology. Front Plant Sci Frontiers. 2011;2:34.
Google Scholar
Reich M, Liefeld T, Gould J, Lerner J, Tamayo P, Mesirov JP. GenePattern 2.0. Nat Genet. 2006;38:500–1.
Article
CAS
PubMed
Google Scholar
Reich M, Liefeld J, Thorvaldsdottir H, Ocana M, Polk E, Jang D, et al. GenomeSpace: An environment for frictionless bioinformatics. Cancer Res. 2012;72:3966–3966.
Article
Google Scholar
GenePattern: A platform for reproducible bioinformatics [Internet]. 2016. Available from http://www.broadinstitute.org/cancer/software/genepattern]. Accessed 1 March 2016.
Wolstencroft K, Haines R, Fellows D, Williams A, Withers D, Owen S, et al. The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud. Nucleic Acids Res. 2013;41:557–61.
Article
Google Scholar
Rex DE, Ma JQ, Toga AW. The LONI pipeline processing environment. Neuroimage. 2003;19:1033–48.
Article
PubMed
Google Scholar
LONI Pipeline Processing Environment [Internet]. 2016. Available from http://www.loni.usc.edu/Software/Pipeline. Accessed 1 March 2016.
Vortex [Internet]. 2016. Available from https://github.com/websecurify/node-vortex. Accessed 1 March 2016.
Amazon Web Services [Internet]. 2016. Available from http://aws.amazon.com. Accessed 1 March 2016.
Google Cloud Platform [Internet]. 2016. Available from https://cloud.google.com/compute. Accessed 1 March 2016.
Microsoft Azure [Internet]. 2016. Available from https://azure.microsoft.com. Accessed 1 March 2016.
lmctfy - Let Me Contain That For You [Internet]. 2016. Available from https://github.com/google/lmctfy. Accessed 1 March 2016.
Warden [Internet]. 2016. Available from http://docs.cloudfoundry.org/concepts/architecture/warden.html. Accessed 1 March 2016.