<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Genet.</journal-id>
<journal-title>Frontiers in Genetics</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Genet.</abbrev-journal-title>
<issn pub-type="epub">1664-8021</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.3389/fgene.2013.00038</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Genetics</subject>
<subj-group>
<subject>Perspective Article</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>Where in the genome are we? A cautionary tale of database use in genomics research</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name><surname>Vaughan</surname> <given-names>Laura K.</given-names></name>
<xref ref-type="author-notes" rid="fn001"><sup>&#x0002A;</sup></xref>
</contrib>
<contrib contrib-type="author">
<name><surname>Srinivasasainagendra</surname> <given-names>Vinodh</given-names></name>
</contrib>
</contrib-group>
<aff><institution>Section on Statistical Genetics, Department of Biostatistics, University of Alabama at Birmingham</institution> <country>Birmingham, AL, USA</country></aff>
<author-notes>
<fn fn-type="edited-by"><p>Edited by: Danielle Posthuma, VU University Amsterdam, Netherlands</p></fn>
<fn fn-type="edited-by"><p>Reviewed by: Dirk-Jan De Koning, Swedish University of Agricultural Sciences, Sweden; Yulia A. Medvedeva, King Abdullah University of Science and Technology, Saudi Arabia</p></fn>
<fn fn-type="corresp" id="fn001"><p>&#x0002A;Correspondence: Laura K. Vaughan, Section on Statistical Genetics, Department of Biostatistics, University of Alabama at Birmingham, 1665 University Blvd., RPHB 317D, Birmingham, AL 35294, USA. e-mail: <email>lkvaughan&#x00040;uab.edu</email></p></fn>
<fn fn-type="other" id="fn002"><p>This article was submitted to Frontiers in Statistical Genetics and Methodology, a specialty of Frontiers in Genetics.</p></fn>
</author-notes>
<pub-date pub-type="epub">
<day>21</day>
<month>03</month>
<year>2013</year>
</pub-date>
<pub-date pub-type="collection">
<year>2013</year>
</pub-date>
<volume>4</volume>
<elocation-id>38</elocation-id>
<history>
<date date-type="received">
<day>14</day>
<month>08</month>
<year>2012</year>
</date>
<date date-type="accepted">
<day>04</day>
<month>03</month>
<year>2013</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#x000A9; 2013 Vaughan and Srinivasasainagendra.</copyright-statement>
<copyright-year>2013</copyright-year>
<license license-type="open-access" xlink:href="http://creativecommons.org/licenses/by/3.0/"><p>This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and subject to any copyright notices concerning any third-party graphics etc.</p>
</license>
</permissions>
<abstract><p>With the advent of high throughput data genomic technologies the volume of available data is now staggering. In addition databases that provide resources to annotate, translate, and connect biological data have grown exponentially in content and use. The availability of such data emphasizes the importance of bioinformatics and computational biology in genomics research and has led to the development of thousands of tools to integrate and utilize these resources. When utilizing such resources, the principles of reproducible research are often overlooked. In this manuscript we provide selected case studies illustrating issues that may arise while working with genes and genetic polymorphisms. These case studies illustrate potential sources of error which can be introduced if the practices of reproducible research are not employed and non-concurrent databases are used. We also show examples of a lack of transparency when these databases are concerned when using popular bioinformatics tools. These examples highlight that resources are constantly evolving, and in order to provide reproducible results, research should be aware of and connected to the correct release of the data, particularly when implementing computational tools.</p></abstract>
<kwd-group>
<kwd>reproducible research</kwd>
<kwd>database</kwd>
<kwd>bioinformatics</kwd>
<kwd>gene mapping</kwd>
</kwd-group>
<counts>
<fig-count count="0"/>
<table-count count="3"/>
<equation-count count="0"/>
<ref-count count="21"/>
<page-count count="6"/>
<word-count count="3705"/>
</counts>
</article-meta>
</front>
<body>
<sec sec-type="introduction" id="s1">
<title>Introduction</title>
<p>When conducting genetics research, whether from the perspective of a candidate gene study or genome wide association study (GWAS), researchers must be able to accurately identify and translate where molecular markers are located on the genome in reference to the coordinates of known genes. While this may seem straightforward, it can be quite complicated and is often overlooked (Hong et al., <xref ref-type="bibr" rid="B8">2009</xref>; Wang et al., <xref ref-type="bibr" rid="B21">2010</xref>). The mapping of markers to genes and subsequent data mining of information about these genes is further complicated by the ever increasing amounts of data and resulting evolution in databases, which in turn can lead to changes in genomic coordinates, annotations, and other information. Additionally, few studies (and methodologies) report the version of databases that are used in the bioinformatic workflow process. For example, the popular bioinformatics resource DAVID lists the download date of the various databases used for the Knowledgebase (the most recent of which is in 2009), but does not provide the version of those databases (Huang et al., <xref ref-type="bibr" rid="B11">2009a</xref>,<xref ref-type="bibr" rid="B12">b</xref>). This lack of reporting can make subsequent analysis and reproduction of others research difficult, if not impossible. In this manuscript we describe the key steps involved in the use of database resources for the mapping of markers to genes (and vice versa) in a typical candidate gene based study and highlight several ambiguities that can have potentially serious consequences in subsequent research.</p>
</sec>
<sec>
<title>Workflow case studies</title>
<p>The steps in identifying SNPs from a list of candidate genes can be described as (1) determining the candidate gene pool, (2) annotating, or retrieving information about those genes, (3) determining the location (and boundaries) of those genes, and (4) identifying molecular markers (e.g., single nucleotide polymorphisms, SNPs) within those boundaries. Similar steps are involved in identifying genes that are related to SNPs found to be significant from a GWAS.</p>
<p>In a collaborative research setting investigators will often provide a list of gene names that they are interested in to their bioinformatic collaborators who then retrieve information related to these genes for further analysis. The naming of these genes is the first point of ambiguity. Often, these names are common names or synonyms instead of the official names or gene symbols [see HUGO Gene Names Committee HGNC (Seal et al., <xref ref-type="bibr" rid="B19">2011</xref>)]. Due to the structure of most bioinformatic data sources, it can be difficult to correctly identify the gene that an investigator is interested in when the official name is not provided. An example of this is described in Table <xref ref-type="table" rid="T1">1</xref>. In this example the original list of <italic>TOSO</italic>, <italic>PIGR</italic>, <italic>FCAMR</italic>, <italic>ADRA1A</italic>, <italic>ADRA1B</italic>, and <italic>ADRA1D</italic> was provided by a collaborator. When searching the UCSC Genome Browser or Entrez Gene databases (accessed July 2011), we see that <italic>TOSO</italic> is not an HGNC official gene symbol, but is instead a synonym for the gene Fas apoptotic inhibitory molecule 3 (official symbol <italic>FAIM3</italic>, geneID 9214), and searching earlier versions of UCSC Genome Browser or into the Gene Accession Conversion tool in DAVID, <italic>TOSO</italic> would not produce any results.</p>
<table-wrap position="float" id="T1">
<label>Table 1</label>
<caption><p><bold>Inconsistency in gene &#x0201C;names&#x0201D; and locations</bold>.</p></caption>
<table frame="hsides" rules="groups">
<thead>
<tr>
<th align="left"><bold>GeneID</bold></th>
<th align="left"><bold>Symbol</bold></th>
<th align="left"><bold>Synonyms</bold></th>
<th align="left"><bold>Chr</bold></th>
<th align="left"><bold>Description</bold></th>
<th align="center" colspan="3"><bold>Genome build/base pair location</bold></th>
</tr>
<tr>
<th/>
<th/>
<th/>
<th/>
<th/>
<th align="left"><bold>Hg17</bold></th>
<th align="left"><bold>Hg18</bold></th>
<th align="left"><bold>Hg19</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">9214</td>
<td align="left"><italic>FAIM3</italic></td>
<td align="left"><bold>TOSO</bold></td>
<td align="left">1</td>
<td align="left">Fas apoptotic inhibitory molecule 3</td>
<td align="left">203466126-203483738</td>
<td align="left">205144354-205161966</td>
<td align="left">207076633-207095378</td>
</tr>
<tr>
<td align="left">5284</td>
<td align="left"><bold><italic>PIGR</italic></bold></td>
<td align="left">FLJ22667|MGC125361|MGC125362</td>
<td align="left">1</td>
<td align="left">Polymeric immunoglobulin receptor</td>
<td align="left">203490267-203508202</td>
<td align="left">205168495-205186430</td>
<td align="left">207101869-207119811</td>
</tr>
<tr>
<td align="left">83953</td>
<td align="left"><bold><italic>FCAMR</italic></bold></td>
<td align="left">FCA/MR|FKSG87</td>
<td align="left">1</td>
<td align="left">Fc receptor, IgA, IgM, high affinity</td>
<td align="left">Information Not available</td>
<td align="left">205198027-205210593</td>
<td align="left">207131404-207143970</td>
</tr>
<tr>
<td align="left">148</td>
<td align="left"><bold><italic>ADRA1A</italic></bold></td>
<td align="left">ADRA1C|ADRA1L1|ALPHA1AAR</td>
<td align="left">8</td>
<td align="left">Adrenergic, alpha-1A-, receptor</td>
<td align="left">26683139-26778839</td>
<td align="left">26683139-26778839</td>
<td align="left">26627222-26722922</td>
</tr>
<tr>
<td align="left">147</td>
<td align="left"><bold><italic>ADRA1B</italic></bold></td>
<td align="left">ADRA1|ALPHA1BAR</td>
<td align="left">5</td>
<td align="left">Adrenergic, alpha-1B-, receptor</td>
<td align="left">159276318-159332129</td>
<td align="left">159276318-159332595</td>
<td align="left">159343740-159400017</td>
</tr>
<tr>
<td align="left">146</td>
<td align="left"><italic>ADRA1D</italic></td>
<td align="left">ADRA1|<bold>ADRA1A</bold>|ADRA1R|ALPHA1|DAR|dJ779E11.2</td>
<td align="left">20</td>
<td align="left">Adrenergic, alpha-1D-, receptor</td>
<td align="left">4149816-4177659</td>
<td align="left">4149278-4177659</td>
<td align="left">4201278-4229659</td>
</tr>
</tbody>
</table>
<table-wrap-foot>
<p><italic>Bold genes indicate terms from original list from collaborator. Annotation information retrieved for the candidate gene list through Entrez gene&#x00027;s GeneInfo. GeneInfo data can be downloaded from Entrez gene&#x00027;s FTP location on September 10th 2010 (<ext-link ext-link-type="uri" xlink:href="ftp://ftp.ncbi.nih.gov/gene/DATA/">ftp://ftp.ncbi.nih.gov/gene/DATA/</ext-link>)</italic>.</p>
</table-wrap-foot>
</table-wrap>
<p>The second source of ambiguity, gene annotation, is also illustrated in Table <xref ref-type="table" rid="T1">1</xref>. In general, capturing gene level annotations (HGNC id, geneID, synonyms, chromosome, description, etc.), not only provides more information, but also allows investigators to perform quality checks. In the example discussed here, <italic>ADRA1A</italic> is listed as both an HGNC official gene symbol (adrenergic alpha-1B-receptor, gene ID 147) and as a synonym for <italic>ADRA1D</italic> (adrenergic alpha-1D-receptor, gene ID 146). Without scrutiny, it is difficult to tell which gene(s) the investigator is indeed interested. A survey of the most recent build of Entrez Gene (Hg 19) reveals that there are 43,037 unique gene symbols, 53,215 unique gene synonyms and 1122 instances where a term is both an official gene symbol and a synonym for at least one other gene, and 2632 terms that occur as synonyms for multiple genes, Although it may seem trivial in this example where there are only a few genes, in situations where there are dozens to hundreds of genes this manual verification of genes represents a significant investment of time and potential sources of error. It is also important to note that inconsistencies between databases can also introduce significant errors when translating gene IDs from one source to another. Even when using one of the several ID converters available (e.g., DAVID ID Converter or GeneCruiser), errors can be introduced when synonyms, HGNC symbols and other identifiers are inconsistently mapped or when the timelines for the database releases are not correctly matched or are out of date (discussed further below).</p>
<p>In the candidate gene framework, the gene coordinates can be identified from databases such as Entrez Gene or UCSC Genome Browser relatively easily. However the third ambiguity, determining the gene location, is illustrated in part by inconstancies in the use of gene symbol vs. gene synonym (Table <xref ref-type="table" rid="T1">1</xref>). For example, the genes <italic>ADRA1A</italic> and <italic>ADRA1D</italic> (discussed above) are located on separate chromosomes. Choosing the wrong gene will result in choosing a completely inappropriate location which will have obviously significant potential implications on downstream analysis. This ambiguity is perhaps more of a concern when taking the approach common for GWAS of identifying genes related to interesting SNPs. For a GWAS, usually both the SNP coordinates and genes that contain those SNPs are provided by the manufacturer of the genotyping platform. However, how these coordinates and genes are identified is often unclear, and these annotation files themselves are often additional source of errors. This is strikingly illustrated in Table <xref ref-type="table" rid="T2">2</xref>. In this instance a SNP (rs2844871) was identified as interesting based on an association study genotyped on the Affymetrix Genome Wide Human SNP Array 6.0. When following the bioinformatic workflow to identify the gene of interest, it was discovered that the SNP is mapped to different genes based on not only different databases, but also on different versions of those databases. A query of build 135 of dbSNP identified 1,226,430 SNPs that have multiple coordinates, 805,555 of which have more than one distinct chromosome assigned to the same rs ID (or 1.5% of the 54,212,080 SNPs). Additionally, 1,164,480 single base pair coordinates were found to be associated with multiple rsID&#x00027;s (with the maximum of 97 rsID&#x00027;s associated with the coordinates for one single nucleotide polymorphism). Searching for annotation information for rs2844871 in the UCSC Genome Browser, dbSNP, HapMap and Affymetrix databases not only provided different genomic locations based on which build that was accessed, but different (and multiple) chromosomes. Although in this case the multiple locations are likely due to a duplication event [a BLAST (<ext-link ext-link-type="uri" xlink:href="http://blast.ncbi.nlm.nih.gov/">http://blast.ncbi.nlm.nih.gov/</ext-link>) search of the 100 base pair sequence surrounding the SNP shows that regions of &#x0003E;90% identity occur on chromosomes 22, 14, 2, 4, and 21], it serves to dramatically illustrate errors that can be introduced with the use of different databases and lack of stringent quality controls.</p>
<table-wrap position="float" id="T2">
<label>Table 2</label>
<caption><p><bold>Discrepancies in the location of SNP rs2844871</bold>.</p></caption>
<table frame="hsides" rules="groups">
<thead>
<tr>
<th align="left"><bold>Database</bold></th>
<th align="left"><bold>Database version</bold></th>
<th align="left"><bold>Human genome build</bold></th>
<th align="left"><bold>Human reference chromosome/NCBI build</bold></th>
<th align="left"><bold>dbSNP build</bold></th>
<th align="left"><bold>Location (bp)</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td align="left" rowspan="3">Affymetrix 6.0</td>
<td align="left">NetAffx 30</td>
<td align="left">Hg18</td>
<td align="left">NCBI36</td>
<td align="left">Not specified</td>
<td align="left">chr22:14459243</td>
</tr>
<tr>
<td align="left">NetAffx 31</td>
<td align="left">Hg19</td>
<td align="left">GRCH 37</td>
<td align="left">131</td>
<td align="left">SNP is listed, but no position information given</td>
</tr>
<tr>
<td align="left">NetAffx 32</td>
<td align="left">Hg19</td>
<td align="left">GRCH 37</td>
<td align="left">132</td>
<td align="left">SNP is listed, but no position information given</td>
</tr>
<tr>
<td align="left" rowspan="9">UCSC genome browser</td>
<td align="left">July 2011</td>
<td align="left">Hg18</td>
<td align="left">NCBI36</td>
<td align="left">128</td>
<td align="left">chr22: 14459242</td>
</tr>
<tr>
<td/>
<td align="left">Hg18</td>
<td align="left">NCBI36</td>
<td align="left">130</td>
<td align="left">chr14: 19763716</td>
</tr>
<tr>
<td/>
<td/>
<td/>
<td/>
<td align="left">chr22: 16079242</td>
</tr>
<tr>
<td/>
<td align="left">Hg 19</td>
<td align="left">GRCH 37</td>
<td align="left">132</td>
<td align="left">chr14:19763467</td>
</tr>
<tr>
<td/>
<td/>
<td/>
<td/>
<td align="left">chr22:16078993</td>
</tr>
<tr>
<td/>
<td/>
<td/>
<td/>
<td align="left">chrUn_gl000244:34403</td>
</tr>
<tr>
<td/>
<td align="left">Hg 19</td>
<td align="left">GRCH 37</td>
<td align="left">135</td>
<td align="left">chr14:19763467</td>
</tr>
<tr>
<td/>
<td/>
<td/>
<td/>
<td align="left">chr22:16078993</td>
</tr>
<tr>
<td/>
<td/>
<td/>
<td/>
<td align="left">chrUn_gl000244:34403</td>
</tr>
<tr>
<td align="left" rowspan="5">dbSNP</td>
<td align="left">July 2011</td>
<td align="left">Hg 19</td>
<td align="left">NCBI 37.1</td>
<td align="left">132</td>
<td align="left">chr2: 125655701</td>
</tr>
<tr>
<td align="left">October 2012</td>
<td align="left">Hg19</td>
<td align="left">NCBI 37.3</td>
<td align="left">137</td>
<td align="left">chr14: 19763717</td>
</tr>
<tr>
<td/>
<td/>
<td/>
<td/>
<td align="left">chr22: 16079243</td>
</tr>
<tr>
<td/>
<td/>
<td/>
<td/>
<td align="left">NA</td>
</tr>
<tr>
<td/>
<td/>
<td/>
<td/>
<td align="left">chr2:124523528</td>
</tr>
<tr>
<td align="left">HapMap</td>
<td align="left">Release 27</td>
<td align="left">Hg18</td>
<td align="left">Not specified</td>
<td align="left">Not specified</td>
<td align="left">chr22: 14459243<xref ref-type="table-fn" rid="TN1"><sup>&#x0002A;</sup></xref></td>
</tr>
<tr>
<td align="left">HapMap</td>
<td align="left">Release 28</td>
<td align="left">Hg18</td>
<td align="left">NCBI36</td>
<td align="left">126</td>
<td align="left">chr22: 14459243</td>
</tr>
</tbody>
</table>
<table-wrap-foot>
<fn id="TN1"><label>&#x0002A;</label><p><italic>Genotyped on Affy 6.0 for Phase II samples, no dates or other information was given in HapMart</italic>.</p></fn>
<p><italic>Search results for SNP rs284487 from various sources. Search conducted on Jul 11th, 2011 and updated October 29th, 2012</italic>.</p>
</table-wrap-foot>
</table-wrap>
<p>Additionally, when mapping a marker to a gene, investigators are often not just interested in a SNP that lies directly with the gene boundary, but also genes that lie within a certain distance or are in linkage disequilibrium with a SNP of interest. The accurate identification of SNPs and related genes is dependent on both an accurate identification of gene boundaries and the synchronization of multiple databases, which often leads to the final source of ambiguity and is discussed further below.</p>
<p>The final source of ambiguity, variation between databases and across time, is intrinsic to every step of the workflow outlined above. Bioinformatic analysis is dependent on key database resources such as dbSNP, Entrez Gene, UCSC Genome Browser and Ensembl (Sherry, <xref ref-type="bibr" rid="B20">2001</xref>; Fujita et al., <xref ref-type="bibr" rid="B5">2011</xref>; Maglott et al., <xref ref-type="bibr" rid="B14">2011</xref>; Flicek et al., <xref ref-type="bibr" rid="B3">2012</xref>). These databases are in a state of dynamic flux, and are constantly being updated, sometimes resulting in significant changes (Data Changes that Occur Between Builds, <xref ref-type="bibr" rid="B2">2005</xref>; Fujita et al., <xref ref-type="bibr" rid="B5">2011</xref>). More often than not, investigators fail to provide the date and database version of each of the data sources that was used in the process of their analysis. Comparing the number of official gene symbols and synonyms for Hg18 and Hg19 highlights the differences in database builds. As discussed above, there are 43,037 and 53,215 unique gene symbols and synonyms respectively in Hg19, compared to 38,586 and 53,475 in Hg18, with 23,325 gene symbols overlapping between the two versions of the human genome builds. Tables <xref ref-type="table" rid="T1">1</xref> and <xref ref-type="table" rid="T2">2</xref> illustrate how the version of the database used can have an effect on the data that is retrieved. For example, (1) <italic>FACMR</italic> (gene ID 83953) was not included in the human genome build 17 (Hg17) and has gene boundary location that is shifted by almost two million bases from Hg18 to Hg19 and (2) when the coordinates from Hg17 are used to search UCSC Genome Browser using Hg19 the gene <italic>OPTC</italic> (gene ID 26254) is retrieved instead of <italic>FAIM3</italic>. Although these shifts in boundaries are a result of updates to the genome builds, one can see how using gene boundary coordinates from Hg19 for data that was originally built on Hg17, without first correcting for the change, can introduce errors. Tools such as the UCSC Genome Browser LiftOver Utility are available to convert genome coordinates between assemblies; however to correctly apply the tool, researchers must first be aware of the issue.</p>
<p>Furthermore, the timeline of these changes is not coordinated across databases. A search conducted in January 2010 in dbSNP would have been based on Hg18 instead of Hg19 which the NCBI released in early 2009 (build &#x00023;37) and the August 2010 HapMap data release (release &#x00023;28) for both Phase II and III data is based on NCBI build 36 and dbSNP release 126 (from 2006). As discussed above, the location of a gene may change with different builds of the human genome, sometimes significantly, and investigators should take the necessary steps to ensure that they are using coordinated builds of the different resources. When conducting research that is based on a genotyping chip, investigators should also carefully consider the version of databases used for bioinformatic analysis. If the corresponding changes in the coordinates of the markers on the genotyping chip are not also accounted for, SNPs could be mapped to incorrect genes, which can result in very costly mistakes (Karow, <xref ref-type="bibr" rid="B13">2010</xref>).</p>
</sec>
<sec sec-type="discussion" id="s2">
<title>Discussion</title>
<p>In recent years there has been a paradigm shift in the field of genetics. In the not too distant past, researchers were limited by their ability to acquire data. Now, with the availability of genome scale DNA and RNA platforms and recent introduction of affordable whole genome sequencing technologies, scientists are limited by their ability to effectively organize and analyze vast amounts of data. Part of this process is the accurate and consistent annotation of genomic information as part of the bioinformatics workflow. As describe above, changes in database versions and genome builds throughout the life of a study can have potentially significant impact.</p>
<p>In the case study discussed here we highlight several ambiguities that can be introduced in a candidate gene or SNP based study. When going across database versions using gene name, coordinates or rsID&#x00027;s up to the individual researcher. In the candidate gene based approach where the aim is to identify variants within a gene, one will typically use the coordinates of both genes and SNPs to identify SNPs for further study. As described above, one first verify they have the correct gene, and then must either stay within the same human genome version for each database used, or must correctly convert coordinates in order to avoid introducing errors. For the complementary approach based on identifying genes related to interesting SNPs, often the only data provided is the rsID for that SNP and no coordinates or genome build information is provided. Without this extra information errors can again be introduced when, as shown above, multiple positions, and therefore multiple genes are associated with a variant.</p>
<p>One way to prevent these errors is for investigators to involve bioinformaticians in all stages of a study, and for everyone involved to follow the principles of reproducible research. Reproducibility in research has been defined by the uniform Guidelines of the International Committee of Medical Journal Editors as the responsibility of authors to &#x0201C;identify the methods, apparatus and procedures in sufficient detail to allow other workers to reproduce the results.&#x0201D; Young scientists are taught to include in the methods and materials section of manuscripts the details which would be needed for successful repetition and extension of their work (Hothorn and Leisch, <xref ref-type="bibr" rid="B9">2011</xref>). Unfortunately, the same attention that is given to laboratory based experimental details and protocols have not been applied to the bioinformatics or computational components of many large genetic studies. This is beginning to change, especially in the domains of bioinformatics and biocomputing, where there has been growing interest in following the philosophy and best principles of reproducibility and repeatability in scientific research (Hothorn et al., <xref ref-type="bibr" rid="B10">2009</xref>; Mesirov, <xref ref-type="bibr" rid="B15">2010</xref>). As we move toward fully embracing the concepts of reproducible research, there is an increasing need for reproducible research modules in many of the software and tools where underlying computer code and data tend to change over time.</p>
<p>The continued growth in data volume has introduced a new set of issues that must be considered and addressed in genomics studies. The examples discussed above illustrate the importance of involving bioinformaticians in the entire process of a study. Researchers can avoid these pitfalls by implementing procedures that follow the principles of reproducible research. Similar to the use of a notebook in a wet lab, a wiki based notebook (our own group uses a Confluence powered wiki), employing a Reproducible Research Systems (RSS) approach or using tools such as myExperiment, GenePattern GRRD, Galaxy or Sweave, can be used to detail the workflow involved in the computational analysis of complex genomic data (Friedrich Leisch, <xref ref-type="bibr" rid="B4">2002</xref>; Reich et al., <xref ref-type="bibr" rid="B18">2006</xref>; Goble et al., <xref ref-type="bibr" rid="B6">2010</xref>; Goecks et al., <xref ref-type="bibr" rid="B7">2010</xref>; Hothorn and Leisch, <xref ref-type="bibr" rid="B9">2011</xref>). Accurate depiction of the research process will become even more important as journals follow the trend set by <italic>Biometrical Journal</italic>, <italic>Journal of Epidemiology</italic>, and <italic>Biostatistics</italic> which now suggest that authors go beyond the common practice of making data freely accessible, but also meet some standard of reproducibility (Peng et al., <xref ref-type="bibr" rid="B17">2006</xref>; Peng, <xref ref-type="bibr" rid="B16">2009</xref>; Mesirov, <xref ref-type="bibr" rid="B15">2010</xref>).</p>
</sec>
<sec sec-type="conclusion" id="s3">
<title>Conclusion</title>
<p>The importance of following the principles of reproducible research has been recently highlighted with several high profile examples (Hothorn et al., <xref ref-type="bibr" rid="B10">2009</xref>; Baggerly and Coombes, <xref ref-type="bibr" rid="B1">2011</xref>). Seemingly small mistakes can have significant downstream consequences in any data analysis that utilizes large amounts of data and multiple steps of analysis. As exemplified here, the simple mistake of not reporting, or using an incorrect version of a database can affect the interpretability and reproducibility of a study. To prevent these issues from having a greater impact, it is important for the research community as a whole to embrace the concepts of reproducible research and make a conscious effort toward moving toward that goal.</p>
<sec>
<title>Conflict of interest statement</title>
<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p></sec>
</sec>
</body>
<back>
<ack>
<p>The authors would like to thank the members of the Section on Statistical Genetics Programming Team for their helpful comments and suggestions. This work was supported by National Institutes of Health K01DK080188 and R03DK096071.</p>
</ack>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Baggerly</surname> <given-names>K.</given-names></name> <name><surname>Coombes</surname> <given-names>K.</given-names></name></person-group> (<year>2011</year>). <article-title>What information should be required to support clinical &#x0201C;Omics&#x0201D; publications?</article-title> <source>Clin. Chem</source>. <volume>57</volume>, <fpage>688</fpage>&#x02013;<lpage>690</lpage>. <pub-id pub-id-type="doi">10.1373/clinchem.2010.158618</pub-id><pub-id pub-id-type="pmid">21364027</pub-id></citation>
</ref>
<ref id="B2">
<citation citation-type="other"><person-group person-group-type="author"><collab>Data Changes that Occur Between Builds.</collab></person-group> (<year>2005</year>). Bookshelf ID NBK44467.</citation>
</ref>
<ref id="B3">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Flicek</surname> <given-names>P.</given-names></name> <name><surname>Ridwan Amode</surname> <given-names>M.</given-names></name> <name><surname>Barrell</surname> <given-names>D.</given-names></name> <name><surname>Beal</surname> <given-names>K.</given-names></name> <name><surname>Brent</surname> <given-names>S.</given-names></name> <name><surname>Carvalho-Silva</surname> <given-names>D.</given-names></name> <etal/></person-group>. (<year>2012</year>). <article-title>Ensembl 2012</article-title>. <source>Nucleic Acids Res</source>. <volume>40</volume>, <fpage>D84</fpage>&#x02013;<lpage>D90</lpage>. <pub-id pub-id-type="doi">10.1093/nar/gkr991</pub-id><pub-id pub-id-type="pmid">22086963</pub-id></citation>
</ref>
<ref id="B4">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Friedrich Leisch</surname> <given-names>J.</given-names></name></person-group> (<year>2002</year>). <article-title>Sweave: dynamic generation of statistical reports using literate data analysis</article-title>, in <source>Compstat 2002 - Proceed-ings in Computational Statistics</source>, eds <person-group person-group-type="editor"><name><surname>Wolfgang</surname> <given-names>H.</given-names></name> <name><surname>Bernd</surname> <given-names>R.</given-names></name></person-group> (<publisher-loc>Heidelberg</publisher-loc>: <publisher-name>Physica Verlag</publisher-name>), <fpage>575</fpage>&#x02013;<lpage>580</lpage>.</citation>
</ref>
<ref id="B5">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Fujita</surname> <given-names>P. A.</given-names></name> <name><surname>Rhead</surname> <given-names>B.</given-names></name> <name><surname>Zweig</surname> <given-names>A. S.</given-names></name> <name><surname>Hinrichs</surname> <given-names>A. S.</given-names></name> <name><surname>Karolchik</surname> <given-names>D.</given-names></name> <name><surname>Cline</surname> <given-names>M. S.</given-names></name> <etal/></person-group>. (<year>2011</year>). <article-title>The UCSC Genome Browser database: update 2011</article-title>. <source>Nucleic Acids Res</source>. <volume>39</volume>, <fpage>D876</fpage>&#x02013;<lpage>D882</lpage>. <pub-id pub-id-type="doi">10.1093/nar/gkq963</pub-id><pub-id pub-id-type="pmid">20959295</pub-id></citation>
</ref>
<ref id="B6">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Goble</surname> <given-names>C. A.</given-names></name> <name><surname>Bhagat</surname> <given-names>J.</given-names></name> <name><surname>Aleksejevs</surname> <given-names>S.</given-names></name> <name><surname>Cruickshank</surname> <given-names>D.</given-names></name> <name><surname>Michaelides</surname> <given-names>D.</given-names></name> <name><surname>Newman</surname> <given-names>D.</given-names></name> <etal/></person-group>. (<year>2010</year>). <article-title>myExperiment: a repository and social network for the sharing of bioinformatics workflows</article-title>. <source>Nucleic Acids Res</source>. <volume>38</volume>, <fpage>W677</fpage>&#x02013;<lpage>W682</lpage>. <pub-id pub-id-type="doi">10.1093/nar/gkq429</pub-id><pub-id pub-id-type="pmid">20501605</pub-id></citation>
</ref>
<ref id="B7">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Goecks</surname> <given-names>J.</given-names></name> <name><surname>Nekrutenko</surname> <given-names>A.</given-names></name> <name><surname>Taylor</surname> <given-names>J.</given-names></name></person-group> (<year>2010</year>). <article-title>Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences</article-title>. <source>Genome Biol</source>. <volume>11</volume>:<fpage>R86</fpage>. <pub-id pub-id-type="doi">10.1186/gb-2010-11-8-r86</pub-id><pub-id pub-id-type="pmid">20738864</pub-id></citation>
</ref>
<ref id="B8">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hong</surname> <given-names>M.-G.</given-names></name> <name><surname>Pawitan</surname> <given-names>Y.</given-names></name> <name><surname>Magnusson</surname> <given-names>P. K. E.</given-names></name> <name><surname>Prince</surname> <given-names>J.</given-names></name></person-group> (<year>2009</year>). <article-title>Strategies and issues in the detection of pathway enrichment in genome-wide association studies</article-title>. <source>Hum. Genet</source>. <volume>126</volume>, <fpage>289</fpage>&#x02013;<lpage>301</lpage>. <pub-id pub-id-type="doi">10.1007/s00439-009-0676-z</pub-id><pub-id pub-id-type="pmid">19408013</pub-id></citation>
</ref>
<ref id="B10">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hothorn</surname> <given-names>T.</given-names></name> <name><surname>Held</surname> <given-names>L.</given-names></name> <name><surname>Friede</surname> <given-names>T.</given-names></name></person-group> (<year>2009</year>). <article-title>Biometrical journal and reproducible research</article-title>. <source>Biom. J</source>. <volume>51</volume>, <fpage>553</fpage>&#x02013;<lpage>555</lpage>. <pub-id pub-id-type="doi">10.1002/bimj.200900154</pub-id><pub-id pub-id-type="pmid">19688756</pub-id></citation>
</ref>
<ref id="B9">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hothorn</surname> <given-names>T.</given-names></name> <name><surname>Leisch</surname> <given-names>F.</given-names></name></person-group> (<year>2011</year>). <article-title>Case studies in reproducibility</article-title>. <source>Brief. Bioinformatics</source> <volume>12</volume>, <fpage>288</fpage>&#x02013;<lpage>300</lpage>. <pub-id pub-id-type="doi">10.1093/bib/bbq084</pub-id><pub-id pub-id-type="pmid">21278369</pub-id></citation>
</ref>
<ref id="B11">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Huang</surname> <given-names>D. W.</given-names></name> <name><surname>Sherman</surname> <given-names>B. T.</given-names></name> <name><surname>Lempicki</surname> <given-names>R. A.</given-names></name></person-group> (<year>2009a</year>). <article-title>Bioinformatics enrichment tools: paths toward the comprehensive functional analysis of large gene lists</article-title>. <source>Nucleic Acids Res</source>. <volume>37</volume>, <fpage>1</fpage>&#x02013;<lpage>13</lpage>. <pub-id pub-id-type="doi">10.1093/nar/gkn923</pub-id><pub-id pub-id-type="pmid">19033363</pub-id></citation>
</ref>
<ref id="B12">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Huang</surname> <given-names>D. W.</given-names></name> <name><surname>Sherman</surname> <given-names>B. T.</given-names></name> <name><surname>Lempicki</surname> <given-names>R. A.</given-names></name></person-group> (<year>2009b</year>). <article-title>Systematic and integrative analysis of large gene lists using DAVID bioinformatics resources</article-title>. <source>Nat. Protoc</source>. <volume>4</volume>, <fpage>44</fpage>&#x02013;<lpage>57</lpage>. <pub-id pub-id-type="doi">10.1038/nprot.2008.211</pub-id><pub-id pub-id-type="pmid">19131956</pub-id></citation>
</ref>
<ref id="B13">
<citation citation-type="web"><person-group person-group-type="author"><name><surname>Karow</surname> <given-names>J.</given-names></name></person-group> (<year>2010</year>). <source>NimbleGen Array Capture Outperforms Two Target-Enrichment Methods in ABRF Research Group Comparison In Sequence Sequencing GenomeWeb</source>. GenomeWeb In Sequence. Available online at: <ext-link ext-link-type="uri" xlink:href="http://www.genomeweb.com/sequencing/nimblegen-array-capture-outperforms-two-target-enrichment-methods-abrf-research-">http://www.genomeweb.com/sequencing/nimblegen-array-capture-outperforms-two-target-enrichment-methods-abrf-research-</ext-link></citation>
</ref>
<ref id="B14">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Maglott</surname> <given-names>D.</given-names></name> <name><surname>Ostell</surname> <given-names>J.</given-names></name> <name><surname>Pruitt</surname> <given-names>K. D.</given-names></name> <name><surname>Tatusova</surname> <given-names>T.</given-names></name></person-group> (<year>2011</year>). <article-title>Entrez Gene: gene-centered information at NCBI</article-title>. <source>Nucleic Acids Res</source>. <volume>39</volume>, <fpage>D52</fpage>&#x02013;<lpage>D57</lpage>. <pub-id pub-id-type="doi">10.1093/nar/gkq1237</pub-id><pub-id pub-id-type="pmid">21115458</pub-id></citation>
</ref>
<ref id="B15">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Mesirov</surname> <given-names>J. P.</given-names></name></person-group> (<year>2010</year>). <article-title>Accessible reproducible research</article-title>. <source>Science</source> <volume>327</volume>, <fpage>415</fpage>&#x02013;<lpage>416</lpage>. <pub-id pub-id-type="doi">10.1126/science.1179653</pub-id><pub-id pub-id-type="pmid">20093459</pub-id></citation>
</ref>
<ref id="B16">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Peng</surname> <given-names>R. D.</given-names></name></person-group> (<year>2009</year>). <article-title>Reproducible research and Biostatistics</article-title>. <source>Bio-statistics</source> <volume>10</volume>, <fpage>405</fpage>&#x02013;<lpage>408</lpage>. <pub-id pub-id-type="doi">10.1093/biostatistics/kxp014</pub-id><pub-id pub-id-type="pmid">19535325</pub-id></citation>
</ref>
<ref id="B17">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Peng</surname> <given-names>R. D.</given-names></name> <name><surname>Dominici</surname> <given-names>F.</given-names></name> <name><surname>Zeger</surname> <given-names>S. L.</given-names></name></person-group> (<year>2006</year>). <article-title>Reproducible epidemiologic research</article-title>. <source>Am. J. Epidemiol</source>. <volume>163</volume>, <fpage>783</fpage>&#x02013;<lpage>789</lpage>. <pub-id pub-id-type="doi">10.1093/aje/kwj093</pub-id><pub-id pub-id-type="pmid">16510544</pub-id></citation>
</ref>
<ref id="B18">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Reich</surname> <given-names>M.</given-names></name> <name><surname>Liefeld</surname> <given-names>T.</given-names></name> <name><surname>Gould</surname> <given-names>J.</given-names></name> <name><surname>Lerner</surname> <given-names>J.</given-names></name> <name><surname>Tamayo</surname> <given-names>P.</given-names></name> <name><surname>Mesirov</surname> <given-names>J. P.</given-names></name></person-group> (<year>2006</year>). <article-title>GenePattern 2.0</article-title>. <source>Nat. Genet</source>. <volume>38</volume>, <fpage>500</fpage>&#x02013;<lpage>501</lpage>. <pub-id pub-id-type="doi">10.1038/ng0506-500</pub-id><pub-id pub-id-type="pmid">16642009</pub-id></citation>
</ref>
<ref id="B19">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Seal</surname> <given-names>R. L.</given-names></name> <name><surname>Gordon</surname> <given-names>S. M.</given-names></name> <name><surname>Lush</surname> <given-names>M. J.</given-names></name> <name><surname>Wright</surname> <given-names>M. W.</given-names></name> <name><surname>Bruford</surname> <given-names>E. A.</given-names></name></person-group> (<year>2011</year>). <article-title>Genenames.org: the HGNC resources in 2011</article-title>. <source>Nucleic Acids Res</source>. <volume>39</volume>, <fpage>D514</fpage>&#x02013;<lpage>D519</lpage>. <pub-id pub-id-type="doi">10.1093/nar/gkq892</pub-id><pub-id pub-id-type="pmid">20929869</pub-id></citation>
</ref>
<ref id="B20">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Sherry</surname> <given-names>S. T.</given-names></name></person-group> (<year>2001</year>). <article-title>dbSNP: the NCBI database of genetic variation</article-title>. <source>Nucleic Acids Res</source>. <volume>29</volume>, <fpage>308</fpage>&#x02013;<lpage>311</lpage>. <pub-id pub-id-type="pmid">11125122</pub-id></citation>
</ref>
<ref id="B21">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wang</surname> <given-names>K.</given-names></name> <name><surname>Li</surname> <given-names>M.</given-names></name> <name><surname>Hakonarson</surname> <given-names>H.</given-names></name></person-group> (<year>2010</year>). <article-title>Analysing biological pathways in genome-wide association studies</article-title>. <source>Nat. Rev. Genet</source>. <volume>11</volume>, <fpage>843</fpage>&#x02013;<lpage>854</lpage>. <pub-id pub-id-type="doi">10.1038/nrg2884</pub-id><pub-id pub-id-type="pmid">21085203</pub-id></citation>
</ref>
</ref-list>
<sec>
<title>URLs</title>
<p>
<table-wrap position="float">
<table frame="hsides" rules="groups">
<thead>
<tr>
<th align="left"><bold>Resource</bold></th>
<th align="left"><bold>URL</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">Entrez Gene</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://www.ncbi.nlm.nih.gov/gene">www.ncbi.nlm.nih.gov/gene</ext-link></td>
</tr>
<tr>
<td align="left">UCSC Genome Browser</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://genome.ucsc.edu">http://genome.ucsc.edu</ext-link></td>
</tr>
<tr>
<td align="left">Affymetrix NetAffx Analysis Center</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://www.affymetrix.com/analysis/index.affx">www.affymetrix.com/analysis/index.affx</ext-link></td>
</tr>
<tr>
<td align="left">dbSNP</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://www.ncbi.nlm.nih.gov/projects/SNP/">www.ncbi.nlm.nih.gov/projects/SNP/</ext-link></td>
</tr>
<tr>
<td align="left">UCSC LiftOver</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://genome.ucsc.edu/cgi-bin/hgLiftOver">http://genome.ucsc.edu/cgi-bin/hgLiftOver</ext-link></td>
</tr>
<tr>
<td align="left">My Experiment</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://www.myexperiment.org/">www.myexperiment.org/</ext-link></td>
</tr>
<tr>
<td align="left">GenePattern GRRD</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://www.broadinstitute.org/cancer/software/genepattern/grrd/">www.broadinstitute.org/cancer/software/genepattern/grrd/</ext-link></td>
</tr>
<tr>
<td align="left">Sweave</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://www.stat.uni-muenchen.de/&#x0007E;leisch/Sweave/">www.stat.uni-muenchen.de/&#x0007E;leisch/Sweave/</ext-link></td>
</tr>
<tr>
<td align="left">HUGO</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://www.genenames.org/aboutHGNC.html">www.genenames.org/aboutHGNC.html</ext-link></td>
</tr>
<tr>
<td align="left">DAVID</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://david.abcc.ncifcrf.gov/">http://david.abcc.ncifcrf.gov/</ext-link></td>
</tr>
<tr>
<td align="left">GeneCruiser</td>
<td align="left"><ext-link ext-link-type="uri" xlink:href="http://genecruiser.broadinstitute.org/genecruiser3/">http://genecruiser.broadinstitute.org/genecruiser3/</ext-link></td>
</tr>
</tbody>
</table>
</table-wrap></p>
</sec>
</back>
</article>