<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article article-type="research-article" dtd-version="2.3" xml:lang="EN" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Cell Dev. Biol.</journal-id>
<journal-title>Frontiers in Cell and Developmental Biology</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Cell Dev. Biol.</abbrev-journal-title>
<issn pub-type="epub">2296-634X</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="publisher-id">1201673</article-id>
<article-id pub-id-type="doi">10.3389/fcell.2023.1201673</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Cell and Developmental Biology</subject>
<subj-group>
<subject>Original Research</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>Reproducibility and FAIR principles: the case of a segment polarity network model</article-title>
<alt-title alt-title-type="left-running-head">Mendes</alt-title>
<alt-title alt-title-type="right-running-head">
<ext-link ext-link-type="uri" xlink:href="https://doi.org/10.3389/fcell.2023.1201673">10.3389/fcell.2023.1201673</ext-link>
</alt-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name>
<surname>Mendes</surname>
<given-names>Pedro</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
<xref ref-type="corresp" rid="c001">&#x2a;</xref>
<uri xlink:href="https://loop.frontiersin.org/people/2273628/overview"/>
</contrib>
</contrib-group>
<aff id="aff1">
<sup>1</sup>
<institution>Center for Cell Analysis and Modeling</institution>, <institution>University of Connecticut School of Medicine</institution>, <addr-line>Farmington</addr-line>, <addr-line>CT</addr-line>, <country>United States</country>
</aff>
<aff id="aff2">
<sup>2</sup>
<institution>Department of Cell Biology</institution>, <institution>University of Connecticut School of Medicine</institution>, <addr-line>Farmington</addr-line>, <addr-line>CT</addr-line>, <country>United States</country>
</aff>
<author-notes>
<fn fn-type="edited-by">
<p>
<bold>Edited by:</bold> <ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/1473117/overview">Susan Mertins</ext-link>, Leidos Biomedical Research, Inc., United States</p>
</fn>
<fn fn-type="edited-by">
<p>
<bold>Reviewed by:</bold> <ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/125317/overview">Andreas Dr&#xe4;ger</ext-link>, University of T&#xfc;bingen, Germany</p>
<p>
<ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/34505/overview">David Phillip Nickerson</ext-link>, The University of Auckland, New Zealand</p>
</fn>
<corresp id="c001">&#x2a;Correspondence: Pedro Mendes, <email>pmendes@uchc.edu</email>
</corresp>
</author-notes>
<pub-date pub-type="epub">
<day>06</day>
<month>06</month>
<year>2023</year>
</pub-date>
<pub-date pub-type="collection">
<year>2023</year>
</pub-date>
<volume>11</volume>
<elocation-id>1201673</elocation-id>
<history>
<date date-type="received">
<day>06</day>
<month>04</month>
<year>2023</year>
</date>
<date date-type="accepted">
<day>30</day>
<month>05</month>
<year>2023</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#xa9; 2023 Mendes.</copyright-statement>
<copyright-year>2023</copyright-year>
<copyright-holder>Mendes</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/">
<p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.</p>
</license>
</permissions>
<abstract>
<p>The issue of reproducibility of computational models and the related FAIR principles (findable, accessible, interoperable, and reusable) are examined in a specific test case. I analyze a computational model of the segment polarity network in <italic>Drosophila</italic> embryos published in 2000. Despite the high number of citations to this publication, 23&#xa0;years later the model is barely accessible, and consequently not interoperable. Following the text of the original publication allowed successfully encoding the model for the open source software COPASI. Subsequently saving the model in the SBML format allowed it to be <italic>reused</italic> in other open source software packages. Submission of this SBML encoding of the model to the BioModels database enables its <italic>findability</italic> and <italic>accessibility</italic>. This demonstrates how the FAIR principles can be successfully enabled by using open source software, widely adopted standards, and public repositories, facilitating reproducibility and reuse of computational cell biology models that will outlive the specific software used.</p>
</abstract>
<kwd-group>
<kwd>reproducibility</kwd>
<kwd>model reuse</kwd>
<kwd>computational modeling</kwd>
<kwd>ODE modeling</kwd>
<kwd>systems biology</kwd>
<kwd>SBML</kwd>
<kwd>segment polarity network</kwd>
</kwd-group>
<contract-sponsor id="cn001">National Institutes of Health<named-content content-type="fundref-id">10.13039/100000002</named-content>
</contract-sponsor>
<custom-meta-wrap>
<custom-meta>
<meta-name>section-at-acceptance</meta-name>
<meta-value>Morphogenesis and Patterning</meta-value>
</custom-meta>
</custom-meta-wrap>
</article-meta>
</front>
<body>
<sec id="s1">
<title>1 Introduction</title>
<p>Embryonic development is characterized by frequent dynamic changes in gene expression that lead to the formation of different tissues and organs. Several patterns form during development caused by the interaction of biochemical reactions and diffusion, which was first suggested by the pioneering work of <xref ref-type="bibr" rid="B68">Turing (1952)</xref>. Since then computational models have been used to attempt to rationalize the formation of various patterns that are crucial in development. One of these is the formation of segments in the body of insects, studied intensively in the <italic>Drosophila</italic> embryo (<xref ref-type="bibr" rid="B31">Jaeger, 2009</xref>). Insects, and other arthropods, have segmented bodies with each segment being a unit bearing a pair of appendages (such as legs). The formation of these segments during embryogenesis originates from periodic patterns of gene expression that occur in various stages. First, genes maternally expressed determine the broad regions of the body (anterior, posterior and terminal), followed by the expression of &#x201c;gap genes&#x201d; and then by &#x201c;pair-rule genes&#x201d;. Mutations on the gap genes delete contiguous segments, while mutations on pair-rule genes affect every other segment. These stages happen when cell boundaries (membranes) have not yet formed and thus multiple nuclei share a common cytoplasm (a syncytium). After separation of nuclei into separate cells, by formation of plasma membranes, the &#x201c;segment polarity genes&#x201d; are expressed at different levels in each cell forming a pattern that will ensure the persistent polarity of the segments throughout the rest of embryonic development.</p>
<p>The year 2000 is often considered to mark the beginning of the modern systems biology era. This derives from several events that happened in that year, such as the founding of the Institute for Systems Biology, the first International Conference for Systems Biology, and the publication of various articles that are now considered &#x201c;classics&#x201d;. One of those publications, by <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref>, describes a model of the <italic>Drosophila</italic> segment polarity network, where a gene regulatory network operates in each one of a series of neighboring cells, with their protein products also interacting across cells (hereafter, the &#x201c;SPN model&#x201d;). The main conclusion, from a set of computer simulations sampling the SPN model&#x2019;s parameter space, was that it is &#x201c;remarkably&#x201d; robust as many more random combinations of parameter values than expected give rise to the characteristic spatial gene expression pattern required for segmentation. The inference that the network structure, rather than a narrow set of parameter values, is determinant to the phenotype has been cited as a general property of systems by more than one thousand publications to this date. Another conclusion derived from those results is that the phenotype is therefore robust against perturbation of the parameters&#x2014;and this has also frequently been assumed to be a general property of biological systems.</p>
<p>An important activity in computational systems biology is the deposition of models in public repositories using standard formats like SBML (<xref ref-type="bibr" rid="B24">Hucka et al., 2003</xref>) or CellML (<xref ref-type="bibr" rid="B19">Hedley et al., 2001</xref>). This allows any scientist to easily find and access those models and use them to run simulations or derive new ones using several compatible software applications. Through the last couple decades most classic models have been added to model repositories.</p>
<p>Surprisingly, being described in such a highly cited publication, the SPN model is not available in any of the four major systems biology model repositories: BioModels database (<xref ref-type="bibr" rid="B38">Le Nov&#xe8;re et al., 2006</xref>; <xref ref-type="bibr" rid="B42">Malik-Sheriff et al., 2020</xref>), the Physiome model repository (<xref ref-type="bibr" rid="B79">Yu et al., 2011</xref>), JWS online (<xref ref-type="bibr" rid="B51">Olivier and Snoep, 2004</xref>), or the database of Virtual Cell published models (<xref ref-type="bibr" rid="B49">Moraru et al., 2008</xref>). To make matters worse, the software Ingeneue (<xref ref-type="bibr" rid="B45">Meir et al., 2002</xref>; <xref ref-type="bibr" rid="B35">Kim, 2009</xref>), used to create this model, is no longer available, not even through the Wayback Machine (<xref ref-type="bibr" rid="B29">Internet Archive, 1996</xref>). Web searches revealed an SBML implementation (<xref ref-type="bibr" rid="B61">Sethna, 2008</xref>) which encodes the mathematics of the model in a 4 &#xd7; 6 grid of cells, but not the biochemical network.</p>
<p>Given the importance that the results obtained from the SPN model have had in systems biology it seems important that they be available in a well-supported software simulator and distributed in a standard format by one of the model repositories. I therefore set to encode this model with COPASI (<xref ref-type="bibr" rid="B21">Hoops et al., 2006</xref>; <xref ref-type="bibr" rid="B6">Bergmann et al., 2017</xref>) and to make sure that it was correctly implemented, use it to reproduce the simulation results of <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref>, at least partially. It has been noted that reproducing results from computational studies in general (<xref ref-type="bibr" rid="B47">Mesirov, 2010</xref>; <xref ref-type="bibr" rid="B52">Peng, 2011</xref>; <xref ref-type="bibr" rid="B63">Stodden et al., 2016</xref>), and also computatational systems biology (<xref ref-type="bibr" rid="B71">Waltemath and Wolkenhauer, 2016</xref>; <xref ref-type="bibr" rid="B46">Mendes, 2018</xref>; <xref ref-type="bibr" rid="B66">Tiwari et al., 2021</xref>), is as hard as with laboratory experiments. This has also been the case here and the obstacles encountered are described below.</p>
<p>Through a careful examination of the publications that cite <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref>, I was able to identify 15 cases where the SPN model was reused (<xref ref-type="table" rid="T1">Table 1</xref>). Only two actually reproduced their results (<xref ref-type="bibr" rid="B28">Ingolia, 2004</xref>; <xref ref-type="bibr" rid="B41">Ma et al., 2006</xref>), and another expanded the analysis to diploidy (<xref ref-type="bibr" rid="B34">Kim and Fernandes, 2009</xref>). Several authors used the SPN model to illustrate other issues, such as robustness (<xref ref-type="bibr" rid="B9">Chaves et al., 2009</xref>; <xref ref-type="bibr" rid="B13">Dayarian et al., 2009</xref>; <xref ref-type="bibr" rid="B2">Albert et al., 2011</xref>), &#x201c;sloppyness&#x201d; (<xref ref-type="bibr" rid="B17">Gutenkunst et al., 2007</xref>; <xref ref-type="bibr" rid="B12">Daniels et al., 2008</xref>), or new methodologies (<xref ref-type="bibr" rid="B65">Tegner et al., 2003</xref>; <xref ref-type="bibr" rid="B80">Za&#xf1;udo et al., 2017</xref>; <xref ref-type="bibr" rid="B58">Rozum and Albert, 2018</xref>; <xref ref-type="bibr" rid="B44">Marazzi et al., 2022</xref>). Several software applications were used, such as the original Ingeneue (<xref ref-type="bibr" rid="B45">Meir et al., 2002</xref>; <xref ref-type="bibr" rid="B35">Kim, 2009</xref>) and Little b (<xref ref-type="bibr" rid="B43">Mallavarapu et al., 2009</xref>), both now unavailable, and bespoke C programs that were never distributed (<xref ref-type="bibr" rid="B28">Ingolia, 2004</xref>; <xref ref-type="bibr" rid="B41">Ma et al., 2006</xref>)&#x2014;all those results are now difficult to reproduce. Only the Sethna group publications (<xref ref-type="bibr" rid="B17">Gutenkunst et al., 2007</xref>; <xref ref-type="bibr" rid="B12">Daniels et al., 2008</xref>) resulted in a version of the model that is runnable in several simulators; <xref ref-type="bibr" rid="B44">Marazzi et al. (2022)</xref> re-used that model and also provided a COPASI version in their GitHub repository.</p>
<table-wrap id="T1" position="float">
<label>TABLE 1</label>
<caption>
<p>Publications that reproduced or re-used the <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref> SPN model.</p>
</caption>
<table>
<thead valign="top">
<tr>
<th align="left">References</th>
<th align="left">Description</th>
<th align="left">Approach</th>
<th align="left">Software</th>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">
<xref ref-type="bibr" rid="B69">von Dassow and Odell (2002)</xref>
</td>
<td align="left">re-used original SPN model</td>
<td align="left">ODE</td>
<td align="left">Ingeneue<xref ref-type="table-fn" rid="Tfn1">
<sup>a</sup>
</xref>
</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B1">Albert and Othmer (2003)</xref>
</td>
<td align="left">Boolean network similar but not equal to original SPN</td>
<td align="left">Boolean</td>
<td align="left">unknown</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B65">Tegner et al. (2003)</xref>
</td>
<td align="left">Single-cell version of original SPN, without diffusive transitions</td>
<td align="left">ODE</td>
<td align="left">unknown</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B28">Ingolia (2004)</xref>
</td>
<td align="left">re-coded original SPN model</td>
<td align="left">ODE</td>
<td align="left">C program<xref ref-type="table-fn" rid="Tfn2">
<sup>b</sup>
</xref>
</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B41">Ma et al. (2006)</xref>
</td>
<td align="left">re-coded original SPN model</td>
<td align="left">ODE</td>
<td align="left">C program<xref ref-type="table-fn" rid="Tfn2">
<sup>b</sup>
</xref>
</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B17">Gutenkunst et al. (2007)</xref>
</td>
<td align="left">re-coded original SPN model</td>
<td align="left">ODE</td>
<td align="left">SloppyCell<xref ref-type="table-fn" rid="Tfn3">
<sup>c</sup>
</xref>
</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B12">Daniels et al. (2008)</xref>
</td>
<td align="left">re-used code from <xref ref-type="bibr" rid="B17">Gutenkunst et al. (2007)</xref>
<sup>d</sup>
</td>
<td align="left">ODE</td>
<td align="left">SloppyCell<xref ref-type="table-fn" rid="Tfn3">
<sup>c</sup>
</xref>
</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B9">Chaves et al. (2009)</xref>
</td>
<td align="left">simplification of SPN model ODEs<sup>e</sup>
</td>
<td align="left">algebraic</td>
<td align="left">N/A</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B13">Dayarian et al. (2009)</xref>
</td>
<td align="left">simplification of SPN model ODEs<sup>e</sup>
</td>
<td align="left">algebraic</td>
<td align="left">unknown</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B34">Kim and Fernandes (2009)</xref>
</td>
<td align="left">re-coded diploid version of SPN model</td>
<td align="left">ODE</td>
<td align="left">Mathematica<xref ref-type="table-fn" rid="Tfn2">
<sup>b</sup>
</xref>
</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B43">Mallavarapu et al. (2009)</xref>
</td>
<td align="left">re-coded original SPN model</td>
<td align="left">ODE</td>
<td align="left">Little b<xref ref-type="table-fn" rid="Tfn1">
<sup>a</sup>
</xref>
</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B2">Albert et al. (2011)</xref>
</td>
<td align="left">re-coded original SPN model</td>
<td align="left">algebraic</td>
<td align="left">MATLAB<xref ref-type="table-fn" rid="Tfn2">
<sup>b</sup>
</xref>
</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B80">Za&#xf1;udo et al. (2017)</xref>
</td>
<td align="left">re-used original SPN model</td>
<td align="left">ODE</td>
<td align="left">Python<xref ref-type="table-fn" rid="Tfn2">
<sup>b</sup>
</xref>
</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B58">Rozum and Albert (2018)</xref>
</td>
<td align="left">re-coded single-cell version of original SPN model</td>
<td align="left">algebraic</td>
<td align="left">Python</td>
</tr>
<tr>
<td align="left">
<xref ref-type="bibr" rid="B44">Marazzi et al. (2022)</xref>
</td>
<td align="left">re-used SBML model from <xref ref-type="bibr" rid="B12">Daniels et al. (2008)</xref>
<sup>d</sup>
</td>
<td align="left">ODE</td>
<td align="left">COPASI</td>
</tr>
</tbody>
</table>
<table-wrap-foot>
<fn id="Tfn1">
<label>
<sup>a</sup>
</label>
<p>Software no longer available.</p>
</fn>
<fn id="Tfn2">
<label>
<sup>b</sup>
</label>
<p>Code not publicly available.</p>
</fn>
<fn id="Tfn3">
<label>
<sup>c</sup>
</label>
<p>Software available from <ext-link ext-link-type="uri" xlink:href="https://sloppycell.sourceforge.net/">https://sloppycell.sourceforge.net/</ext-link>.</p>
</fn>
<fn id="Tfn4">
<label>
<sup>d</sup>
</label>
<p>SBML version available from <ext-link ext-link-type="uri" xlink:href="https://sethna.lassp.cornell.edu/Sloppy/vonDassow/model.html">https://sethna.lassp.cornell.edu/Sloppy/vonDassow/model.html</ext-link>.</p>
</fn>
<fn id="Tfn5">
<label>
<sup>e</sup>
</label>
<p>Used a square grid of cells.</p>
</fn>
</table-wrap-foot>
</table-wrap>
<p>This exercise identifies issues that hinder reproducibility and reuse of biomodels, and illustrates how they can be overcome with modern open science practices addressing the FAIR principles (<xref ref-type="bibr" rid="B76">Wilkinson et al., 2016</xref>). Reproducing it required a certain level of &#x201c;archeological&#x201d; craft to find missing parts. I hope that this also serves as a demonstration of procedures that make models usable beyond the lifetime of the software that created them. Of course, the SPN model was an important and early application of computational systems biology to developmental biology, and reproducing its results is also not irrelevant.</p>
</sec>
<sec sec-type="methods" id="s2">
<title>2 Methods</title>
<sec id="s2-1">
<title>2.1 Software</title>
<p>Model simulations and parameter sampling were carried out with COPASI version 4.39 (<xref ref-type="bibr" rid="B21">Hoops et al., 2006</xref>; <xref ref-type="bibr" rid="B6">Bergmann et al., 2017</xref>, RRID:<ext-link ext-link-type="uri" xlink:href="rridsoftware:SCR_014260">SCR_014260</ext-link>), Virtual Cell version 7.5.0 (<xref ref-type="bibr" rid="B59">Schaff et al., 1997</xref>; <xref ref-type="bibr" rid="B49">Moraru et al., 2008</xref>, RRID:<ext-link ext-link-type="uri" xlink:href="rridsoftware:SCR_007421">SCR_007421</ext-link>), Tellurium version 2.2.7 (<xref ref-type="bibr" rid="B10">Choi et al., 2018</xref>) that uses libRoadRunner version 2.3.2 (<xref ref-type="bibr" rid="B75">Welsh et al., 2023</xref>, RRID:<ext-link ext-link-type="uri" xlink:href="rridsoftware:SCR_014763">SCR_014763</ext-link>), and AMICI version 0.11.25 (<xref ref-type="bibr" rid="B14">Fr&#xf6;hlich et al., 2021</xref>), which was accessed through runBioSimulations (<xref ref-type="bibr" rid="B62">Shaikh et al., 2021</xref>, RRID:<ext-link ext-link-type="uri" xlink:href="rridsoftware:SCR_019110">SCR_019110</ext-link>). The model file was constructed with python scripts using the BasiCO package that interfaces with COPASI (<xref ref-type="bibr" rid="B7">Bergmann, 2023</xref>). Simulations were run at the local high-performance computing cluster using the Cloud-COPASI web interface (<xref ref-type="bibr" rid="B33">Kent et al., 2012</xref>). Results were visualized with COPASI, with Gnuplot version 5.4.3 (<xref ref-type="bibr" rid="B77">Williams and Kelley, 2022</xref>, RRID:<ext-link ext-link-type="uri" xlink:href="rridsoftware:SCR_008619">SCR_008619</ext-link>), or with the Python libraries Seaborn (<xref ref-type="bibr" rid="B74">Waskom, 2021</xref>) (RRID:<ext-link ext-link-type="uri" xlink:href="rridsoftware:SCR_018132">SCR_018132</ext-link>) and Matplotlib (<xref ref-type="bibr" rid="B26">Hunter, 2007</xref>, RRID:<ext-link ext-link-type="uri" xlink:href="rridsoftware:SCR_008624">SCR_008624</ext-link>). The SBGN diagram of <xref ref-type="fig" rid="F1">Figure 1</xref> was created using Cell Designer version 4.4 (<xref ref-type="bibr" rid="B15">Funahashi et al., 2003</xref>, RRID:<ext-link ext-link-type="uri" xlink:href="rridsoftware:SCR_007263">SCR_007263</ext-link>) and then edited with Inkscape version 1.1 (RRID:<ext-link ext-link-type="uri" xlink:href="rridsoftware:SCR_014479">SCR_014479</ext-link>).</p>
<fig id="F1" position="float">
<label>FIGURE 1</label>
<caption>
<p>Diagram of the segment polarity network following the SBGN standard (<xref ref-type="bibr" rid="B39">Le Nov&#xe8;re et al., 2009</xref>; <xref ref-type="bibr" rid="B67">Tour&#xe9; et al., 2021</xref>). Boxes in light green represent proteins, boxes in yellow represent mRNA. The full model includes several hexagonal cells, this diagram shows only one (cell_0,1) and its interactions with one of its neighbors (cell_0,2). Note that the membrane proteins (EWG, PTC, HH, and PH) exist in six pools, one for each side of the hexagonal cell. Only the proteins in side 5 are shown on the diagram, as well as the proteins on side 2 of the neighboring cell. The membrane proteins are allowed to diffuse between sides of the hexagon, which is also not shown here (<italic>eg.</italic> EGW5_0,1 can transfer reversibly to EGW4_0,1 and EGW6_0,1). The box labeled PTC_T_0,1 represents the sum of all PTC species (from the six sides of the membrane of cell_0,1).</p>
</caption>
<graphic xlink:href="fcell-11-1201673-g001.tif"/>
</fig>
</sec>
<sec id="s2-2">
<title>2.2 Model</title>
<p>The model used here is the segment polarity network model described by <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref>. Briefly it represents a hexagonal array of cells, where each cell can express various genes (<italic>wingless</italic>, <italic>engrailed</italic>, <italic>hedgehog</italic>, <italic>cubitus interruptus</italic>, and <italic>patched</italic>) and where their protein products interact within a cell, and across neighboring cells. <xref ref-type="fig" rid="F1">Figure 1</xref> depicts the interaction network using the SBGN standard (<xref ref-type="bibr" rid="B39">Le Nov&#xe8;re et al., 2009</xref>; <xref ref-type="bibr" rid="B67">Tour&#xe9; et al., 2021</xref>). Note that <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref> analyze two versions of this model, one having less interactions than the other. Here we only look at their full model (i.e., including the dashed arrows in the diagram of their Box 1). Since a 1 &#xd7; 4 grid of cells is enough to replicate the results (<xref ref-type="bibr" rid="B70">von Dassow et al., 2000</xref>), that was used here to obtain all results.</p>
<p>My implementation of the model was first created for the widely used software COPASI (<xref ref-type="bibr" rid="B21">Hoops et al., 2006</xref>; <xref ref-type="bibr" rid="B6">Bergmann et al., 2017</xref>) through a Python script that creates a model with arbitrary number of cells at the user&#x2019;s desire. A second script was created to generate the same model with only one cell, where the interacting species from neighboring cells are included as fixed concentrations. COPASI generates the full set of ODEs automatically based on the network and reaction kinetic rate laws. Unlike the SBML version from <xref ref-type="bibr" rid="B61">Sethna (2008)</xref>, here we have the full reaction network, not just the differential equations. A small formal difference between this version and the original SPN model, is that COPASI expresses ODEs in terms of the species amounts rather than concentrations, but since the cell volumes are not variable this makes no difference and both sets of equations are equivalent.</p>
<p>The model makes extensive use of Hill-type functions where various terms appear in the form <italic>base</italic>
<sup>
<italic>exponent</italic>
</sup>. This is often problematic in IEEE floating point since, for non-integer exponents, those operations are carried out based on the equivalence:<disp-formula id="e1">
<mml:math id="m1">
<mml:msup>
<mml:mrow>
<mml:mtext mathvariant="italic">base</mml:mtext>
</mml:mrow>
<mml:mrow>
<mml:mtext mathvariant="italic">exponent</mml:mtext>
</mml:mrow>
</mml:msup>
<mml:mo>&#x3d;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi>e</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mtext mathvariant="italic">exponent</mml:mtext>
<mml:mo>&#xd7;</mml:mo>
<mml:mi>log</mml:mi>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mtext mathvariant="italic">base</mml:mtext>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
</mml:msup>
<mml:mo>.</mml:mo>
</mml:math>
<label>(1)</label>
</disp-formula>
</p>
<p>Therefore, calculations fail when <italic>base</italic> is negative, even if infinitesimally small (generates a NaN, which in COPASI is translated to an error &#x201c;Invalid state&#x201d;). Unfortunately, due to the nature of predictor-corrector integration algorithms, this can easily happen during a time course integration if one species concentration becomes very close to zero. In order to avoid this problem one can use a kind of &#x201c;guarded&#x201d; exponentiation:<disp-formula id="e2">
<mml:math id="m2">
<mml:msup>
<mml:mrow>
<mml:mtext mathvariant="italic">base</mml:mtext>
</mml:mrow>
<mml:mrow>
<mml:mtext mathvariant="italic">exponent</mml:mtext>
</mml:mrow>
</mml:msup>
<mml:mo>&#x2243;</mml:mo>
<mml:mi>max</mml:mi>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mi>&#x3f5;</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext mathvariant="italic">base</mml:mtext>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:mtext mathvariant="italic">exponent</mml:mtext>
</mml:mrow>
</mml:msup>
<mml:mo>,</mml:mo>
<mml:mi>&#x3f5;</mml:mi>
<mml:mo>&#x3e;</mml:mo>
<mml:mn>0</mml:mn>
<mml:mo>.</mml:mo>
</mml:math>
<label>(2)</label>
</disp-formula>
</p>
<p>Applying this protection to the model changes the rate laws. For example, the rate law for transcription with inducer-repressor pair changes from the original:<disp-formula id="e3">
<mml:math id="m3">
<mml:mi>V</mml:mi>
<mml:mo>&#x22c5;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mi>I</mml:mi>
<mml:mo>&#x22c5;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mn>1</mml:mn>
<mml:mo>&#x2212;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:msup>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x2b;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x2b;</mml:mo>
<mml:mi>I</mml:mi>
<mml:mo>&#x22c5;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mn>1</mml:mn>
<mml:mo>&#x2212;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:msup>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x2b;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
</mml:mfrac>
</mml:math>
<label>(3)</label>
</disp-formula>to the alternative:<disp-formula id="e4">
<mml:math id="m4">
<mml:mi>V</mml:mi>
<mml:mo>&#x22c5;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mi>I</mml:mi>
<mml:mo>&#x22c5;</mml:mo>
<mml:mi>max</mml:mi>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mi>&#x3f5;</mml:mi>
<mml:mo>,</mml:mo>
<mml:mn>1</mml:mn>
<mml:mo>&#x2212;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mi>max</mml:mi>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mi>&#x3f5;</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>R</mml:mi>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x2b;</mml:mo>
<mml:mi>max</mml:mi>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mi>&#x3f5;</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>R</mml:mi>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x2b;</mml:mo>
<mml:mi>I</mml:mi>
<mml:mo>&#x22c5;</mml:mo>
<mml:mi>max</mml:mi>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mi>&#x3f5;</mml:mi>
<mml:mo>,</mml:mo>
<mml:mn>1</mml:mn>
<mml:mo>&#x2212;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mi>max</mml:mi>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mi>&#x3f5;</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>R</mml:mi>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x2b;</mml:mo>
<mml:mi>max</mml:mi>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mi>&#x3f5;</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>R</mml:mi>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:mrow>
</mml:mfrac>
<mml:mo>.</mml:mo>
</mml:math>
<label>(4)</label>
</disp-formula>
</p>
<p>The terms <inline-formula id="inf1">
<mml:math id="m5">
<mml:msubsup>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> and <inline-formula id="inf2">
<mml:math id="m6">
<mml:msubsup>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>h</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> are not protected by a &#x201c;guard&#x201d; because <italic>k</italic>
<sub>1</sub> and <italic>k</italic>
<sub>2</sub> are constants that are always positive. In the results presented here I have used <italic>&#x3f5;</italic> &#x3d; 10<sup>&#x2013;80</sup>, which reduced the incidence of simulations with NaNs from around 10%&#x2013;0.1%. <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref> did not describe how they avoided this problem within the software Ingeneue. Use of these alternative rate laws was necessary for the random parameter sampling, but for specific time course simulations one can almost always use the original rate laws as described in <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref>.</p>
<p>Several aspects of the original SPN model were not fully described by <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref> and I have had to resort to later publications to infer what they could be. For the sake of complete transparency, here are all the details that had to be inferred from sources other than the original article.<list list-type="simple">
<list-item>
<p>&#x2022; Parameter <italic>H</italic>
<sub>
<italic>EWG</italic>
</sub> does not feature in the differential equations of the <xref ref-type="sec" rid="s10">Supplementary Material S1</xref> or in <xref ref-type="bibr" rid="B69">von Dassow and Odell (2002)</xref>, instead there the proteins <italic>EWG</italic> and <italic>IWG</italic> have the same half-life (<italic>H</italic>
<sub>
<italic>IWG</italic>
</sub>). However the parameter is clearly described as one of the 48 parameters sampled in <xref ref-type="bibr" rid="B45">Meir et al. (2002)</xref>, from the same group. Thus in my implementation <italic>EWG</italic> has its own half-life <italic>H</italic>
<sub>
<italic>EWG</italic>
</sub>.</p>
</list-item>
<list-item>
<p>&#x2022; The identity of the 48 parameters that are sampled was not described unequivocally. There are in fact 53 parameters in the model (when considering 4 cells), so while 46 were obvious from their <xref ref-type="sec" rid="s10">Supplementary Table S1</xref>, the other 2 could have been any of the remaining 7 &#x2026; Again, a Figure in <xref ref-type="bibr" rid="B45">Meir et al. (2002)</xref> provided the identity of the 48 parameters (which include the one mentioned in the previous bullet).</p>
</list-item>
<list-item>
<p>&#x2022; The ranges for parameter samplings are provided in <xref ref-type="sec" rid="s10">Supplementary Table S1</xref>, however it missed including the ranges for parameters <italic>PTC</italic>
<sub>0</sub> and <italic>HH</italic>
<sub>0</sub>. <xref ref-type="bibr" rid="B35">Kim (2009)</xref> mentions this range as 1&#x2013;1000 (their table 3, parameters &#x201c;max&#x201d;), while an Ingeneue network file (named <ext-link ext-link-type="uri" xlink:href="http://spg1_01_4cell.net">spg1_01_4cell.net</ext-link>), recovered from the Internet Archive (<xref ref-type="bibr" rid="B36">Kim, 2010</xref>), suggests it could be 10<sup>3</sup>&#x2013;10<sup>6</sup>. I ran simulations with both ranges, and the range 1&#x2013;1000 produces results closer to those reported by <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref>.</p>
</list-item>
<list-item>
<p>&#x2022; The score function used to identify parameter sets that result in the desired properties was described without sufficient detail. This scoring function is a composite of a function to identify the gene expression pattern (Eq. 15 of their <xref ref-type="sec" rid="s10">Supplementary Material S1</xref>), and another to detect stable stripes (Eq. 16 of their <xref ref-type="sec" rid="s10">Supplementary Material S1</xref>); the final score being the largest of these two. The text does not specify clearly what the symbols of Eq. (16) mean, particularly the <italic>StripeScore</italic>. Thus I only used Eq. (15) for scoring. By definition my results should identify more parameter sets than the full scoring criterion (since we are looking for scores below a threshold of 0.2).</p>
</list-item>
<list-item>
<p>&#x2022; The initial conditions probed in each line of <xref ref-type="table" rid="T1">Table 1</xref> of the original paper are not specified exactly, instead they provide ranges, such as <inline-formula id="inf3">
<mml:math id="m7">
<mml:mo>&#x3c;</mml:mo>
<mml:mn>20</mml:mn>
</mml:math>
</inline-formula>% value, or 20%&#x2013;60%, not saying whether the values used were random within that range or some actual specific values. I used 0.15 for when they indicate <inline-formula id="inf4">
<mml:math id="m8">
<mml:mo>&#x3c;</mml:mo>
<mml:mn>20</mml:mn>
</mml:math>
</inline-formula>%, 0.4 for when they specify 20%&#x2013;60%, and 0.9 when they specify 60%&#x2013;100%. For the &#x201c;degraded&#x201d; initial condition this is even more problematic as they only provided a bar chart without axes, rather than actual values. The values I used here are specified in the Python code and in the COPASI and SBML files for the time course described below.</p>
</list-item>
</list>
</p>
<p>As described in the Interoperability section of Results, below, the model can be exported from COPASI in standard formats, particularly the systems biology markup language (SBML, <xref ref-type="bibr" rid="B24">Hucka et al., 2003</xref>; <xref ref-type="bibr" rid="B32">Keating et al., 2020</xref>) and the OMEX format (<xref ref-type="bibr" rid="B5">Bergmann et al., 2014</xref>) containing a SBML file for the model and a SED-ML (<xref ref-type="bibr" rid="B73">Waltemath et al., 2011b</xref>) file with the simulation specification.</p>
</sec>
</sec>
<sec sec-type="results" id="s3">
<title>3 Results</title>
<sec id="s3-1">
<title>3.1 Reproducibility</title>
<p>It is rather unfortunate that the term &#x201c;reproducibility&#x201d; has itself been used with various different meanings. This confusion in terminology was discussed in detail by <xref ref-type="bibr" rid="B16">Goodman et al. (2016)</xref>, <xref ref-type="bibr" rid="B54">Plesser (2018)</xref>, <xref ref-type="bibr" rid="B48">Mi&#x142;kowski et al. (2018)</xref>, and especially <xref ref-type="bibr" rid="B4">Barba (2018)</xref>. As previously (<xref ref-type="bibr" rid="B46">Mendes, 2018</xref>), I will follow the definitions of <xref ref-type="bibr" rid="B16">Goodman et al. (2016)</xref>, which specifies three distinct types of reproducibility.<list list-type="simple">
<list-item>
<p>&#x2022; <italic>reproducibility of methods</italic> requires one to be able to exactly reproduce the results using the same methods on the same data;</p>
</list-item>
<list-item>
<p>&#x2022; <italic>reproducibility of results</italic> requires one to obtain similar results in an independent study applying similar procedures;</p>
</list-item>
<list-item>
<p>&#x2022; <italic>reproducibility of inferences</italic> requires the same conclusions to be reached in an independent replication potentially following a different methodology.</p>
</list-item>
</list>
</p>
<p>Because the software Ingeneue, originally used to build and simulate the SPN model, has now disappeared from circulation, reproducibility of methods can no longer be effectively carried out. In a later publication <xref ref-type="bibr" rid="B69">von Dassow and Odell (2002)</xref> appear to have reproduced the results with the same software (see <xref ref-type="table" rid="T1">Table 1</xref>), however since these are the original authors, that can hardly be seen as independent verification. Of all the works listed in <xref ref-type="table" rid="T1">Table 1</xref>, only <xref ref-type="bibr" rid="B28">Ingolia (2004)</xref> and <xref ref-type="bibr" rid="B41">Ma et al. (2006)</xref> can be seen as independent reproductions of the original results. Unfortunately those two publications used their own C programs but did not publish them. It was work in Sethna&#x2019;s lab (<xref ref-type="bibr" rid="B17">Gutenkunst et al., 2007</xref>; <xref ref-type="bibr" rid="B12">Daniels et al., 2008</xref>) that resulted in an electronic version of the model being created in the SBML format that is still available (see notes to <xref ref-type="table" rid="T1">Table 1</xref>), and which was re-used by <xref ref-type="bibr" rid="B44">Marazzi et al. (2022)</xref>. However this SBML implementation coded the ODEs directly without representing the reaction network, an important limitation.</p>
<p>I attempted to reproduce the results of <xref ref-type="table" rid="T1">Table 1</xref> in <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref>, displayed in our <xref ref-type="table" rid="T2">Table 2</xref>. Overall these results match the original ones fairly well. There are some discrepancies in two samplings, but these are likely due to the uncertainty on the actual initial values, as pointed out in Methods. Bear in mind that these are very small samples of a 48-dimensional parameter space and the differences may just be due to random sampling. <xref ref-type="fig" rid="F2">Figure 2</xref> displays the succesful parameter sets in the sampling with crisp initial conditions, corresponding to <xref ref-type="fig" rid="F2">Figure 2A</xref> in <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref>. Careful comparison between the Figure and the original one reveals similar distributions. For example, in both cases <italic>&#x3ba;</italic>
<sub>
<italic>CNen</italic>
</sub> rarely takes large values. The conclusions taken by <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref> would not change if their <xref ref-type="fig" rid="F2">Figure 2A</xref> was substituted by this <xref ref-type="fig" rid="F2">Figure 2</xref>. Taking these results together, I propose that the current implementation of the SPN model matches the results of the original&#x2014;<italic>reproducibility of results.</italic>
</p>
<table-wrap id="T2" position="float">
<label>TABLE 2</label>
<caption>
<p>Frequency of solutions as a function of initial conditions.</p>
</caption>
<table>
<thead valign="top">
<tr>
<th align="left"/>
<th colspan="3" align="center">
<xref ref-type="bibr" rid="B70">Von Dassow et al. (2000)</xref>
</th>
<th colspan="3" align="center">This work</th>
</tr>
<tr>
<td align="left">Initial conditions</td>
<td align="left">Hits</td>
<td align="left">Tries</td>
<td align="left">Hit rate</td>
<td align="left">Hits</td>
<td align="left">Tries</td>
<td align="left">Hit rate</td>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">Crisp</td>
<td align="left">1,192</td>
<td align="left">240,000</td>
<td align="left">1/201</td>
<td align="left">1,015</td>
<td align="left">239,272</td>
<td align="left">1/236</td>
</tr>
<tr>
<td align="left">Degraded</td>
<td align="left">149</td>
<td align="left">750,000</td>
<td align="left">1/5,000</td>
<td align="left">22</td>
<td align="left">749,988</td>
<td align="left">1/34,090</td>
</tr>
<tr>
<td align="left">Crisp, plus ubiquitous low-level <italic>ci</italic> and <italic>ptc</italic>
</td>
<td align="left">110</td>
<td align="left">41,258</td>
<td align="left">1/375</td>
<td align="left">91</td>
<td align="left">41,941</td>
<td align="left">1/461</td>
</tr>
<tr>
<td align="left">3-cell band of <italic>ci</italic>, <italic>wg</italic> stripe on posterior margin</td>
<td align="left">69</td>
<td align="left">40,338</td>
<td align="left">1/585</td>
<td align="left">97</td>
<td align="left">41,994</td>
<td align="left">1/433</td>
</tr>
<tr>
<td align="left">3-cell band of <italic>ptc</italic>, <italic>en</italic> stripe on anterior margin</td>
<td align="left">127</td>
<td align="left">36,196</td>
<td align="left">1/285</td>
<td align="left">102</td>
<td align="left">37,994</td>
<td align="left">1/372</td>
</tr>
<tr>
<td align="left">3-cell band of <italic>ptc</italic>, out-of-phase 3-cell band of <italic>ci</italic>
</td>
<td align="left">16</td>
<td align="left">226,084</td>
<td align="left">1/14,130</td>
<td align="left">168</td>
<td align="left">229,996</td>
<td align="left">1/1,369</td>
</tr>
<tr>
<td align="left">10.5281/zenodo.7772570 Close to target pattern</td>
<td align="left">464</td>
<td align="left">21,526</td>
<td align="left">1/46</td>
<td align="left">556</td>
<td align="left">21,992</td>
<td align="left">1/39</td>
</tr>
</tbody>
</table>
</table-wrap>
<fig id="F2" position="float">
<label>FIGURE 2</label>
<caption>
<p>Graphic representation of &#x201d;solutions&#x201d; obtained with crisp initial conditions. All 1,015 parameter sets with a score below 0.2 are displayed. Black lines plot mean and standard deviation. Each spoke represents the log-scale range of one parameter. Half-lives and cooperativity coefficients are omitted, as in Panel 2A of <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref>. This figure was created with the open source software Gnuplot and its source is included with the available data sets (see Data Availability Statement).</p>
</caption>
<graphic xlink:href="fcell-11-1201673-g002.tif"/>
</fig>
</sec>
<sec id="s3-2">
<title>3.2 Interoperability</title>
<p>To demonstrate that this implementation of the SPN model is interoperable across different software, a specific time course was chosen to be run by several simulators (herafter named <italic>timecourse1</italic>). One of the successful parameter sets generated in the random sampling with the &#x201c;degraded&#x201d; initial condition was chosen and saved as a native COPASI file, an SBML Level 3 Version 1 file (<xref ref-type="bibr" rid="B25">Hucka et al., 2018</xref>), and an OMEX file (<xref ref-type="bibr" rid="B5">Bergmann et al., 2014</xref>). Both the COPASI and OMEX files include the specification of the time course (end time of 1100 time units, sampled every 5 time units), though the SBML file requires that time course to be specified separately in the destination simulator.</p>
<p>Timecourse1 was simulated in four different software tools: COPASI, Virtual Cell (<xref ref-type="bibr" rid="B59">Schaff et al., 1997</xref>; <xref ref-type="bibr" rid="B49">Moraru et al., 2008</xref>), Tellurium (<xref ref-type="bibr" rid="B10">Choi et al., 2018</xref>), and AMICI (<xref ref-type="bibr" rid="B14">Fr&#xf6;hlich et al., 2021</xref>). It was run locally with COPASI, Virtual Cell, and Tellurium, and through the web service runBioSimulations (<xref ref-type="bibr" rid="B62">Shaikh et al., 2021</xref>) with AMICI. COPASI used the native file format, Tellurium used the SBML (through a small Python script runTellurium.py), while Virtual Cell and AMICI used the OMEX file.</p>
<p>
<xref ref-type="fig" rid="F3">Figures 3</xref>, <xref ref-type="fig" rid="F4">4</xref> display the time course simulations obtained with four different software. There are no visible differences in the trajectories displayed confirming that these packages are all equally able to reproduce the results. Note that different ODE solvers were used by each one: COPASI used LSODA (<xref ref-type="bibr" rid="B53">Petzold, 1983</xref>), Virtual Cell used a fixed-step size Adams-Moulton method (<xref ref-type="bibr" rid="B18">Han and Han, 2002</xref>), Tellurium used CVODE (using the Adams-Moulton variable order, variable step size method) and AMICI used CVODES, both part of the SUNDIALS suite (<xref ref-type="bibr" rid="B20">Hindmarsh et al., 2005</xref>).</p>
<fig id="F3" position="float">
<label>FIGURE 3</label>
<caption>
<p>Time course simulation of mRNA species in a 1&#xd7;4 arrangement of cells using a parameter set obtained by random sampling from the &#x201c;degraded&#x201d; initial condition (see <xref ref-type="table" rid="T2">Table 2</xref>). Columns represent the different cells; the middle dashed line separating cell 2 and cell 3 represents a parasegmental boundary. Displayed in each plot are the time evolution of all mRNA species in that cell. Note the formation of the expected segment polarity pattern around the parasegmental boundary, with high levels of <italic>wingless</italic> and <italic>patched</italic> in cell 2, and high levels of <italic>engrailed</italic> and <italic>hedgehog</italic> in cell 3. Each row corresponds to simulations carried out by different software. COPASI used the LSODA algorithm with absolute tolerance 10<sup>&#x2013;13</sup> and relative tolerance 10<sup>&#x2013;8</sup>. Virtual Cell used a fixed step size Adams-Moulton algorithm (step size 0.1). Tellurium used CVODE non-stiff algorithm (variable step size, variable order Adams-Moulton) with absolute tolerance of 10<sup>&#x2013;12</sup> and relative tolerance of 10<sup>&#x2013;6</sup>. AMICI used CVODES with absolute tolerance of 10<sup>&#x2013;16</sup> and relative tolerance of 10<sup>&#x2013;8</sup>. Results from the four simulators are visibly the same.</p>
</caption>
<graphic xlink:href="fcell-11-1201673-g003.tif"/>
</fig>
<fig id="F4" position="float">
<label>FIGURE 4</label>
<caption>
<p>Time course simulation of protein species as in <xref ref-type="fig" rid="F3">Figure 3</xref>. Displayed in each plot are the time evolution of some of the protein species in that cell. Species <italic>EWG</italic>
<sub>
<italic>T</italic>
</sub> represents the total amount of <italic>EWG</italic> protein (product of <italic>wingless</italic>) located in the membranes of the six neighboring cells to the one displayed; <italic>PH</italic>
<sub>
<italic>T</italic>
</sub> is the sum of all patched&#x2013;hedgehog complexes located in the six sides of that cell&#x2019;s membrane, and <italic>PTC</italic>
<sub>
<italic>T</italic>
</sub> is the sum of all free patched receptor located in the six sides of that cell&#x2019;s membrane. Each row corresponds to simulations carried out by different software with different algorithms. As in <xref ref-type="fig" rid="F3">Figure 3</xref>, there are no visible differences in the results of the four simulators.</p>
</caption>
<graphic xlink:href="fcell-11-1201673-g004.tif"/>
</fig>
</sec>
<sec id="s3-3">
<title>3.3 Findability and accessability</title>
<p>To promote findability and accessibility, the model files and associated scripts are made available through the following channels: a) a GitHub repository <ext-link ext-link-type="uri" xlink:href="https://github.com/pmendes/models/tree/main/vonDassow2000">https://github.com/pmendes/models/tree/main/vonDassow2000</ext-link>, b) a Zenodo accession <ext-link ext-link-type="uri" xlink:href="doi:DOI%20(doi:10.5281/zenodo.7772570">DOI (doi:10.5281/zenodo.7772570</ext-link>), c) a submission to the Biomodels database (MODEL2304060001), and d) model files deposited in the database of public Virtual Cell models. Note that the complete result files are only accessible through Zenodo since several files were larger than the limit at GitHub.</p>
</sec>
<sec id="s3-4">
<title>3.4 Reuse</title>
<p>To demonstrate how the model can be reused for different purposes, I decided to ask the question &#x201c;how often do parameter sets of the SPN model have multiple steady states?&#x201d; Earlier <xref ref-type="bibr" rid="B69">von Dassow and Odell (2002)</xref> and especially <xref ref-type="bibr" rid="B28">Ingolia (2004)</xref> proposed that the robustness of pattern formation in the SPN model is due to multi-stability of steady states. <xref ref-type="bibr" rid="B28">Ingolia (2004)</xref> showed this in SPN models of a single cell (where the interacting species from the neighboring cells are kept constant). Here I investigate the answer to this question in a 1 &#xd7; 4 array of cells. The strategy I used is as follows.<list list-type="simple">
<list-item>
<p>1. Generate <italic>p</italic> random sets of parameter values;</p>
</list-item>
<list-item>
<p>2. For each set of parameter values generate <italic>i</italic> random sets of initial conditions and calculate their steady state by integration;</p>
</list-item>
<list-item>
<p>3. Determine how many sets of parameter values produced more than one steady state.</p>
</list-item>
</list>
</p>
<p>COPASI can easily to carry out such a study directly with the <italic>Parameter scan</italic> and <italic>Steady state</italic> tasks. The steady state task was applied here disabling the Newton method and therefore only using ODE integration to find the steady state reachable from the initial conditions (the steady state resolution was set to 10<sup>&#x2013;4</sup> and the criterion used was &#x201c;distance and time&#x201d;). With the parameter scan task, 5,000 random parameter sets were sampled, using the same rules as in <xref ref-type="sec" rid="s3-1">Section 3.1</xref> above. Then, for each parameter set, it sampled 15 random initial conditions. Since we use a model of 1 &#xd7; 4 array of cells, the initial conditions are composed of 132 species concentrations that were sampled in the interval [0,1].</p>
<p>From the 5,000 random parameter sets generated, 3,387 had at least one steady state (the remainder are likely to contain limit cycles, but this was not investigated). Of those 3,387 parameter sets with steady states, 498 contained more than one steady state. This rate of 1/10 parameter sets displaying multistability is not entirely surprising given the study by <xref ref-type="bibr" rid="B28">Ingolia (2004)</xref> which highlighted the positive feedbacks contained in the SPN model. Nevertheless it is interesting to investigate if these 498 parameter sets have special characteristics <italic>versus</italic> the other 2,889 that have only one steady state.</p>
<p>The distributions of parameter values that support multiple steady states was compared with those that appear to only support a single steady state. Calculation of the relative change in the median values for each parameter in the single steady state set <italic>versus</italic> the multiple steady state set revealed that only <italic>&#x3ba;</italic>
<sub>
<italic>CNptc</italic>
</sub> shows a large difference, with a median 5-fold larger in the multiple steady state set than in the single steady state set. Three others have much lower differences: <italic>&#x3ba;</italic>
<sub>
<italic>CNen</italic>
</sub> 0.7-fold smaller, <italic>&#x3ba;</italic>
<sub>
<italic>CIptc</italic>
</sub> 0.46-fold smaller, and <italic>HH</italic>
<sub>0</sub> 0.45-fold smaller. The other 44 parameters have smaller differences. <xref ref-type="fig" rid="F5">Figure 5</xref> depicts the distributions of values of <italic>&#x3ba;</italic>
<sub>
<italic>CNptc</italic>
</sub> and <italic>&#x3ba;</italic>
<sub>
<italic>CNen</italic>
</sub> for the two data sets. <xref ref-type="sec" rid="s10">Supplementary Figures S1&#x2013;S3</xref> depict histograms for all of the 48 parameters. There seems to be very few parameter sets that lead to multiple steady states with low values of <italic>&#x3ba;</italic>
<sub>
<italic>CNptc</italic>
</sub>, while many more have high values for this parameter. This suggests that in order to achieve multiple stability the repression of <italic>patched</italic> (<italic>ptc</italic>) transcription by the truncated protein product of <italic>cubitus interruptus</italic> (<italic>CN</italic>) should be weak. Note that there is another negative feedback loop between these two genes, through induction of <italic>ptc</italic> transcription by the full length <italic>cubitus interruptus</italic> protein (<italic>CI</italic>).</p>
<fig id="F5" position="float">
<label>FIGURE 5</label>
<caption>
<p>Distribution of values of <italic>&#x3ba;</italic>
<sub>
<italic>CNptc</italic>
</sub> and <italic>&#x3ba;</italic>
<sub>
<italic>CNen</italic>
</sub> originating single steady states, <italic>versus</italic> those originating multiple steady states. Scatterplot of values of the two parameters and histograms of their distribution. Darker blue circles represent parameter sets for which only one steady state was identified, lighter orange circles represent parameter sets for which more than one steady state could be identified.</p>
</caption>
<graphic xlink:href="fcell-11-1201673-g005.tif"/>
</fig>
</sec>
</sec>
<sec sec-type="discussion" id="s4">
<title>4 Discussion</title>
<p>It is widely recognized that there is a &#x201c;reproducibility crisis&#x201d; in science (<xref ref-type="bibr" rid="B3">Baker, 2016</xref>) that includes computational science (<xref ref-type="bibr" rid="B47">Mesirov, 2010</xref>; <xref ref-type="bibr" rid="B52">Peng, 2011</xref>; <xref ref-type="bibr" rid="B63">Stodden et al., 2016</xref>) and indeed computational modeling of biological systems (<xref ref-type="bibr" rid="B71">Waltemath and Wolkenhauer, 2016</xref>; <xref ref-type="bibr" rid="B46">Mendes, 2018</xref>; <xref ref-type="bibr" rid="B66">Tiwari et al., 2021</xref>). I and others argue that reproducibility of results obtained from computer simulations of biological models (biomodels) could be enhanced by using open source software (<xref ref-type="bibr" rid="B27">Ince et al., 2012</xref>; <xref ref-type="bibr" rid="B46">Mendes, 2018</xref>) that implement widely adopted standards (<xref ref-type="bibr" rid="B71">Waltemath and Wolkenhauer, 2016</xref>; <xref ref-type="bibr" rid="B8">Blinov et al., 2021</xref>; <xref ref-type="bibr" rid="B57">Porubsky et al., 2021</xref>), which are part of various sets of rules proposed in the last 2&#xa0;decades (<xref ref-type="bibr" rid="B37">Le Nov&#xe8;re et al., 2005</xref>; <xref ref-type="bibr" rid="B72">Waltemath et al., 2011a</xref>; <xref ref-type="bibr" rid="B40">Lewis et al., 2016</xref>; <xref ref-type="bibr" rid="B56">Porubsky et al., 2020</xref>). Adoption of such practices, though, will only become widespread when enforced by publishers (<xref ref-type="bibr" rid="B60">Schnell, 2018</xref>; <xref ref-type="bibr" rid="B64">Stodden et al., 2018</xref>) and funding agencies (<xref ref-type="bibr" rid="B78">Yale Law School Roundtable Participants, 2010</xref>). A recent move by the US National Institutes of Health to enforce standards for data management (<xref ref-type="bibr" rid="B50">National Institutes of Health, 2020</xref>) is an encouraging move in that direction.</p>
<p>While reproducibility is a fundamental part of the scientific process (<xref ref-type="bibr" rid="B55">Popper, 1959</xref>), another important aspect is that new discoveries are almost always dependent on previous results, methodologies, and theories. To facilitate reuse of scientific data the community is increasingly adopting the so-called FAIR data principles (<xref ref-type="bibr" rid="B76">Wilkinson et al., 2016</xref>) which promote <italic>Findability</italic>, <italic>Accessibility</italic>, <italic>Interoperability</italic>, and <italic>Reuse</italic> of data. While biomodels are usually seen as mathematics or software, they are operationally complex data objects and these principles ought to apply to them as well. Here I reproduced the reaction network, ODE model and associated simulations described in the classic systems biology paper by <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref> with the software COPASI. I then exported the model and simulation specifications in community-derived standard formats that are supported by many software applications. Finally these files were contributed to model and data repositories. This essentially makes the model available to be manipulated by a large number of software applications, not only extant but likely future ones. Even if the standards used here will be abandoned in the future, it is most likely that converters would be developed to upgrade models to the new standards. Model and data repositories are also expected to last a long time. Thus this classic systems biology and development model is now available to a wide community, enabling its re-use for many decades.</p>
<p>As in previous case studies (e.g., <xref ref-type="bibr" rid="B30">Jablonsky et al., 2011</xref>; <xref ref-type="bibr" rid="B66">Tiwari et al., 2021</xref>), not all required information to reproduce the model and simulations were available in the original publication. Fortunately, there were subsequent publications by the authors and other members of their teams that hinted at the missing pieces. In some cases there is still uncertainty whether I made the correct choices, however the results obtained (<xref ref-type="fig" rid="F2">Figure 2</xref>) are sufficiently close to the original that these choices are at least validated to be highly plausible. This supports previous suggestions (<xref ref-type="bibr" rid="B11">Claerbout and Karrenbach, 1992</xref>; <xref ref-type="bibr" rid="B22">Hothorn and Leisch, 2011</xref>; <xref ref-type="bibr" rid="B63">Stodden et al., 2016</xref>) that true computational reproducibility requires availability of electronic executable versions. Unfortunately textual descriptions are almost always deficient in details, as it is only too easy to miss something.</p>
<p>While the missing information in <xref ref-type="bibr" rid="B70">von Dassow et al. (2000)</xref> could be seen as a negative, I note that at the time the software Ingeneue was distributed together with files that allowed reproduction of the results. Additionally the model was actually described in great detail, so much that I was able to re-implement it. It is not uncommon to come across cases where even the model equations are not listed (see, e.g., <xref ref-type="bibr" rid="B23">H&#xfc;bner et al., 2011</xref>, for a survey). However, this also highlights that publishing an electronic version alone is not guarantee that others in the future will be able to use it. In this case the software Ingeneue is no longer distributed and thus the electronic version is essentially lost (I could have tried to seek a copy from the original authors but I decided not to do so in order to test whether I could reproduce it with the information available). Publication of models in a widely used standard format is essential, as only this will assure the model to be interoperable by future software. Again, this is not a criticism of this 23&#xa0;year-old publication, since at that time the relevant standards were nonexistent.</p>
<p>In conclusion: we have all the tools needed to make computational systems biology models FAIR. They should be encoded in standard formats with relevant metadata and deposited in widely used repositories. Only this will assure that future researchers will be able to study and re-use these models. Any other option, such as only describing model equations, making the model available &#x201c;upon request&#x201d;, or non-standard electronic encodings of the model will likely be lost within a decade or less.</p>
</sec>
</body>
<back>
<sec sec-type="data-availability" id="s5">
<title>Data availability statement</title>
<p>The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: GitHub: <ext-link ext-link-type="uri" xlink:href="https://github.com/pmendes/models/tree/main/vonDassow2000">https://github.com/pmendes/models/tree/main/vonDassow2000</ext-link> Zenodo: <ext-link ext-link-type="uri" xlink:href="https://zenodo.org/record/7772570">https://zenodo.org/record/7772570</ext-link> Biomodels: <ext-link ext-link-type="uri" xlink:href="https://www.ebi.ac.uk/biomodels/MODEL2304060001">https://www.ebi.ac.uk/biomodels/MODEL2304060001</ext-link>.</p>
</sec>
<sec id="s6">
<title>Author contributions</title>
<p>PM created the concept and design of the study, run all computations, wrote the entire manuscript. PM revised, read, and approved the submitted version.</p>
</sec>
<sec id="s7">
<title>Funding</title>
<p>Research reported in this publication was supported by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number R24 GM137787. The content is solely the responsibility of the author and does not necessarily represent the official views of the National Institutes of Health.</p>
</sec>
<ack>
<p>I am grateful to Lauren Marazzi who drew my attention to the issues of findability and accessibility of this model; to Frank Bergmann who improved the BasiCO package at my request with incredible speed; to Ion Moraru and Lucian P. Smith for help with appropriately running Virtual Cell and Tellurium, respectively. I am also grateful to Eran Agmon for many discussions about interoperability of biomodels.</p>
</ack>
<sec sec-type="COI-statement" id="s8">
<title>Conflict of interest</title>
<p>The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p>
</sec>
<sec sec-type="disclaimer" id="s9">
<title>Publisher&#x2019;s note</title>
<p>All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.</p>
</sec>
<sec id="s10">
<title>Supplementary material</title>
<p>The Supplementary Material for this article can be found online at: <ext-link ext-link-type="uri" xlink:href="https://www.frontiersin.org/articles/10.3389/fcell.2023.1201673/full#supplementary-material">https://www.frontiersin.org/articles/10.3389/fcell.2023.1201673/full&#x23;supplementary-material</ext-link>
</p>
<supplementary-material xlink:href="DataSheet1.PDF" id="SM1" mimetype="application/PDF" xmlns:xlink="http://www.w3.org/1999/xlink"/>
</sec>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Albert</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Othmer</surname>
<given-names>H. G.</given-names>
</name>
</person-group> (<year>2003</year>). <article-title>The topology of the regulatory interactions predicts the expression pattern of the segment polarity genes in drosophila melanogaster</article-title>. <source>J. Theor. Biol.</source> <volume>223</volume>, <fpage>1</fpage>&#x2013;<lpage>18</lpage>. <pub-id pub-id-type="doi">10.1016/s0022-5193(03)00035-3</pub-id>
</citation>
</ref>
<ref id="B2">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Albert</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>DasGupta</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Hegde</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Sivanathan</surname>
<given-names>G. S.</given-names>
</name>
<name>
<surname>Gitter</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>G&#xfc;rsoy</surname>
<given-names>G.</given-names>
</name>
<etal/>
</person-group> (<year>2011</year>). <article-title>Computationally efficient measure of topological redundancy of biological and social networks</article-title>. <source>Phys. Rev. E, Stat. Nonlinear, Soft Matter Phys.</source> <volume>84</volume>, <fpage>036117</fpage>. <pub-id pub-id-type="doi">10.1103/PhysRevE.84.036117</pub-id>
</citation>
</ref>
<ref id="B3">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Baker</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2016</year>). <article-title>1,500 scientists lift the lid on reproducibility</article-title>. <source>Nature</source> <volume>533</volume>, <fpage>452</fpage>&#x2013;<lpage>454</lpage>. <pub-id pub-id-type="doi">10.1038/533452a</pub-id>
</citation>
</ref>
<ref id="B4">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Barba</surname>
<given-names>L. A.</given-names>
</name>
</person-group> (<year>2018</year>). <source>Terminologies for reproducible research</source>. <comment>
<italic>arXiv preprint</italic>
</comment>. <pub-id pub-id-type="doi">10.48550/arXiv.1802.03311</pub-id>
</citation>
</ref>
<ref id="B5">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Bergmann</surname>
<given-names>F. T.</given-names>
</name>
<name>
<surname>Adams</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Moodie</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Cooper</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Glont</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Golebiewski</surname>
<given-names>M.</given-names>
</name>
<etal/>
</person-group> (<year>2014</year>). <article-title>COMBINE archive and OMEX format: One file to share all information to reproduce a modeling project</article-title>. <source>BMC Bioinforma.</source> <volume>15</volume>, <fpage>369</fpage>. <pub-id pub-id-type="doi">10.1186/s12859-014-0369-z</pub-id>
</citation>
</ref>
<ref id="B6">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Bergmann</surname>
<given-names>F. T.</given-names>
</name>
<name>
<surname>Hoops</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Klahn</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Kummer</surname>
<given-names>U.</given-names>
</name>
<name>
<surname>Mendes</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Pahle</surname>
<given-names>J.</given-names>
</name>
<etal/>
</person-group> (<year>2017</year>). <article-title>COPASI and its applications in biotechnology</article-title>. <source>J. Biotechnol.</source> <volume>261</volume>, <fpage>215</fpage>&#x2013;<lpage>220</lpage>. <pub-id pub-id-type="doi">10.1016/j.jbiotec.2017.06.1200</pub-id>
</citation>
</ref>
<ref id="B7">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Bergmann</surname>
<given-names>F. T.</given-names>
</name>
</person-group> (<year>2023</year>). <source>basico: a simplified python interface to COPASI</source>. <pub-id pub-id-type="doi">10.5281/zenodo.7665294</pub-id>
</citation>
</ref>
<ref id="B8">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Blinov</surname>
<given-names>M. L.</given-names>
</name>
<name>
<surname>Gennari</surname>
<given-names>J. H.</given-names>
</name>
<name>
<surname>Karr</surname>
<given-names>J. R.</given-names>
</name>
<name>
<surname>Moraru</surname>
<given-names>I. I.</given-names>
</name>
<name>
<surname>Nickerson</surname>
<given-names>D. P.</given-names>
</name>
<name>
<surname>Sauro</surname>
<given-names>H. M.</given-names>
</name>
</person-group> (<year>2021</year>). <article-title>Practical resources for enhancing the reproducibility of mechanistic modeling in systems biology</article-title>. <source>Curr. Opin. Syst. Biol.</source> <volume>27</volume>, <fpage>100350</fpage>. <pub-id pub-id-type="doi">10.1016/j.coisb.2021.06.001</pub-id>
</citation>
</ref>
<ref id="B9">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Chaves</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Sengupta</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Sontag</surname>
<given-names>E. D.</given-names>
</name>
</person-group> (<year>2009</year>). <article-title>Geometry and topology of parameter space: Investigating measures of robustness in regulatory networks</article-title>. <source>J. Math. Biol.</source> <volume>59</volume>, <fpage>315</fpage>&#x2013;<lpage>358</lpage>. <pub-id pub-id-type="doi">10.1007/s00285-008-0230-y</pub-id>
</citation>
</ref>
<ref id="B10">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Choi</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Medley</surname>
<given-names>J. K.</given-names>
</name>
<name>
<surname>K&#xf6;nig</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Stocking</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Smith</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Gu</surname>
<given-names>S.</given-names>
</name>
<etal/>
</person-group> (<year>2018</year>). <article-title>Tellurium: An extensible python-based modeling environment for systems and synthetic biology</article-title>. <source>Bio Syst.</source> <volume>171</volume>, <fpage>74</fpage>&#x2013;<lpage>79</lpage>. <pub-id pub-id-type="doi">10.1016/j.biosystems.2018.07.006</pub-id>
</citation>
</ref>
<ref id="B11">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Claerbout</surname>
<given-names>J. F.</given-names>
</name>
<name>
<surname>Karrenbach</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>1992</year>). &#x201c;<article-title>Electronic documents give reproducible research a new meaning</article-title>,&#x201d; in <source>SEG technical program expanded abstracts 1992</source> (<publisher-loc>Houston, TX</publisher-loc>: <publisher-name>Society of Exploration Geophysicists</publisher-name>), <fpage>601</fpage>&#x2013;<lpage>604</lpage>. <pub-id pub-id-type="doi">10.1190/1.1822162</pub-id>
</citation>
</ref>
<ref id="B12">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Daniels</surname>
<given-names>B. C.</given-names>
</name>
<name>
<surname>Chen</surname>
<given-names>Y.-J.</given-names>
</name>
<name>
<surname>Sethna</surname>
<given-names>J. P.</given-names>
</name>
<name>
<surname>Gutenkunst</surname>
<given-names>R. N.</given-names>
</name>
<name>
<surname>Myers</surname>
<given-names>C. R.</given-names>
</name>
</person-group> (<year>2008</year>). <article-title>Sloppiness, robustness, and evolvability in systems biology</article-title>. <source>Curr. Opin. Biotechnol.</source> <volume>19</volume>, <fpage>389</fpage>&#x2013;<lpage>395</lpage>. <pub-id pub-id-type="doi">10.1016/j.copbio.2008.06.008</pub-id>
</citation>
</ref>
<ref id="B13">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Dayarian</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Chaves</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Sontag</surname>
<given-names>E. D.</given-names>
</name>
<name>
<surname>Sengupta</surname>
<given-names>A. M.</given-names>
</name>
</person-group> (<year>2009</year>). <article-title>Shape, size, and robustness: Feasible regions in the parameter space of biochemical networks</article-title>. <source>PLoS Comput. Biol.</source> <volume>5</volume>, <fpage>e1000256</fpage>. <pub-id pub-id-type="doi">10.1371/journal.pcbi.1000256</pub-id>
</citation>
</ref>
<ref id="B14">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Fr&#xf6;hlich</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Weindl</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Sch&#xe4;lte</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Pathirana</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Paszkowski</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Lines</surname>
<given-names>G. T.</given-names>
</name>
<etal/>
</person-group> (<year>2021</year>). <article-title>Amici: High-performance sensitivity analysis for large ordinary differential equation models</article-title>. <source>Bioinformatics</source> <volume>37</volume>, <fpage>3676</fpage>&#x2013;<lpage>3677</lpage>. <pub-id pub-id-type="doi">10.1093/bioinformatics/btab227</pub-id>
</citation>
</ref>
<ref id="B15">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Funahashi</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Morohashi</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Kitano</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Tanimura</surname>
<given-names>N.</given-names>
</name>
</person-group> (<year>2003</year>). <article-title>CellDesigner: A process diagram editor for gene-regulatory and biochemical networks</article-title>. <source>BIOSILICO</source> <volume>1</volume>, <fpage>159</fpage>&#x2013;<lpage>162</lpage>. <pub-id pub-id-type="doi">10.1016/S1478-5382(03)02370-9</pub-id>
</citation>
</ref>
<ref id="B16">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Goodman</surname>
<given-names>S. N.</given-names>
</name>
<name>
<surname>Fanelli</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Ioannidis</surname>
<given-names>J. P. A.</given-names>
</name>
</person-group> (<year>2016</year>). <article-title>What does research reproducibility mean?</article-title> <source>Sci. Transl. Med.</source> <volume>8</volume>, <fpage>341ps12</fpage>. <pub-id pub-id-type="doi">10.1126/scitranslmed.aaf5027</pub-id>
</citation>
</ref>
<ref id="B17">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Gutenkunst</surname>
<given-names>R. N.</given-names>
</name>
<name>
<surname>Waterfall</surname>
<given-names>J. J.</given-names>
</name>
<name>
<surname>Casey</surname>
<given-names>F. P.</given-names>
</name>
<name>
<surname>Brown</surname>
<given-names>K. S.</given-names>
</name>
<name>
<surname>Myers</surname>
<given-names>C. R.</given-names>
</name>
<name>
<surname>Sethna</surname>
<given-names>J. P.</given-names>
</name>
</person-group> (<year>2007</year>). <article-title>Universally sloppy parameter sensitivities in systems biology models</article-title>. <source>PLoS Comput. Biol.</source> <volume>3</volume>, <fpage>1871</fpage>&#x2013;<lpage>1878</lpage>. <pub-id pub-id-type="doi">10.1371/journal.pcbi.0030189</pub-id>
</citation>
</ref>
<ref id="B18">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Han</surname>
<given-names>T. M.</given-names>
</name>
<name>
<surname>Han</surname>
<given-names>Y.</given-names>
</name>
</person-group> (<year>2002</year>). <article-title>Solving implicit equations arising from Adams-Moulton methods</article-title>. <source>BIT Numer. Math.</source> <volume>42</volume>, <fpage>336</fpage>&#x2013;<lpage>350</lpage>. <pub-id pub-id-type="doi">10.1023/A:1021951025649</pub-id>
</citation>
</ref>
<ref id="B19">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hedley</surname>
<given-names>W. J.</given-names>
</name>
<name>
<surname>Nelson</surname>
<given-names>M. R.</given-names>
</name>
<name>
<surname>Bellivant</surname>
<given-names>D. P.</given-names>
</name>
<name>
<surname>Nielsen</surname>
<given-names>P. F.</given-names>
</name>
</person-group> (<year>2001</year>). <article-title>A short introduction to CellML</article-title>. <source>Philosophical Trans. R. Soc. Lond. Ser. A</source> <volume>359</volume>, <fpage>1073</fpage>&#x2013;<lpage>1089</lpage>. <pub-id pub-id-type="doi">10.1098/rsta.2001.0817</pub-id>
</citation>
</ref>
<ref id="B20">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hindmarsh</surname>
<given-names>A. C.</given-names>
</name>
<name>
<surname>Brown</surname>
<given-names>P. N.</given-names>
</name>
<name>
<surname>Grant</surname>
<given-names>K. E.</given-names>
</name>
<name>
<surname>Lee</surname>
<given-names>S. L.</given-names>
</name>
<name>
<surname>Serban</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Shumaker</surname>
<given-names>D. E.</given-names>
</name>
<etal/>
</person-group> (<year>2005</year>). <article-title>Sundials: Suite of nonlinear and differential/algebraic equation solvers</article-title>. <source>ACM Trans. Math. Softw.</source> <volume>31</volume>, <fpage>363</fpage>&#x2013;<lpage>396</lpage>. <pub-id pub-id-type="doi">10.1145/1089014.1089020</pub-id>
</citation>
</ref>
<ref id="B21">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hoops</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Sahle</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Gauges</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Lee</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Pahle</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Simus</surname>
<given-names>N.</given-names>
</name>
<etal/>
</person-group> (<year>2006</year>). <article-title>COPASI&#x2014;A COmplex PAthway SImulator</article-title>. <source>Bioinformatics</source> <volume>22</volume>, <fpage>3067</fpage>&#x2013;<lpage>3074</lpage>. <pub-id pub-id-type="doi">10.1093/bioinformatics/btl485</pub-id>
</citation>
</ref>
<ref id="B22">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hothorn</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Leisch</surname>
<given-names>F.</given-names>
</name>
</person-group> (<year>2011</year>). <article-title>Case studies in reproducibility</article-title>. <source>Briefings Bioinforma.</source> <volume>12</volume>, <fpage>288</fpage>&#x2013;<lpage>300</lpage>. <pub-id pub-id-type="doi">10.1093/bib/bbq084</pub-id>
</citation>
</ref>
<ref id="B23">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>H&#xfc;bner</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Sahle</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Kummer</surname>
<given-names>U.</given-names>
</name>
</person-group> (<year>2011</year>). <article-title>Applications and trends in systems biology in biochemistry</article-title>. <source>FEBS J.</source> <volume>278</volume>, <fpage>2767</fpage>&#x2013;<lpage>2857</lpage>. <pub-id pub-id-type="doi">10.1111/j.1742-4658.2011.08217.x</pub-id>
</citation>
</ref>
<ref id="B24">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hucka</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Finney</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Sauro</surname>
<given-names>H. M.</given-names>
</name>
<name>
<surname>Bolouri</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Doyle</surname>
<given-names>J. C.</given-names>
</name>
<name>
<surname>Kitano</surname>
<given-names>H.</given-names>
</name>
<etal/>
</person-group> (<year>2003</year>). <article-title>The systems biology markup language (SBML): A medium for representation and exchange of biochemical network models</article-title>. <source>Bioinformatics</source> <volume>19</volume>, <fpage>524</fpage>&#x2013;<lpage>531</lpage>. <pub-id pub-id-type="doi">10.1093/bioinformatics/btg015</pub-id>
</citation>
</ref>
<ref id="B25">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hucka</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Bergmann</surname>
<given-names>F. T.</given-names>
</name>
<name>
<surname>Dr&#xe4;ger</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Hoops</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Keating</surname>
<given-names>S. M.</given-names>
</name>
<name>
<surname>Nov&#xe8;re</surname>
<given-names>N. L.</given-names>
</name>
<etal/>
</person-group> (<year>2018</year>). <article-title>The systems biology markup language (SBML): Language specification for level 3 version 2 core</article-title>. <source>J. Integr. Bioinforma.</source> <volume>15</volume>, <fpage>20170081</fpage>. <pub-id pub-id-type="doi">10.1515/jib-2017-0081</pub-id>
</citation>
</ref>
<ref id="B26">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hunter</surname>
<given-names>J. D.</given-names>
</name>
</person-group> (<year>2007</year>). <article-title>Matplotlib: A 2d graphics environment</article-title>. <source>Comput. Sci. Eng.</source> <volume>9</volume>, <fpage>90</fpage>&#x2013;<lpage>95</lpage>. <pub-id pub-id-type="doi">10.1109/MCSE.2007.55</pub-id>
</citation>
</ref>
<ref id="B27">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Ince</surname>
<given-names>D. C.</given-names>
</name>
<name>
<surname>Hatton</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Graham-Cumming</surname>
<given-names>J.</given-names>
</name>
</person-group> (<year>2012</year>). <article-title>The case for open computer programs</article-title>. <source>Nature</source> <volume>482</volume>, <fpage>485</fpage>&#x2013;<lpage>488</lpage>. <pub-id pub-id-type="doi">10.1038/nature10836</pub-id>
</citation>
</ref>
<ref id="B28">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Ingolia</surname>
<given-names>N. T.</given-names>
</name>
</person-group> (<year>2004</year>). <article-title>Topology and robustness in the Drosophila segment polarity network</article-title>. <source>PLoS Biol.</source> <volume>2</volume>, <fpage>e123</fpage>. <pub-id pub-id-type="doi">10.1371/journal.pbio.0020123</pub-id>
</citation>
</ref>
<ref id="B29">
<citation citation-type="book">
<collab>Internet Archive</collab> (<year>1996</year>). <source>Wayback machine</source>. <comment>Available at: <ext-link ext-link-type="uri" xlink:href="https://web.archive.org/">https://web.archive.org/</ext-link>
</comment>.</citation>
</ref>
<ref id="B30">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Jablonsky</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Bauwe</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Wolkenhauer</surname>
<given-names>O.</given-names>
</name>
</person-group> (<year>2011</year>). <article-title>Modeling the calvin-benson cycle</article-title>. <source>BMC Syst. Biol.</source> <volume>5</volume>, <fpage>185</fpage>. <pub-id pub-id-type="doi">10.1186/1752-0509-5-185</pub-id>
</citation>
</ref>
<ref id="B31">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Jaeger</surname>
<given-names>J.</given-names>
</name>
</person-group> (<year>2009</year>). <article-title>Modelling the <italic>Drosophila</italic> embryo</article-title>. <source>Mol. Biosyst.</source> <volume>5</volume>, <fpage>1549</fpage>&#x2013;<lpage>1568</lpage>. <pub-id pub-id-type="doi">10.1039/b904722k</pub-id>
</citation>
</ref>
<ref id="B32">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Keating</surname>
<given-names>S. M.</given-names>
</name>
<name>
<surname>Waltemath</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>K&#xf6;nig</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Zhang</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Dr&#xe4;ger</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Chaouiya</surname>
<given-names>C.</given-names>
</name>
<etal/>
</person-group> (<year>2020</year>). <article-title>SBML level 3: An extensible format for the exchange and reuse of biological models</article-title>. <source>Mol. Syst. Biol.</source> <volume>16</volume>, <fpage>e9110</fpage>. <pub-id pub-id-type="doi">10.15252/msb.20199110</pub-id>
</citation>
</ref>
<ref id="B33">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Kent</surname>
<given-names>E.</given-names>
</name>
<name>
<surname>Hoops</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Mendes</surname>
<given-names>P.</given-names>
</name>
</person-group> (<year>2012</year>). <article-title>Condor-COPASI: High-throughput computing for biochemical networks</article-title>. <source>BMC Syst. Biol.</source> <volume>6</volume>, <fpage>91</fpage>. <pub-id pub-id-type="doi">10.1186/1752-0509-6-91</pub-id>
</citation>
</ref>
<ref id="B34">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Kim</surname>
<given-names>K. J.</given-names>
</name>
<name>
<surname>Fernandes</surname>
<given-names>V. M.</given-names>
</name>
</person-group> (<year>2009</year>). <article-title>Effects of ploidy and recombination on evolution of robustness in a model of the segment polarity network</article-title>. <source>PLoS Comput. Biol.</source> <volume>5</volume>, <fpage>e1000296</fpage>. <pub-id pub-id-type="doi">10.1371/journal.pcbi.1000296</pub-id>
</citation>
</ref>
<ref id="B35">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Kim</surname>
<given-names>K. J.</given-names>
</name>
</person-group> (<year>2009</year>). <article-title>Ingeneue: A software tool to simulate and explore genetic regulatory networks</article-title>. <source>Methods Mol. Biol.</source> <volume>500</volume>, <fpage>169</fpage>&#x2013;<lpage>200</lpage>. <pub-id pub-id-type="doi">10.1007/978-1-59745-525-1_6</pub-id>
</citation>
</ref>
<ref id="B36">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Kim</surname>
<given-names>K. J.</given-names>
</name>
</person-group> (<year>2010</year>). <source>IngeneueInMathematica</source>. <comment>Available at: <ext-link ext-link-type="uri" xlink:href="https://web.archive.org/web/20100813195616/http://rusty.fhl.washington.edu/ingeneue/mathematica.html">https://web.archive.org/web/20100813195616/http://rusty.fhl.washington.edu/ingeneue/mathematica.html</ext-link>
</comment>.</citation>
</ref>
<ref id="B37">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Le Nov&#xe8;re</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Finney</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Hucka</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Bhalla</surname>
<given-names>U. S.</given-names>
</name>
<name>
<surname>Campagne</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Collado-Vides</surname>
<given-names>J.</given-names>
</name>
<etal/>
</person-group> (<year>2005</year>). <article-title>Minimum information requested in the annotation of biochemical models (MIRIAM)</article-title>. <source>Nat. Biotechnol.</source> <volume>23</volume>, <fpage>1509</fpage>&#x2013;<lpage>1515</lpage>. <pub-id pub-id-type="doi">10.1038/nbt1156</pub-id>
</citation>
</ref>
<ref id="B38">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Le Nov&#xe8;re</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Bornstein</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Broicher</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Courtot</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Donizelli</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Dharuri</surname>
<given-names>H.</given-names>
</name>
<etal/>
</person-group> (<year>2006</year>). <article-title>BioModels database: A free, centralized database of curated, published, quantitative kinetic models of biochemical and cellular systems</article-title>. <source>Nucleic Acids Res.</source> <volume>34</volume>, <fpage>D689</fpage>&#x2013;<lpage>D691</lpage>. <pub-id pub-id-type="doi">10.1093/nar/gkj092</pub-id>
</citation>
</ref>
<ref id="B39">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Le Nov&#xe8;re</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Hucka</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Mi</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Moodie</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Schreiber</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Sorokin</surname>
<given-names>A.</given-names>
</name>
<etal/>
</person-group> (<year>2009</year>). <article-title>The systems biology graphical notation</article-title>. <source>Nat. Biotechnol.</source> <volume>27</volume>, <fpage>735</fpage>&#x2013;<lpage>741</lpage>. <pub-id pub-id-type="doi">10.1038/nbt.1558</pub-id>
</citation>
</ref>
<ref id="B40">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Lewis</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Breeze</surname>
<given-names>C. E.</given-names>
</name>
<name>
<surname>Charlesworth</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Maclaren</surname>
<given-names>O. J.</given-names>
</name>
<name>
<surname>Cooper</surname>
<given-names>J.</given-names>
</name>
</person-group> (<year>2016</year>). <article-title>Where next for the reproducibility agenda in computational biology?</article-title> <source>BMC Syst. Biol.</source> <volume>10</volume>, <fpage>52</fpage>. <pub-id pub-id-type="doi">10.1186/s12918-016-0288-x</pub-id>
</citation>
</ref>
<ref id="B41">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Ma</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Lai</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Ouyang</surname>
<given-names>Q.</given-names>
</name>
<name>
<surname>Tang</surname>
<given-names>C.</given-names>
</name>
</person-group> (<year>2006</year>). <article-title>Robustness and modular design of the Drosophila segment polarity network</article-title>. <source>Mol. Syst. Biol.</source> <volume>2</volume>, <fpage>70</fpage>. <pub-id pub-id-type="doi">10.1038/msb4100111</pub-id>
</citation>
</ref>
<ref id="B42">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Malik-Sheriff</surname>
<given-names>R. S.</given-names>
</name>
<name>
<surname>Glont</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Nguyen</surname>
<given-names>T. V. N.</given-names>
</name>
<name>
<surname>Tiwari</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Roberts</surname>
<given-names>M. G.</given-names>
</name>
<name>
<surname>Xavier</surname>
<given-names>A.</given-names>
</name>
<etal/>
</person-group> (<year>2020</year>). <article-title>BioModels&#x2014;15 years of sharing computational models in life science</article-title>. <source>Nucleic Acids Res.</source> <volume>48</volume>, <fpage>D407</fpage>&#x2013;<lpage>D415</lpage>. <pub-id pub-id-type="doi">10.1093/nar/gkz1055</pub-id>
</citation>
</ref>
<ref id="B43">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Mallavarapu</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Thomson</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Ullian</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Gunawardena</surname>
<given-names>J.</given-names>
</name>
</person-group> (<year>2009</year>). <article-title>Programming with models: Modularity and abstraction provide powerful capabilities for systems biology</article-title>. <source>J. R. Soc. Interface</source> <volume>6</volume>, <fpage>257</fpage>&#x2013;<lpage>270</lpage>. <pub-id pub-id-type="doi">10.1098/rsif.2008.0205</pub-id>
</citation>
</ref>
<ref id="B44">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Marazzi</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Shah</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Balakrishnan</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Patil</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Vera-Licona</surname>
<given-names>P.</given-names>
</name>
</person-group> (<year>2022</year>). <article-title>Netisce: A network-based tool for cell fate reprogramming</article-title>. <source>npj Syst. Biol. Appl.</source> <volume>8</volume>, <fpage>21</fpage>. <pub-id pub-id-type="doi">10.1038/s41540-022-00231-y</pub-id>
</citation>
</ref>
<ref id="B45">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Meir</surname>
<given-names>E.</given-names>
</name>
<name>
<surname>Munro</surname>
<given-names>E. M.</given-names>
</name>
<name>
<surname>Odell</surname>
<given-names>G. M.</given-names>
</name>
<name>
<surname>Von Dassow</surname>
<given-names>G.</given-names>
</name>
</person-group> (<year>2002</year>). <article-title>Ingeneue: A versatile tool for reconstituting genetic networks, with examples from the segment polarity network</article-title>. <source>J. Exp. Zoology</source> <volume>294</volume>, <fpage>216</fpage>&#x2013;<lpage>251</lpage>. <pub-id pub-id-type="doi">10.1002/jez.10187</pub-id>
</citation>
</ref>
<ref id="B46">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Mendes</surname>
<given-names>P.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>Reproducible research using biomodels</article-title>. <source>Bull. Math. Biol.</source> <volume>80</volume>, <fpage>3081</fpage>&#x2013;<lpage>3087</lpage>. <pub-id pub-id-type="doi">10.1007/s11538-018-0498-z</pub-id>
</citation>
</ref>
<ref id="B47">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Mesirov</surname>
<given-names>J. P.</given-names>
</name>
</person-group> (<year>2010</year>). <article-title>Computer science. Accessible reproducible research</article-title>. <source>Science</source> <volume>327</volume>, <fpage>415</fpage>&#x2013;<lpage>416</lpage>. <pub-id pub-id-type="doi">10.1126/science.1179653</pub-id>
</citation>
</ref>
<ref id="B48">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Mi&#x142;kowski</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Hensel</surname>
<given-names>W. M.</given-names>
</name>
<name>
<surname>Hohol</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail</article-title>. <source>J. Comput. Neurosci.</source> <volume>45</volume>, <fpage>163</fpage>&#x2013;<lpage>172</lpage>. <pub-id pub-id-type="doi">10.1007/s10827-018-0702-z</pub-id>
</citation>
</ref>
<ref id="B49">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Moraru</surname>
<given-names>I.</given-names>
</name>
<name>
<surname>Morgan</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Li</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Loew</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Schaff</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Lakshminarayana</surname>
<given-names>A.</given-names>
</name>
<etal/>
</person-group> (<year>2008</year>). <article-title>Virtual Cell modelling and simulation software environment</article-title>. <source>IET Syst. Biol.</source> <volume>2</volume>, <fpage>352</fpage>&#x2013;<lpage>362</lpage>. <pub-id pub-id-type="doi">10.1049/iet-syb:20080102</pub-id>
</citation>
</ref>
<ref id="B50">
<citation citation-type="book">
<collab>National Institutes of Health</collab> (<year>2020</year>). <source>NOT-OD-21-013: Final NIH policy for data management and sharing</source>. <comment>Available at: <ext-link ext-link-type="uri" xlink:href="https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html">https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html</ext-link>
</comment>.</citation>
</ref>
<ref id="B51">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Olivier</surname>
<given-names>B. G.</given-names>
</name>
<name>
<surname>Snoep</surname>
<given-names>J. L.</given-names>
</name>
</person-group> (<year>2004</year>). <article-title>Web-based kinetic modelling using JWS Online</article-title>. <source>Bioinformatics</source> <volume>20</volume>, <fpage>2143</fpage>&#x2013;<lpage>2144</lpage>. <pub-id pub-id-type="doi">10.1093/bioinformatics/bth200</pub-id>
</citation>
</ref>
<ref id="B52">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Peng</surname>
<given-names>R. D.</given-names>
</name>
</person-group> (<year>2011</year>). <article-title>Reproducible research in computational science</article-title>. <source>Science</source> <volume>334</volume>, <fpage>1226</fpage>&#x2013;<lpage>1227</lpage>. <pub-id pub-id-type="doi">10.1126/science.1213847</pub-id>
</citation>
</ref>
<ref id="B53">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Petzold</surname>
<given-names>L.</given-names>
</name>
</person-group> (<year>1983</year>). <article-title>Automatic selection of methods for solving stiff and nonstiff systems of ordinary differential equations</article-title>. <source>SIAM J. Sci. Stat. Comput.</source> <volume>4</volume>, <fpage>136</fpage>&#x2013;<lpage>148</lpage>. <pub-id pub-id-type="doi">10.1137/0904010</pub-id>
</citation>
</ref>
<ref id="B54">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Plesser</surname>
<given-names>H. E.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>Reproducibility vs. replicability: A brief history of a confused terminology</article-title>. <source>Front. Neuroinformatics</source> <volume>11</volume>, <fpage>76</fpage>. <pub-id pub-id-type="doi">10.3389/fninf.2017.00076</pub-id>
</citation>
</ref>
<ref id="B55">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Popper</surname>
<given-names>K.</given-names>
</name>
</person-group> (<year>1959</year>). <source>The logic of scientific discovery</source>. <publisher-loc>London</publisher-loc>: <publisher-name>Hutchinson</publisher-name>.</citation>
</ref>
<ref id="B56">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Porubsky</surname>
<given-names>V. L.</given-names>
</name>
<name>
<surname>Goldberg</surname>
<given-names>A. P.</given-names>
</name>
<name>
<surname>Rampadarath</surname>
<given-names>A. K.</given-names>
</name>
<name>
<surname>Nickerson</surname>
<given-names>D. P.</given-names>
</name>
<name>
<surname>Karr</surname>
<given-names>J. R.</given-names>
</name>
<name>
<surname>Sauro</surname>
<given-names>H. M.</given-names>
</name>
</person-group> (<year>2020</year>). <article-title>Best practices for making reproducible biochemical models</article-title>. <source>Cell. Syst.</source> <volume>11</volume>, <fpage>109</fpage>&#x2013;<lpage>120</lpage>. <pub-id pub-id-type="doi">10.1016/j.cels.2020.06.012</pub-id>
</citation>
</ref>
<ref id="B57">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Porubsky</surname>
<given-names>V.</given-names>
</name>
<name>
<surname>Smith</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Sauro</surname>
<given-names>H. M.</given-names>
</name>
</person-group> (<year>2021</year>). <article-title>Publishing reproducible dynamic kinetic models</article-title>. <source>Briefings Bioinforma.</source> <volume>22</volume>, <fpage>bbaa152</fpage>. <pub-id pub-id-type="doi">10.1093/bib/bbaa152</pub-id>
</citation>
</ref>
<ref id="B58">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Rozum</surname>
<given-names>J. C.</given-names>
</name>
<name>
<surname>Albert</surname>
<given-names>R.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>Identifying (un)controllable dynamical behavior in complex networks</article-title>. <source>PLoS Comput. Biol.</source> <volume>14</volume>, <fpage>e1006630</fpage>. <pub-id pub-id-type="doi">10.1371/journal.pcbi.1006630</pub-id>
</citation>
</ref>
<ref id="B59">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Schaff</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Fink</surname>
<given-names>C. C.</given-names>
</name>
<name>
<surname>Slepchenko</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Carson</surname>
<given-names>J. H.</given-names>
</name>
<name>
<surname>Loew</surname>
<given-names>L. M.</given-names>
</name>
</person-group> (<year>1997</year>). <article-title>A general computational framework for modeling cellular structure and function</article-title>. <source>Biophysical J.</source> <volume>73</volume>, <fpage>1135</fpage>&#x2013;<lpage>1146</lpage>. <pub-id pub-id-type="doi">10.1016/S0006-3495(97)78146-3</pub-id>
</citation>
</ref>
<ref id="B60">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Schnell</surname>
<given-names>S.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>&#x201c;Reproducible&#x201d; research in mathematical sciences requires changes in our peer review culture and modernization of our current publication approach</article-title>. <source>Bull. Math. Biol.</source> <volume>80</volume>, <fpage>3095</fpage>&#x2013;<lpage>3105</lpage>. <pub-id pub-id-type="doi">10.1007/s11538-018-0500-9</pub-id>
</citation>
</ref>
<ref id="B61">
<citation citation-type="web">
<person-group person-group-type="author">
<name>
<surname>Sethna</surname>
<given-names>J. P.</given-names>
</name>
</person-group> (<year>2008</year>). <article-title>Segment polarity model</article-title>. <comment>Available at: <ext-link ext-link-type="uri" xlink:href="https://sethna.lassp.cornell.edu/Sloppy/vonDassow/model.html">https://sethna.lassp.cornell.edu/Sloppy/vonDassow/model.html</ext-link>
</comment>.</citation>
</ref>
<ref id="B62">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Shaikh</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Marupilla</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Wilson</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Blinov</surname>
<given-names>M. L.</given-names>
</name>
<name>
<surname>Moraru</surname>
<given-names>I. I.</given-names>
</name>
<name>
<surname>Karr</surname>
<given-names>J. R.</given-names>
</name>
</person-group> (<year>2021</year>). <article-title>RunBioSimulations: An extensible web application that simulates a wide range of computational modeling frameworks, algorithms, and formats</article-title>. <source>Nucleic Acids Res.</source> <volume>49</volume>, <fpage>W597</fpage>&#x2013;<lpage>W602</lpage>. <pub-id pub-id-type="doi">10.1093/nar/gkab411</pub-id>
</citation>
</ref>
<ref id="B63">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Stodden</surname>
<given-names>V.</given-names>
</name>
<name>
<surname>McNutt</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Bailey</surname>
<given-names>D. H.</given-names>
</name>
<name>
<surname>Deelman</surname>
<given-names>E.</given-names>
</name>
<name>
<surname>Gil</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Hanson</surname>
<given-names>B.</given-names>
</name>
<etal/>
</person-group> (<year>2016</year>). <article-title>Enhancing reproducibility for computational methods</article-title>. <source>Science</source> <volume>354</volume>, <fpage>1240</fpage>&#x2013;<lpage>1241</lpage>. <pub-id pub-id-type="doi">10.1126/science.aah6168</pub-id>
</citation>
</ref>
<ref id="B64">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Stodden</surname>
<given-names>V.</given-names>
</name>
<name>
<surname>Seiler</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Ma</surname>
<given-names>Z.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>An empirical analysis of journal policy effectiveness for computational reproducibility</article-title>. <source>Proc. Natl. Acad. Sci. U. S. A.</source> <volume>115</volume>, <fpage>2584</fpage>&#x2013;<lpage>2589</lpage>. <pub-id pub-id-type="doi">10.1073/pnas.1708290115</pub-id>
</citation>
</ref>
<ref id="B65">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Tegner</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Yeung</surname>
<given-names>M. K. S.</given-names>
</name>
<name>
<surname>Hasty</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Collins</surname>
<given-names>J. J.</given-names>
</name>
</person-group> (<year>2003</year>). <article-title>Reverse engineering gene networks: Integrating genetic perturbations with dynamical modeling</article-title>. <source>Proc. Natl. Acad. Sci. U. S. A.</source> <volume>100</volume>, <fpage>5944</fpage>&#x2013;<lpage>5949</lpage>. <pub-id pub-id-type="doi">10.1073/pnas.0933416100</pub-id>
</citation>
</ref>
<ref id="B66">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Tiwari</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Kananathan</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Roberts</surname>
<given-names>M. G.</given-names>
</name>
<name>
<surname>Meyer</surname>
<given-names>J. P.</given-names>
</name>
<name>
<surname>Sharif Shohan</surname>
<given-names>M. U.</given-names>
</name>
<name>
<surname>Xavier</surname>
<given-names>A.</given-names>
</name>
<etal/>
</person-group> (<year>2021</year>). <article-title>Reproducibility in systems biology modelling</article-title>. <source>Mol. Syst. Biol.</source> <volume>17</volume>, <fpage>e9982</fpage>. <pub-id pub-id-type="doi">10.15252/msb.20209982</pub-id>
</citation>
</ref>
<ref id="B67">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Tour&#xe9;</surname>
<given-names>V.</given-names>
</name>
<name>
<surname>Dr&#xe4;ger</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Luna</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Dogrusoz</surname>
<given-names>U.</given-names>
</name>
<name>
<surname>Rougny</surname>
<given-names>A.</given-names>
</name>
</person-group> (<year>2021</year>). <source>The systems biology graphical notation: Current status and applications in systems medicine</source>. <publisher-loc>Oxford</publisher-loc>: <publisher-name>Academic Press</publisher-name>, <fpage>372</fpage>&#x2013;<lpage>381</lpage>. <pub-id pub-id-type="doi">10.1016/B978-0-12-801238-3.11515-6</pub-id>
</citation>
</ref>
<ref id="B68">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Turing</surname>
<given-names>A. M.</given-names>
</name>
</person-group> (<year>1952</year>). <article-title>The chemical basis of morphogenesis</article-title>. <source>Philosophical Trans. R. Soc. Lond. Ser. B, Biol. Sci.</source> <volume>237</volume>, <fpage>37</fpage>&#x2013;<lpage>72</lpage>. <pub-id pub-id-type="doi">10.1098/rstb.1952.0012</pub-id>
</citation>
</ref>
<ref id="B69">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>von Dassow</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Odell</surname>
<given-names>G. M.</given-names>
</name>
</person-group> (<year>2002</year>). <article-title>Design and constraints of the Drosophila segment polarity module: Robust spatial patterning emerges from intertwined cell state switches</article-title>. <source>J. Exp. Zoology</source> <volume>294</volume>, <fpage>179</fpage>&#x2013;<lpage>215</lpage>. <pub-id pub-id-type="doi">10.1002/jez.10144</pub-id>
</citation>
</ref>
<ref id="B70">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>von Dassow</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Meir</surname>
<given-names>E.</given-names>
</name>
<name>
<surname>Munro</surname>
<given-names>E. M.</given-names>
</name>
<name>
<surname>Odell</surname>
<given-names>G. M.</given-names>
</name>
</person-group> (<year>2000</year>). <article-title>The segment polarity network is a robust developmental module</article-title>. <source>Nature</source> <volume>406</volume>, <fpage>188</fpage>&#x2013;<lpage>192</lpage>. <pub-id pub-id-type="doi">10.1038/35018085</pub-id>
</citation>
</ref>
<ref id="B71">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Waltemath</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Wolkenhauer</surname>
<given-names>O.</given-names>
</name>
</person-group> (<year>2016</year>). <article-title>How modeling standards, software, and initiatives support reproducibility in systems biology and systems medicine</article-title>. <source>IEEE Trans. Bio-Medical Eng.</source> <volume>63</volume>, <fpage>1999</fpage>&#x2013;<lpage>2006</lpage>. <pub-id pub-id-type="doi">10.1109/TBME.2016.2555481</pub-id>
</citation>
</ref>
<ref id="B72">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Waltemath</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Adams</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Beard</surname>
<given-names>D. A.</given-names>
</name>
<name>
<surname>Bergmann</surname>
<given-names>F. T.</given-names>
</name>
<name>
<surname>Bhalla</surname>
<given-names>U. S.</given-names>
</name>
<name>
<surname>Britten</surname>
<given-names>R.</given-names>
</name>
<etal/>
</person-group> (<year>2011a</year>). <article-title>Minimum information about a simulation experiment (MIASE)</article-title>. <source>PLoS Comput. Biol.</source> <volume>7</volume>, <fpage>e1001122</fpage>. <pub-id pub-id-type="doi">10.1371/journal.pcbi.1001122</pub-id>
</citation>
</ref>
<ref id="B73">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Waltemath</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Adams</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Bergmann</surname>
<given-names>F. T.</given-names>
</name>
<name>
<surname>Hucka</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Kolpakov</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Miller</surname>
<given-names>A. K.</given-names>
</name>
<etal/>
</person-group> (<year>2011b</year>). <article-title>Reproducible computational biology experiments with SED-ML&#x2014;The simulation experiment description markup language</article-title>. <source>BMC Syst. Biol.</source> <volume>5</volume>, <fpage>198</fpage>. <pub-id pub-id-type="doi">10.1186/1752-0509-5-198</pub-id>
</citation>
</ref>
<ref id="B74">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Waskom</surname>
<given-names>M. L.</given-names>
</name>
</person-group> (<year>2021</year>). <article-title>seaborn: statistical data visualization</article-title>. <source>J. Open Source Softw.</source> <volume>6</volume>, <fpage>3021</fpage>. <pub-id pub-id-type="doi">10.21105/joss.03021</pub-id>
</citation>
</ref>
<ref id="B75">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Welsh</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Xu</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Smith</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>K&#xf6;nig</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Choi</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Sauro</surname>
<given-names>H. M.</given-names>
</name>
</person-group> (<year>2023</year>). <article-title>libRoadRunner 2.0: a high performance SBML simulation and analysis library</article-title>. <source>Bioinformatics</source> <volume>39</volume>, <fpage>btac770</fpage>. <pub-id pub-id-type="doi">10.1093/bioinformatics/btac770</pub-id>
</citation>
</ref>
<ref id="B76">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Wilkinson</surname>
<given-names>M. D.</given-names>
</name>
<name>
<surname>Dumontier</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Aalbersberg</surname>
<given-names>I. J. J.</given-names>
</name>
<name>
<surname>Appleton</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Axton</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Baak</surname>
<given-names>A.</given-names>
</name>
<etal/>
</person-group> (<year>2016</year>). <article-title>The FAIR guiding principles for scientific data management and stewardship</article-title>. <source>Sci. Data</source> <volume>3</volume>, <fpage>160018</fpage>. <pub-id pub-id-type="doi">10.1038/sdata.2016.18</pub-id>
</citation>
</ref>
<ref id="B77">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Williams</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Kelley</surname>
<given-names>C.</given-names>
</name>
</person-group> (<year>2022</year>). <source>Gnuplot 5.4.3: An interactive plotting program</source>. <comment>Available at: <ext-link ext-link-type="uri" xlink:href="http://www.gnuplot.info/">http://www.gnuplot.info/</ext-link>
</comment>.</citation>
</ref>
<ref id="B78">
<citation citation-type="journal">
<collab>Yale Law School Roundtable Participants</collab> (<year>2010</year>). <article-title>Reproducible research</article-title>. <source>Comput. Sci. Eng.</source> <volume>12</volume>, <fpage>8</fpage>&#x2013;<lpage>13</lpage>. <pub-id pub-id-type="doi">10.1109/MCSE.2010.113</pub-id>
</citation>
</ref>
<ref id="B79">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yu</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Lloyd</surname>
<given-names>C. M.</given-names>
</name>
<name>
<surname>Nickerson</surname>
<given-names>D. P.</given-names>
</name>
<name>
<surname>Cooling</surname>
<given-names>M. T.</given-names>
</name>
<name>
<surname>Miller</surname>
<given-names>A. K.</given-names>
</name>
<name>
<surname>Garny</surname>
<given-names>A.</given-names>
</name>
<etal/>
</person-group> (<year>2011</year>). <article-title>The physiome model repository 2</article-title>. <source>Bioinformatics</source> <volume>27</volume>, <fpage>743</fpage>&#x2013;<lpage>744</lpage>. <pub-id pub-id-type="doi">10.1093/bioinformatics/btq723</pub-id>
</citation>
</ref>
<ref id="B80">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Za&#xf1;udo</surname>
<given-names>J. G. T.</given-names>
</name>
<name>
<surname>Yang</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Albert</surname>
<given-names>R.</given-names>
</name>
</person-group> (<year>2017</year>). <article-title>Structure-based control of complex networks with nonlinear dynamics</article-title>. <source>Proc. Natl. Acad. U. S. A.</source> <volume>114</volume>, <fpage>7234</fpage>&#x2013;<lpage>7239</lpage>. <pub-id pub-id-type="doi">10.1073/pnas.1617387114</pub-id>
</citation>
</ref>
</ref-list>
</back>
</article>