We drew from past iGEM projects to ensure that we were working with realistic examples of DURC risks in synthetic biology. We ran a pilot survey with a separate pool of experts to select projects that posed sufficient DURC concern to merit review, but that were varied enough that we expected them to be assessed differently from one another (see Supplementary Material). Under current United States policy none of these projects would be required to undergo a formal review of their DURC potential, as they do not involve any listed agents, toxins, or experimental outcomes of concern. However, current policy explicitly notes that some DURC may not involve the listed agents, toxins, or outcomes, and encourages PIs and research institutions to “remain vigilant” and manage risks appropriately (Office of Science and Technology Policy, 2024b, p. 12). Our findings are therefore most relevant to potential DURC that evades current list-based policy frameworks (Millett et al., 2023). A description of each of the four projects used in this study is provided in Table 1.

For each of the four projects we created a 4-8 page packet of background material for reviewers, similar to what project leads might provide in a real scenario (see Supplementary Material). The packets were condensed and anonymized versions of team’s responses to the iGEM safety form, a self-assessment of risk that every iGEM team is required to complete for their project midway through its development. We did not include information about iGEM’s original assessments of project risks or their risk management efforts.

We derived a DURC risk assessment form largely from the United States Government’s 2014 DURC policy companion guide for academic institutions (National Institutes of Health, 2014). We chose this guide because it offers a detailed set of questions for assessing the risks and benefits of a project, provides guidance for developing a risk management plan, and is likely widely used by IREs. Other DURC risk assessment forms are focused more narrowly on risks or are intended to assess biotechnologies more broadly rather than individual projects (National Academies of Sciences Engineering and Medicine, 2018; Tucker, 2012; Vennis et al., 2021).

Three years after we collected our data in 2021, the United States Government updated the 2014 DURC policy (Office of Science and Technology Policy, 2014c), integrating it with an updated framework for managing risks of enhanced potential pandemic pathogens (ePPPs) and issuing a new companion guide (Office of Science and Technology Policy, 2024a). However, the methods and results of this study remain applicable to DURC management today. Both the old and new policies would not require IREs to review the projects in this study because they do not involve the required biological agents, toxins, or experimental outcomes, but both policies recognize the potential for such projects to be DURC. Both policies also require IREs to evaluate projects for DURC and have functionally similar guidelines for forming and operating an IRE. Finally, the companion guides for both policies provide very similar guidance on how to assess risks and benefits and select risk mitigation strategies, directly reusing sections of text and covering similar subheadings (e.g., type of misuse, ease of misuse, and potential consequences) (National Institutes of Health, 2014; Office of Science and Technology Policy, 2024a).

In our study, the risk assessment process for a single project first involved participants reading the background material about the project and answering questions about their initial reactions (Students were also asked to provide an initial judgment of the overall DURC risk of the project; see Results for more information.) All participants then completed a series of reflection questions adapted from the DURC policy companion guide about the nature, magnitude, and likelihood of potential harms and benefits of the project, including both open-ended and Likert-scale questions. They were also carefully guided through the United States Government’s definition of DURC. After completing the reflection questions, they provided summative 5-point ratings of the overall perceived DURC risk and overall perceived benefits of the project.

Finally, we asked participants whether they would recommend applying any of eight separate risk management strategies to the project (Table 2). Again, these strategies were derived from those suggested in the DURC policy companion guide and adapted to the iGEM context (e.g., “more monitoring from iGEM leadership” rather than “more monitoring from the United States Government”). For each strategy, participants could select one of four options: “not necessary”, “doesn’t seem necessary but the team could consider it”, “seriously consider”, or “the team should not continue without this strategy”. To simplify analysis, we labeled the former two options as “non-endorsement” of the strategy and the latter two as “endorsement”. All surveys and assessment forms are included in the Supplementary Material.

49 students and 18 experts completed ratings of one or more projects. Details on participant recruitment, data cleaning, and our statistical approach are provided in the Supplementary Material. All participants could choose to assess up to four projects, and students and experts each assessed approximately the same number of projects (student mean = 2.35 projects, expert mean = 2.00 projects, t = . 97 , d f = 32.05 , p = . 34 ). Each individual project was assessed by 8–11 experts and 27–30 students. Participants were equally likely to complete a risk assessment of any of the four projects ( χ 2 = 3.16 , d f = 3 , p = . 37 ).

Before assessing any projects, participants answered a set of questions about their background experience and beliefs and attitudes related to synthetic biology and biological risk. Their responses helped us to contextualize participants’ perspectives, characterize differences in perspective between experts and students, and confirm commonsense notions of “expertise” in terms of training and familiarity with biological risk.

As expected, experts reported more education, experience, and familiarity with synthetic biology and biological risk than did students. 16 of 18 experts held a Masters’ degree or greater, compared to 10 of 49 students. Experts were also more likely than students to report having multiple years of experience with wet-lab work (11 of 18 experts, 16 of 49 students) or with “broader aspects of synthetic biology (e.g., human practices, sociology of science, risk analysis)” (10 of 18 experts, 7 of 49 students), though the groups had comparable dry-lab experience (3 of 18 experts and 7 of 65 students reporting “years” of experience). Finally, on a 1–5 scale, experts reported significantly more prior familiarity with the concept of DURC (expert mean = 4.29, student mean = 2.28, t = 9.03 , d f = 44.08 , p < . 001 ).

Experts and students also differed in their reported perspectives on DURC risks (Figure 1) and risk management (Figure 2) in the field of synthetic biology. We asked participants whether synthetic biology researchers are generally “too cautious” or “too careless” about the risks of their work. No experts reported that synthetic biology researchers are “too cautious”, but 29% of students did (1–7 scale, expert mean = 2.82, student mean = 4.11, t = − 5.23 , d f = 56.40 , p < . 001 ). In addition, experts and students did not significantly differ in their mean estimates of the percentage of current synthetic biology research that poses DURC potential (expert mean = 45.82, student mean = 47.74, t = − 0.24 , d f = 21.61 , p < . 81 ), but experts had far more variance in their estimates, which ranged from 3% to 100% (expert σ 2 = 930.28 , student σ 2 = 544.96 , permutation test p < . 001 ). Taken together, these results suggest that while the experts in our sample recommend care and caution in synthetic biology, they differ substantially from one another in their perceptions of the scale of potential DURC.

After reading each project description, participants rated their understanding of the project and their familiarity with its topic and its methods on 5-point scales. There were no statistically significant differences between expert and student groups on any of these dimensions. After providing summative 5-point ratings of the overall risk and benefits of the project (discussed below), participants also rated their confidence in their own ratings on a 5-point scale. On all of these measures there were also no statistically significant differences between experts and students in omnibus tests or on a per-project basis (all p > . 1 ). This suggests that while experts and students had different levels of background education, attitudes, and familiarity with DURC, they did not significantly differ in their self-assessed familiarity with or confidence in assessments of the specific projects presented in this study.

After rating the magnitude and likelihood of potential harm for each project, participants were asked to rate its overall DURC risk on a 5-point scale, following the United States Government definition of DURC: “Overall, combining your judgments of the potential harms of misuse with the likelihood of misuse … How much risk do you think that the knowledge, information, technology, or products from this project pose to public health and safety, agricultural crops and other plants, animals, the environment, materiel, or national or global security?”

We found that risk ratings for most projects varied widely, with two projects spanning 4 points of the 5-point rating scale even within the small expert group (Figure 3). In other words, expert participants disagreed substantially about the perceived risk of several projects. As a result, the mean risk ratings of three out of the four projects were also statistically indistinguishable from one another ( F ( 2,70 ) = 0.474 , p = . 625 ). The only standout was the “Immune escape” project, whose mean risk rating was marginally or significantly lower than that of the other three projects in pairwise comparisons (with “Insect vectors”: p = . 005 , 95% C.I. = [−1.78, −0.23]; with “Non-canonical AAs”: p = . 06 , 95% C.I. = [−1.60, 0.01]; with “Psilocybin”: p = . 002 , 95% C.I. = [−1.87, −0.31]).

We also examined expert-student differences in risk ratings. As shown in Figure 1, experts and students held a similar average perception of the percentage of synthetic biology research with DURC potential. However, experts rated our selection of projects as about 0.6 scale points less risky on average than did students on a 1-5 scale (expert mean = 2.32, student mean = 2.93, t = − 2.72 , d f = 44 , p = . 0097 ). Examining individual projects once again, we found that differences were largely due to the psilocybin project (expert mean = 1.80, student mean = 3.50, t = − 3.95 , d f = 6.82 , p = . 006 ) and the immune-escape project, though the latter difference was not statistically significant (expert mean = 1.60, student mean = 2.32, t = − 1.93 , d f = 26.93 , p = . 06 ).

We also hypothesized that experts would have less variance than students in their risk ratings because they might draw from a common body of knowledge and experience to arrive at more similar conclusions to each other. However, we did not find evidence that experts’ project risk ratings have any less variance than students’ ratings using an omnibus permutation test across all projects ( p = . 51 ) or subsetting the data to individual projects (all p > . 1 ).

We hypothesized that someone who was familiar with or particularly excited about a project’s topic might also be likely to view it as being less risky. To test this hypothesis, we asked participants how excited they felt about each project, how familiar they were with its topic and methods, the magnitude of the project’s potential benefits, the likelihood that the benefits would be realized, and the overall expected benefits of the project. None of these factors were significantly correlated with a project’s overall DURC risk rating (all p > . 1 ).

We also compared experts and students’ ratings of the overall expected benefits of the projects. On average, experts judged the four projects in this study as being 0.53 points less beneficial than did students on a 5-point scale, a marginally significant result (expert mean = 2.50, student mean = 3.03, t = 2.38 , χ 2 = 3.11 , p = 0.078 ).

We found substantial disagreement between expert participants about the appropriate actions to take to manage the DURC risks of research projects. On average, two randomly selected experts who evaluated the same project only agreed on whether to endorse about five out of eight possible strategies (60%). Experts typically did not endorse modifying or stopping projects and were therefore much more likely to agree about those strategies (81%) than about other, less disruptive strategies (53%; see Figure 4). Still, 2 out of 8 experts endorsed stopping “Insect vectors” and 1 out of 8 endorsed stopping “Psilocybin”. Expert agreement also varied depending on the project under consideration, from 50% of strategies for “Insect vectors” to 70% for “Psilocybin”.

We also observed differences between students and experts in the number and type of strategies that they endorsed (Figure 4). Despite more frequently reporting that synthetic biologists are too cautious in general, students endorsed a larger number of strategies per project on average than did experts (expert mean = 2.43, student mean = 3.61, χ 2 = 4.60 , p = 0.032 ). In particular, across every project a greater percentage of students than experts endorsed the strategies of “More training” and “Study countermeasures.” Students and experts also appeared to disagree strongly about the “Modify project” strategy for the non-canonical amino acids project, and about the “More training” and “Improve biosafety” strategies for the psilocybin project.

Overall, the four projects did not vary in their average number of endorsed risk management strategies ( F ( 3,96 ) = 1.52 , p = . 21 ), though an individual’s rating of a project’s DURC risk was highly correlated with the number of strategies that they endorsed ( r = . 49 , p < . 001 ). Across all projects, some risk management strategies were also far more likely to be endorsed than others ( F ( 7,792 ) = 14.21 , p < . 001 ). “Study countermeasures” and “Manage communications” were the most popular (60 and 58 endorsements respectively out of 100 total evaluations), and “Modify project” and “Stop project” were the least popular (22 and 11 endorsements respectively). The most and least popular strategies also varied from project to project, suggesting that reviewers were sensitive to the specific features of each project. For example, both experts and students were relatively likely to endorse “Improve biosecurity” for the psilocybin project (presumably to prevent the theft of illegal drugs), but not for other projects.

Student participants were asked to rate the DURC risk of each project twice - once immediately after reading about the project and a second time after completing the reflection questions based on the United States Government’s DURC policy companion guide. We hypothesized that the process of completing the reflection questions might cause students to shift their initial risk estimates. For example, the questions might help students to realize their own degree of uncertainty about the potential benefits and harms of a project, a phenomenon known in general as the illusion of explanatory depth (Fernbach et al., 2013; Rozenblit and Keil, 2002). However, in paired t-tests, we found that students’ post-reflection risk ratings were statistically indistinguishable from their pre-reflection ratings across all four projects ( t = − 0.21 , d f = 73 , p = 0.83 ) and for each individual project (all p > . 1 ).

The observation that experts disagree about DURC risk management is concerning because it suggests that the public is being exposed to inconsistent standards of risk tolerance regarding potential DURC. A central premise of risk management is that decisions should be aligned with the risk tolerance of all stakeholders exposed to the risk. For example, air safety regulators should take the risk tolerance of passengers into account when deciding how much to spend on mitigating risks of plane crashes (Aven, 2015). By definition, DURC projects have the potential for broad effects, both positive and negative, that could affect large fractions of society no matter where they are carried out. Therefore, DURC management decisions should arguably be aligned with the risk tolerance of the public. ¹

But DURC reviewers disagree substantially about the risks and appropriate management strategies for the projects in our study, and about the fraction of synthetic biology research with DURC potential. This suggests that reviewers at different research institutions likely apply different standards of risk tolerance to the projects that they see, and these standards cannot all be aligned with the public’s risk tolerance at the same time. Therefore, some reviewers are likely making decisions about potential DURC that do not align with the public’s risk tolerance.

Diversity of perspective is a key reason why groups are commonly used in DURC review. Ideologically diverse groups are more likely to arrive at a position that is both higher quality and more consistent with the independent opinions of other such groups (McBrayer, 2024). Ideally, IREs should be composed of individuals who will collectively raise all perspectives on a given project that are worthy of serious consideration.

However, our findings suggest that IREs may sometimes lack perspectives that are common enough to plausibly deserve serious consideration, but also rare enough to be left out purely by chance. Our reasoning is as follows.

For two out of the four projects we studied (“Non-canonical AAs” and “Immune escape”), experts were in full agreement to continue the project, and we will not consider them further. For the other two projects, 2 out of 8 experts (25%) endorsed stopping “Insect vectors” and 1 out of 8 (12.5%) endorsed stopping “Psilocybin”. While we lack ground truth about whether stopping these projects is worthy of serious consideration, it seems plausible, given that they reflect more than 10% of expert opinion in each case and that the costs of a mistake may be high.

If we treat the opinions in this sample as representative of a larger population of experts, how likely is it that a randomly selected group of five reviewers (the minimum size required for an IRE by the United States Government) (Office of Science and Technology Policy, 2024b) contains no one who would wish to stop each project? For “Insect vectors”, the estimate is ( 1 − 2 / 8 ) 5 = 23.7 % . For “Psilocybin”, the probability is ( 1 − 1 / 8 ) 5 = 51.3 % . In both cases, there is a non-trivial probability that a five-person IRE would fail to include anyone who believes that the project should be stopped. Furthermore, if an IRE lacks an initial dissenting perspective, group polarization (whereby a group that is already in agreement further cements its agreement through discussion) would make it less likely that any individual, much less an entire group, would change its position (Sunstein, 1999) (This phenomenon is consistent with anecdotal reports from institutions that their IREs approve nearly all projects (Bowman et al., 2020)).

Our study may overestimate the variance in experts’ risk ratings compared to real-world DURC risk assessments for at least three reasons. First, as we have noted above, the projects in this study were assessed by individuals rather than groups, but previous expert recommendations and federal guidelines posit that DURC risk assessment should involve multiple reviewers with diverse backgrounds in discussion with one another. Groups tend to outperform individuals on a wide range of decisions, including forecasting decisions (Kerr and Tindale, 2004; Tindale and Winget, 2019). We expect that group assessments of DURC risk would have less variance than individual assessments, on average, because groups can collectively access and integrate a wider range of perspectives and evidence. For example, groups might be more likely than individuals to know about the capabilities of malicious actors, the base rates of biotechnology project success, or technical methods of dispersing pathogens.

Second, we sought out example iGEM projects that had raised concern in the past, but the rules of the iGEM competition exclude a great deal of work that could be clearly judged as high-risk, such as work with pathogens that would require BSL-3 or BSL-4 precautions. As a result, the projects in this study were likely to be neither obviously safe nor obviously dangerous. Our findings may therefore be most helpful for gauging the level of current consensus about particularly high-variance “edge cases” that can fall outside of typical guidance frameworks and are most in need of expert review (Evans and Palmer, 2018). Future empirical work should attempt to include clear low- or high-risk projects as benchmarks for comparison. Our personal opinion is that the large majority of synthetic biology research projects have little to no DURC potential, but the findings of this study suggest that a substantial fraction of iGEM students and expert reviewers may disagree. More work is essential to clarify and resolve this disagreement.

Third, our study may overestimate the real-world variance in risk ratings because it did not give reviewers the opportunity to ask research proposers for more information about their projects, as is recommended by the United States Government companion guide for DURC assessment (Office of Science and Technology Policy, 2024b). Several reviewers in our study wrote in open responses that they found it difficult to assess the risks of some projects because they felt that they lacked sufficient information.

On the other hand, our study also simplified reviewers’ opinions about risk management strategies in a way that could have reduced variance in opinion. We converted opinions from a four-point scale of “not necessary”, “doesn’t seem necessary but the team could consider it”, “seriously consider”, or “the team should not continue without this strategy” into a binary variable of endorsement vs non-endorsement. While this simplification allows for a much more straightforward analysis, it may mask important differences of opinion. The most extreme scale points more clearly represent the reviewers’ opinions about whether to use a given strategy, while the middle two points are less clear. Future studies, or reanalyses of the data in this study, could reveal important disagreements about risk management.

In addition, geographical factors could have contributed to differences in opinion between experts and students. All 18 experts were from the United States, Canada, or Europe, while the students were from all inhabited continents except Australia. It is possible that the students in this study were less capable than the experts at understanding the English-language written materials, though both groups expressed similar levels of understanding of each project. It is also possible that the students held a different or wider range of cultural expectations regarding risk management than experts, though the experts and students had similar levels of variance in their final judgments of project risk.

What should be done if independent assessments of potential DURC, performed with groups and given the opportunity for researcher feedback, could still plausibly come to radically different conclusions about how to manage risks? We offer several suggestions below, divided into three categories: improving group deliberation, aligning standards for risk tolerance, and reducing the costs of risk management.