To illustrate clinical heterogeneity we looked at three elements of study design; (i) outcomes measured, (ii) samples collected, and (iii) assays employed, then further characterized elements into sub-elements and identified use of each across all studies. Overall, the body of research was complex applying different combinations of samples collected, assays used, and outcomes measured, both within and across studies.

We used methods of network analyses to generate displays showing the scope and relatedness of each of the three elements as applied across all studies (N = 16). Figure 1 and Supplementary Figures S1, S2, display the number of studies jointly including the same sub-elements of outcomes measured, samples collected, and assays used across all studies. The Delphi method for reaching consensus is a preferred approach for developing a COS and this type of visual can, at the onset of the process, help Delphi study participants to understand the extent of existing clinical heterogeneity (9, 12, 21, 26).

All studies included as outcomes measures of immunoglobulin responses and of virus detection; inclusion of other outcomes was variable (see Table 1). Samples collected and assays employed to measure each of the six outcomes varied across all 16 studies and are summarized in Supplementary Figures S3, S4, respectively.

Nasal swabs were used to collect samples for virus detection in 14/16 studies (Supplementary Figure S3). Three approaches were taken to demonstrate direct evidence of virus in piglets post challenge (Supplementary Figure S4); (1) culture of live virus (n = 13), (2) detection of viral RNA nucleic acids using polymerase chain reaction (PCR) methods (n = 6), and (3) histological identification in situ of viral antigen in sectioned tissues [immunohistochemistry (IHC)] (n = 5). Virus was quantified using cell culture methods (n = 10) and RT-qPCR methods (n = 4).

Serum immunoglobulin titres were measured in all but 1 study using enzyme linked immunosorbent assays (ELISA) (n = 15) (Supplementary Figure S4). ELISAs also varied with respect to the type of immunoglobulins measured; as IgG and /or IgA isotypes or as immunoglobulins specifically against conserved viral proteins NP (nucleoprotein) or M (matrix protein). The HAI assay, which is the gold standard correlate of protection, was used in most studies (n = 13) to identify sub-type and strain specific immunologic responses to influenza A viruses of swine.

Although few studies included outcomes measuring cell mediated immune responses (CMI), the assays used were varied; multiparameter flow cytometry (n = 6) was used to assess T-cell proliferation (staining for detection of CD4+, CD8+, CD3+, γδ TCR+ peripheral blood mononuclear (PBMN) cell populations) and T-cell priming (staining for CD25, IFN-gamma, and IL-10) (Supplementary Figure S4). ELISA based assays including the enzyme-linked immunospot (ELISpot) assay (n = 1), used for detection of IFN-gamma secreting cells, and a multiplex ELISA (n = 2) used to detect cytokine production. Clinical signs were measured as a stand-alone measure or as part of a composite score in 12 of 16 studies and are summarized in Supplementary Figure S5. Fever was the most consistently measured and reported clinical sign.

Incomplete reporting of study methods and results hinders assessment for internal and external validity (27, 28), and in veterinary medicine is a cause of frequent exclusion of primary research studies from systematic reviews and meta-analyses (18, 29–34). In our review, studies were excluded from sub-group meta-analyses if hemagglutinin and neuraminidase antigens in vaccines were poorly characterized (23). This further diminished the value of inferences derived through synthesis. Therefore, we encourage funders and journal editors require research protocols and manuscripts adhere to established reporting guidelines such as REFLECT (available on the Meridian Network at https://meridian-network.org/).

Additionally, we developed a reporting checklist specific to influenza vaccine challenge trails in swine (see Supplementary Tables S1, S2) and encourage researchers to complete and include the checklist as a supplemental document with all manuscript submissions. The checklist itemizes sub-elements of study design important for overall contextual understanding of influenza vaccine research (i.e., outcomes, samples and assays). The intent is that it serves as a reminder to researchers of essential items to report (35), and as an aid for synthesis researchers when identifying studies and extracting data for inclusion in systematic reviews and meta-analyses (31, 36). Consistent reporting of outcome, assay and sample sub-elements may also help to improve researcher, reviewer, and reader awareness of the full extent of avoidable clinical heterogeneity (8, 9).

Vaccine protection is a non-specific term and context is important for interpretation; protection from infection must be distinguished from protection against an undesired clinical endpoint (37) and no single outcome conveys protection against infection or disease (38). From a societal perspective, desired endpoints are conditioned on stakeholders’ need to inform decisions on implementation of vaccines in the field. For researchers involved in vaccine development work, desired endpoints are a function of the phase of research for which they are engaged (39, 40). Examples of researcher objectives, societal perspectives, and corresponding vaccine research endpoints and measures of protection against in influenza A viruses in swine (IAV-S) are listed in Supplementary Table S3.

In human influenza vaccine research, researchers’ objectives differ by each distinct phase of research (see Supplementary Table S3; Supplementary Figure S6) (38, 41–43). Phase “0″ is the proof-of-concept phase for candidate vaccines. In Phase I to III clinical trials, convenient methods and assays to measure efficacy are validated, safety, dosages, and schedules are established, and approval for market use is established. In Phase IV clinical trials, post marketing evaluation of field effectiveness is established. Compared to human vaccines, approval of veterinary vaccines is expedited due to:

Desired endpoints overlap in early phase and late phases of vaccine research (Supplementary Table S3) (38). Validating correlates of protection (CoP), specifically immune CoPs, is often done using challenge trials in animal models, where same-species challenge models are optimal (47–49). Historically, hemagglutination inhibition (HAI) serologic assays have been the gold standard CoP for evaluating efficacy of inactivated vaccines against influenza A virus (IAV) (50), however because understanding of immune responses to influenza is incomplete, multiple outcomes must be assessed to accurately predict vaccine performance (37, 51). Additionally, use of live vaccines and newer vaccine technologies means both cell-mediated CoPs and direct measures of virus infection are needed to assess protection (50, 52–54).

There are gaps in understanding of how a host’s immune responses are coordinated to eliminate virus following primary infections, and of the impact of prior or original virus exposure on immunologic responses to secondary strain-homologous and to strain-heterologous IAV exposures (55). Universal vaccines and novel vaccine platforms typically elicit immunologic responses to non-dominate epitopes (i.e., the dominant HA receptor binding sites are not targeted) and as emphasis shifts to the design of such vaccines, so too will study of unintended adverse outcomes such as vaccine enhanced acute respiratory disease (VEARD) (56–61).

Although rare in human medicine (45, 62), challenge trials were the most frequently reported study design in IAV-S vaccine research in swine published since 1990 (25). For each of the 16 MDI challenge studies, the author’s stated objective(s) and study endpoints are summarized in Supplementary Table S4, indicating also where objectives correspond to one or more of the five critical R&D immunologic issues outlined in The Influenza Vaccine Research and Development (R&D) Roadmap (51).

Within a One Health context., IAV-S vaccine challenge research in swine contributes to the larger influenza research community of practice, and in this light, development of a COS for IAV-S research in swine (using a Delphi process) may serve as a template for development of COS for influenza vaccine research in other animal species (63, 64). Similarly, adoption of the proposed checklist may also serve as a multi-species template for consistent reporting of essential elements important for research synthesis and ultimately for the interpretation of vaccine challenge trial studies in animals.