Sensitivity coefficients are the key ingredients for the GLLS methodology, as they are used to relate the response variations to the model parameter variations, with the latter assumed to represent the dominant sources of uncertainties. A sensitivity coefficient measures the first-order variation of a response that is caused by a change in one input model parameter. For the numerical experiments employed in this paper, we focus on the multiplication factor, k _eff , i.e., critical eigenvalue, as the response of interest and the reaction-wise cross-sections by isotope and energy group as the model parameters.

While sensitivity coefficients can be readily evaluated using finite differencing, the adjoint-based perturbation theory methodology (Usachev, 1964; Gandini, 1967; Stacey, 1974; Oblow, 1976; Cacuci and Ionescu-Bujor, 2010) has been adopted as the most efficient way to calculate derivatives. This follows because adjoint-based sensitivity theory requires one adjoint solution per response, implying that one can calculate the first-order derivatives of the given response with respect to all cross-sections using a single adjoint model evaluation, whereas finite differencing requires an additional forward model evaluation for each cross-section. The general idea behind adjoint-based sensitivity analysis is summarized below for the evaluation of the first order derivatives of the critical eigenvalue.

The Boltzmann transport equation for an assembly containing fissionable material, referred to as the forward model, can be symbolically expressed as (Williams, Broadhead and Parks, 2001): ( M ( α ) − 1 k F ( α ) ) ψ = 0 (1) where

M ( α ) = Multigroup form of the Boltzmann loss operator.

F ( α ) = Multigroup form of the Boltzmann production operator.

ψ = ψ ( r , Ω , g ) = Multigroup angular flux. where ψ is a function of

r = position.

Ω = neutron moving direction.

g = energy group.

The two operators M ( α ) and F ( α ) are functions of the reference multi-group cross-section data which may be described by an n-dimensional vector, α = [ α 1 α 2 ⋯ α n ] T whose n components are the reaction-wise cross-sections by energy-group and isotope ⁵ . Thus, the solution of this equation yielding the eigenvalue may be compactly written as follows: k j = f j ( α ) where k j is the code-predicted eigenvalue ⁶ , i.e., k _eff , of jth critical experiment model and f j is a compact representation of the solution of Eq. 1 implying its dependence on α . Analytically, the first-order derivatives of k _eff with respect to each cross-section can be expressed in a relative sense—referred to as the sensitivity coefficients—using a first-order Taylor series expansion: S k j , α i = α i k j f j ( α 1 , … , α i + Δ α i , … , α n ) − f j ( α 1 , … , α i , … , α n ) Δ α i (2) and aggregated in a vector (referred to as the sensitivity vector or profile) with the superscript “T” representing vector/matrix transpose: s j = [ S k j , α 1 S k j , α 2 ⋯ S k j , α n ] T

Eq. 2 implies that a finite-difference-based sensitivity analysis would require n + 1 forward model executions, one with the reference cross-section values, and one additional execution per cross-section. For most practical neutronic problems, this is computationally infeasible because cross-sections often number in the 10⁴ to 10⁵ range.

The adjoint formulation of sensitivity coefficients may be described by the following equation: S k j α i = α i k j 〈 ψ ∗ ∂ ∂ α i ( 1 k F − M ) ψ 〉 〈 ψ ∗ F ψ 〉 (3)

The brackets represent an inner product operation corresponding to an integration over entire phase-space (e.g., energy groups, direction of neutron travel, and space) using the forward solution obtained from Eq. 1 and a new quantity, called the adjoint solution, obtained from: ( M ∗ − 1 k F ∗ ) ψ ∗ = 0 where M ∗ and F ∗ are the adjoint operators of the forward operators M and F , evaluated at the reference cross-section values. The ∂ ∂ α i term differentiates the operators M and F with respect to the ith cross-section, with the derivative taken around the reference cross-section values. Since the mathematical expressions for M and F are known, these changes can be analytically calculated. More importantly, they do not require re-execution of the forward or the adjoint model. The implication is that one can evaluate the derivatives with respect to all cross-sections using a single forward and a single adjoint model evaluation.

Several computer codes have embodied the adjoint methodology to calculate sensitivity coefficients (Lucius et al., 1981; Becker et al., 1982; Gerald; Rimpault et al., 2002). Of interest to us is the SCALE TSUNAMI methodology (Rearden and Jessee, 2016) which is used as a basis for the evaluation of the sensitivity coefficients for the GLLS-based TSURFER code, discussed in the next section.

As discussed earlier, the main goal of GLLS is to consolidate knowledge from computations and experiments. This is illustrated in Figure 1 using two representative PDFs describing the best available knowledge about the k _eff from the experiments (shown in red) and model predictions (blue). The spread of each PDF is taken as a measure of confidence. The confidence in the model predictions is determined by the propagated prior uncertainties, and the experiment’s confidence is tied to its measurement uncertainties. The GLLS methodology represents a disciplined mathematical approach to consolidate these two PDFs into one (yellow) that provides higher confidence for the calculated response as compared to the prior confidence from model predictions.

To achieve that, GLLS assumes that the uncertainties originate from the cross-sections. Therefore, it attempts to identify the optimal cross-section adjustments which minimize the discrepancies between measured and predicted responses. Based on the optimal adjustments, one can calculate the corresponding change in the application’s response, with the application representing the conditions for which no experimental values exist. The change in the code-calculated application response, i.e., from its prior value, is denoted by the application bias.

Considering that the analyst is interested in calculating the bias for the k _eff value for a given application, and there exist M available experiments, the corresponding prior values may be aggregated in a vector k ∈ ℝ M + 1 such that: k = [ k 1       k 2       ⋯       k M + 1 ] T where the last component is the prior value for the application k _eff . The corresponding measurements for the first M values are designated by another vector m ∈ ℝ M + 1 . In this formulation, the last element of m is set to the prior value of k _eff , assumed to have no corresponding experimental value.

The prior cross-section uncertainties are described by a multi-variable Gaussian PDF with a vector of means representing the reference multi-group cross-sections and a covariance matrix given by: C αα = [ c o v ( α 1 , α 1 ) c o v ( α 1 , α 2 ) ⋯ c o v ( α 1 , α n ) c o v ( α 2 , α 1 ) c o v ( α 2 , α 2 ) ⋯ c o v ( α 2 , α n ) ⋮ ⋮ ⋱ ⋮ c o v ( α n , α 1 ) c o v ( α n , α 2 ) ⋯ c o v ( α n , α n ) ] ∈ ℝ n × n

The adjusted cross-sections are calculated as the minimizer of the following minimization problem subject to the constraint k ′ ( α ′ ) = m ′ : α ∗ = argmin α ′ [ α ′ − α ] T C αα − 1 [ α ′ − α ] + [ m ′ − m ] T C mm − 1 [ m ′ − m ] (4) where C mm ∈ ℝ ( M + 1 ) × ( M + 1 ) is the covariance matrix for the measured k _eff . The constraint implies that the adjusted cross-sections α ′ will update the best-estimated values (the components of m ′ ) for all M experiments as well as the application. The last element of the vector m ′ is taken to represent the best-estimate for the application k _eff value, and the last component of m ′ − m is referred to as the application bias. Note that k j − m j represents the initial discrepancy between the prior code-calculated and measured k _eff values for the jth experiment, and m ′ − m represents the discrepancy after the cross-sections are adjusted.

The objective function in Eq. 4 may be re-written in terms of the calculated and adjusted k _eff values as: χ M 2 = [ k ′ − k ] T C kk − 1 [ k ′ − k ] + [ m ′ − m ] T C mm − 1 [ m ′ − m ] where χ M 2 is the M -degrees of freedom chi-square value describing the discrepancies between the prior and adjusted k _eff values. The C kk ∈ ℝ ( M + 1 ) × ( M + 1 ) matrix denotes the prior covariance matrix for the calculated k _eff values given by: C kk = S kα C αα S kα T (5) and S kα ∈ ℝ ( M + 1 ) × n matrix aggregates the sensitivity profiles for all M experiments and the application: S kα = [ S 1,1 S 1,2 ⋯ S 1 , i ⋯ S 1 , n S 2,1 S 2,2 ⋯ S 2 , i ⋯ S 2 , n ⋮ ⋮ ⋱ ⋮ ⋮ ⋮ S j , 1 S j , 2 ⋯ S j , i ⋯ S j , n ⋮ ⋮ ⋯ ⋮ ⋱ ⋮ S M , 1 S M , 2 ⋯ S M , i ⋯ S M , n S M + 1,1 S M + 1,2 ⋯ S M + 1 , i ⋯ S M + 1 , n ] where (j, i) element represents the relative sensitivity coefficient of the jth experiment [or application, ( M + 1 ) th row] with respect to the ith multi-group cross-section.

Assuming that the linearization of the constraint k ′ ( α ′ ) = m ′ is sufficiently accurate within the limitations of first-order sensitivity theory, the minimizer of the objective function in Eq. 4 may be given by: Δ k = − C kk ( C kk + C mm ) − 1 d where Δ k = k ′ − k and d ∈ ℝ M + 1 is the discrepancy vector, d = k − m

The posterior (i.e., post the consolidation of experimental and prior values) covariance matrix for the k _eff values is given by: C k ′ k ′ = C kk − C kk ( C kk + C mm ) − 1 C kk (6)

The diagonal elements of this matrix describe the confidence one has in the posterior k _eff values. The ( M + 1 ) th diagonal element of the C kk matrix ( C kk [ M + 1 , M + 1 ] ) measures the prior confidence one has in the calculated application k _eff value. If the experiments are relevant to the application, the posterior confidence, ( C k ′ k ′ [ M + 1 , M + 1 ] ) should provide higher confidence, i.e., C k ′ k ′ [ M + 1 , M + 1 ] < C kk [ M + 1 , M + 1 ] .

The definition of the similarity index c _k naturally appears in the GLLS formulation of the prior covariance matrix. Specifically, one can expand Eq. 5 as follows: C kk = [ s 1 T C αα s 1 s 1 T C αα s 2 ⋯ s 1 T C αα s M + 1 s 2 T C αα s 2 s 2 T C αα s 2 ⋯ s 2 T C αα s M + 1 ⋮ ⋮ ⋱ ⋮ s M + 1 T C αα s 1 s M + 1 T C αα s 2 ⋯ s M + 1 T C αα s M + 1 ] (7)

The diagonal entries of this matrix represent the uncertainty (in the units of variance) of the prior k _eff values and the off-diagonal entries are the correlations between these uncertainties. Ideally, one would want to maximize the correlations between the application and all experiments, described by the last row or last column of the matrix. A standardized form of this matrix may be obtained by multiplying it from both sides by the inverse of the square root of its diagonal elements to produce the matrix R ∈ ℝ ( M + 1 ) × ( M + 1 ) : R = [ 1 s 1 T C αα s 2 s 1 T C αα s 1 s 2 T C αα s 2 ⋯ s 1 T C αα s M + 1 s 1 T C αα s 1 s M + 1 T C αα s M + 1 s 2 T C αα s 1 s 2 T C αα s 2 s 1 T C αα s 1 1 ⋯ s 2 T C αα s M + 1 s 2 T C αα s 2 s M + 1 T C αα s M + 1 ⋮ ⋮ ⋱ ⋮ s M + 1 T C αα s 1 s M + 1 T C αα s M + 1   s 1 T C αα s 1 s M + 1 T C αα s 2 s M + 1 T C αα s M + 1 s 2 T C αα s 2 ⋯ 1 ]

In this representation, the value of any off-diagonal terms is standardized between − 1 .0 and 1 .0 which is equivalent to the definition of the standard correlation coefficient between two random variables. Appearing naturally in the GLLS formulation, these off-diagonal terms have been adopted as to measure experimental relevance. Specifically, the similarity index c _k between the application and the jth experiment is given by: c k ( k j , k M + 1 ) = s j T C αα s M + 1 s j T C αα s j s M + 1 T C αα s M + 1 (8)

This equation may be used to pre-calculate the similarities of all available experiments with respect to the given application. To achieve that one needs to calculate the corresponding sensitivity profiles for the experiments and the application which are readily calculated using the adjoint sensitivity theory presented in Section 2.1. In our work, the SCALE TSUNAMI code is employed to calculate the sensitivity profiles as well as the similarity indices.

Next, it is instructive to give a geometric interpretation of the similarity index. To do that, rewrite the expressions in Eq. 8 using the Cholesky decomposition of C kk as: C kk = Γ T Γ where Γ = [ γ 1 γ 2 ⋯ γ M + 1 ] , where the inner-product of any of two columns of Γ gives: γ i T γ j = C kk [ i , j ] , i , j = 1,2 , … ,   M + 1

Performing this transformation for both the numerator and denominator in Eq. 8, the c _k index reduces to: c k ( k j , k M + 1 ) = γ j T γ M + 1 γ j T γ j γ M + 1 T γ M + 1 = C kk [ j , M + 1 ] C kk [ j , j ] C kk [ M + 1 , M + 1 ] = cos ⁡ θ ( γ j , γ M + 1 ) (9)

As illustrated in Figure 2, this expression is interpreted as cosine angle between two vectors, one related to the application and the other to an experiment.

Further, it has been shown in earlier research (Huang et al., 2020) that one can calculate c _k using randomized forward model evaluations taking advantage of the equivalence between the c _k definition and the standard correlation coefficient. This requires sampling of the cross-sections within their prior uncertainties a few hundred times to obtain sufficiently approximate estimates of the similarity index, as shown below. As demonstrated in earlier work using the Sampler code under the SCALE environment (Wieselquist, 2016), this forward-based approach provides two advantages; first, it allows one to calculate similarity indices when the adjoint solver is not available; and second, it provides a way to verify the results of adjoint-based calculations.

It has been shown in earlier work that these two vectors may be interpreted as the directional sensitivity profiles with respect to the dominant eigen directions of the prior covariance matrix. To illustrate the mechanics of the forward-based approach for calculating the similarity index, first consider re-writing the cross-section covariance matrix decomposed by Cholesky methodology as follows: C αα = U Λ 2 U T (10) where U ∈ ℝ n × n a unitary matrix;   U T U = U U T = I and Λ ∈ ℝ n × n is a diagonal matrix whose elements are square root of the singular values of C αα .

If ξ ( p ) ∈ ℝ n is a Gaussian variable, one can generate N random samples for the cross-sections which respect their covariance structure, such that: Δ α ( p ) = U Λ ξ ( p ) , p = 1,2 , … , N

By the law of large numbers, one can show that as N increases (Stark and Woods, 2012) lim N → ∞ ( 1 N ∑ p = 1 N ξ ( p ) ξ ( p ) T ) = I

This limit is readily reached with a few hundred samples. Verification with numerical experiments is provided in the following section.

Then, Eq. 10 can be re-written by the cross-section samples, such as: C αα = lim N → ∞ U Λ ( 1 N ∑ p = 1 N ξ ( p ) ξ ( p ) T ) Λ T U T = lim N → ∞ ( 1 N ∑ p = 1 N Δ α ( p ) Δ α ( p ) T ) With the linearity assumption valid, e.g., S kα Δ α ( p ) = Δ k ( p ) , samples for the code-calculated responses also can be calculated by the sandwich rule in Eq. 5, and thus the covariance matrix for calculated responses can be re-written as: C kk = S kα U Λ 2 U T S kα T = lim N → ∞ ( 1 N ∑ p = 1 N S kα Δ α ( p ) Δ α ( p ) T S kα T ) = lim N → ∞ ( 1 N ∑ p = 1 N Δ k ( p ) Δ k ( p ) T ) where Δ k transforms the sensitivity matrix S kα using the Chelosky factor of C αα .

Thus, the construction of the C kk matrix effectively reduces to the following three steps; first, it transforms the original variables into a set that are uncorrelated; second, it calculates the sensitivities along the directions of the transformed variables (referred to as directional sensitivity in the calculus literature); and third, it weighs each directional sensitivity by its corresponding prior uncertainty.

The deviation vector of jth experiment (or application denoted by subscript M + 1 ) code-calculated samples from its reference value can be written as: Δ k j = [ k j ( 1 ) − k j ( r e f ) k j ( r e f ) k j ( 2 ) − k j ( r e f ) k j ( r e f ) ⋮ k j ( N ) − k j ( r e f ) k j ( r e f ) ] , j = 1,2 , … , M + 1

Each term in Δ k j may be considered as a sample resulting from executing the forward model with a random cross-section perturbation. Thus, Δ k j is a vector of N sampled perturbations of the jth response. Graphically, these perturbation vectors can be displayed via a scatter plot. For example, Figure 3 shows a representative scatter plot of the perturbation vectors for the jth experiment and the application.

Their similarity index thus reduces to the standard correlation coefficient between the two vectors Δ k j and Δ k M + 1 similarly to Eq. 9 such that: cos ⁡ θ ( Δ k j , Δ k M + 1 ) = Δ k j T Δ k M + 1 Δ k j T Δ k j Δ k M + 1 T Δ k M + 1 With a large number of samples, the inner-products of any two sample vectors reduce to the elements of the C kk matrix, such that: lim N → ∞ Δ k i T Δ k j = C kk [ i , j ]

Thus, c k ( k j , k M + 1 ) = lim N → ∞ Δ k j T Δ k M + 1 Δ k j T Δ k j Δ k M + 1 T Δ k M + 1

In this manuscript, the ACCRUE index will be calculated using both the analytical definition, presented in Section 3, as well as the noted forward-based approach for verification.

Different from the c _k index which relies on the C kk matrix, the j _k index leverages the C kk + C mm matrix which appears in the GLLS procedure to weigh the prior values and the experimental measurements. This matrix can be shown to be the covariance matrix for the discrepancies between the calculated and measured values, whereas the C kk matrix is the covariance matrix for the calculated values only.

The discrepancies can be aggregated in a vector such that: d = [ k 1 − m 1 k 2 − m 2 ⋯ k M − m M k M + 1 ] T where the last element, k M + 1 , of the discrepancy vector remains the same as k , since measurement uncertainty for the application is not applicable.

Then the covariance matrix for the discrepancy vector, d , is: C dd = [ D 1,1 D 1,2 ⋯ D 1 , M D 1 , M + 1 D 2,1 D 2,2 ⋯ D 2 , M D 2 , M + 1 ⋮ ⋮ ⋱ ⋮ ⋮ D M , 1 D M , 2 ⋯ D M , M D M , M + 1 D M + 1,1 D M + 1,2 ⋯ D M + 1 , M D M + 1 , M + 1 ] ∈ ℝ ( M + 1 ) × ( M + 1 ) (11) where D i , j = C kk [ i , j ] + C mm [ i , j ] and C mm [ i , j ] describes the possible correlations between the experimental measurements. The elements of the last column or the last row of the C dd are the same as those of the C kk matrix since no measurement is applicable for the application. If the experiments are independent, then the C mm matrix reduces to a diagonal matrix.

Similarly to before, consider the expressions in Eq. 11 using the Cholesky decomposition of C dd as: C dd = D T D where D = [ δ 1 δ 2 ⋯ δ M + 1 ] , where the inner-product of any of two columns of D gives: 7 δ i T δ j = C dd [ i , j ] , i , j = 1,2 , … , M + 1

Geometrically, the ACCRUE index j _k with a single experiment thus is defined as the cosine angle defined by two vectors of D , specifically one related to the application and the other to an experiment as: j k ( d j ; d M + 1 ) = δ j T δ M + 1 δ j T δ j δ M + 1 T δ M + 1 = C dd [ j , M + 1 ] C dd [ j , j ] C dd [ M + 1 , M + 1 ] = cos ⁡ θ ( δ j , δ M + 1 ) (12)

Figure 4 graphically shows how the measurement uncertainty impacts on the j _k value with a single experiment (the jth experiment denoted by subscript “j”) as compared to the c _k value. Due to the measurement uncertainty, the cosine angles estimated by c _k and j _k respectively change from cos ⁡ θ j to cos ⁡ φ j . If the associate measurement uncertainty of the given experiment is zero, the j _k value reduces to the c _k value. In any other cases where the measurement uncertainty of the given experiment is not zero, the cosine angle given by j _k is always smaller than that by c _k as illustrated, e.g., cos ⁡ φ j < cos ⁡ θ j or φ j > θ j . This can be readily proved by comparing the analytical expressions for each c _k and j _k value shown in Eqs. 9, 12, respectively.

With regard to the matrix notations of both equations, C kk [ j , M + 1 ] and C dd [ j , M + 1 ] , C kk [ M + 1 , M + 1 ] and C dd [ M + 1 , M + 1 ] are the same respectively since measurement is not applicable for the application, while C dd [ j , j ] is always greater than C kk [ j , j ] since C dd [ j , j ] = C kk [ j , j ] + C mm [ j , j ] . As a result, the cosine angle is reduced proportional to C kk [ j , j ] / C dd [ j , j ] with experimental measurement uncertainty present.

As discussed in Section 2.3, the covariance matrix, C mm , can be written as: C mm = L L T = lim N → ∞ ( L 1 N ∑ i = 1 N ζ ( p ) ζ ( p ) T L T ) = lim N → ∞ ( 1 N ∑ p = 1 N Δ m ( p ) Δ m ( p ) T ) where L ∈ ℝ M × M a low triangular matrix.

Then the measurement sample vector for jth experiment, Δ m j ∈ ℝ N , is defined as: Δ m j = [ m j ( 1 ) − m j ( r e f ) m j ( r e f ) m j ( 2 ) − m j ( r e f ) m j ( r e f ) ⋮ m j ( N ) − m j ( r e f ) m j ( r e f ) ] , j = 1,2 , … , M

Thus, the discrepancy sample vector for jth experiment, Δ d j ∈ ℝ N , is: Δ d j = Δ k j − F j Δ m j where F j is a scalar quantity representing a ratio of experimental to calculated response values of jth experiment lim N → ∞ Δ d i T Δ d j = C dd [ i , j ] Thus, j k ( d j ; d M + 1 ) = lim N → ∞ Δ d j T Δ d M + 1 Δ d j T Δ d j Δ d M + 1 T Δ d M + 1

The ACCRUE index may be viewed as the similarity between a group of experiments and the application of interest when the experimental uncertainties are excluded from the analysis. Its true value however is in assessing the relevance of a new experiment by taking into account both the experiment’s measurement uncertainty and the value gleaned from past experiments. In this section, the detailed analytical derivation for j _k value with multiple experiments is provided with the δ vectors. However, if analysts are not interested in including the impact of measurement uncertainties, they can work directly with the γ vectors instead of δ vectors.

Analytically, the j _k index for the first L experiments is given using the elements of the C dd matrix in Eq. 11 by: j k = ∑ i = 1 L ( E i , M + 1 ) 2 E i , i   D M + 1 , M + 1 (13) where the E terms are defined by the following recurring relation: E 1 , j = D 1 , j E i , j = D i , j − ∑ k = 1 i − 1 E k , i E k , k E k , j

These terms can be evaluated analytically given access to the sensitivity coefficients, the prior cross-section covariance matrix, and the measurement uncertainties. By way of an example, consider the simple case with L = 1 , i.e., a single experiment, where the j _k reduces to: j k ( d 1 ; d M + 1 ) = ( D 1 , M + 1 ) 2 D 1,1   D M + 1 , M + 1 = s 1 C αα s M + 1 T s 1 T C αα s 1 + σ m 1 2 s M + 1 T C αα s M + 1 (14)

This expression equivalent to the c _k index assuming zero measurement uncertainty, i.e., σ m 1 2 = 0.0 . The implication is the j _k will always have a lower value than the corresponding c _k value for any realistic non-zero experimental uncertainties. With L = 2 , and assuming the measurement uncertainties are uncorrelated, i.e., C mm [ 1,2 ] = C mm [ 2,1 ] = 0 , the j _k value becomes: j k ( d 1 , d 2 ; d M + 1 ) = ( E 1 , M + 1 ) 2 E 1,1   D M + 1 , M + 1 + ( E 2 , M + 1 ) 2 E 2,2   D M + 1 , M + 1 = ( D 1 , M + 1 ) 2 D 1,1   D M + 1 , M + 1 + ( D 2 , M + 1 − D 1,2 D 1,1 D 1 , M + 1 ) 2 ( D 2,2 − D 1,2 D 1,1 D 1,2 )   D M + 1 , M + 1   = ( s 1 T C αα s a ) 2 ( s 1 T C αα s 1 + σ m 1 2 ) ( s a T C αα s a ) + ( s 2 T C αα s a − s 1 T C αα s 2 s 1 T C αα s 1 + σ m 1 2 s 2 T C αα s a ) 2 ( s 2 T C αα s 2 + σ m 2 2 − s 1 T C αα s 2 s 1 T C αα s 1 + σ m 1 2 s 1 T C αα s 2 ) s a T C αα s a (15)

Eq. 15 shows that the relevance of two experiments may be expressed as the sum of two terms, one very similar to the c _k index representing the relevance of the first experiment, and the other subtracting away the impact of the first experiment. To help interpret the j _k index for L experiments, we resort to the geometrical and statistical interpretations provided earlier in Section 3.1.

Consider the case with two experiments (denoted by the subscripts “1” and “2”, respectively) as illustrated in Figure 5, where j _k calculates the angle between the δ M + 1 vector (representing the application) and the subspace formed by δ 1 and δ 2 (representing the two experiments). As long as the second experiment provides additional information which is not duplicated by the first experiment, the j _k value is expected to increase as compared to the value obtained with a single experiment, i.e., j k ( δ 1 , δ 2 ; δ M + 1 ) > j k ( δ 1 ; δ M + 1 ) or φ 2 < φ 1 as shown in Figure 5. In the case that the second experiment carries no additional information, e.g., the j _k value remains the same. The implication is that the second experiments provides no value to the GLLS inference procedure, and hence can be excluded.

The cosine angle between any two subspaces can be calculated by orthogonal projection techniques, e.g., QR decomposition or Singular Value Decomposition (SVD). For example, consider the j _k value of the first L experiments and the application, the corresponding δ vectors can be aggregated as: D L = [ δ 1 δ 2 ⋯ δ L ]

Then, the associated j _k expression can be written as: j k ( D L ; δ M + 1 ) = cos ⁡ θ ( D L , δ M + 1 ) = cos ⁡ θ ( Q L , v ) = | | Q L T v | | = ( q 1 T v ) 2 + ( q 2 T v ) 2 + ⋯ ( q L T v ) 2 where q j   ( j = 1,2 , … , L ) is jth q vector of the Q L matrix from QR decomposition and v is a normalized δ M + 1 vector for the application such as v = δ M + 1 / | | δ M + 1 | | .

For example, consider the case where D 1 contains only δ 1 vector corresponding to the first experiment, i.e., D 1 = δ 1 , then the j _k value is calculated as: j k ( D 1 , δ M + 1 ) = ( q 1 T v ) 2 where q 1 is a normalized directional vector of δ 1 : q 1 = δ 1 | | δ 1 | |

Thus, the j _k value is j k = ( δ 1 T δ M + 1 ) 2 δ 1 T δ 1 δ M + 1 T δ M + 1 = δ 1 T δ M + 1 δ 1 T δ 1 δ M + 1 T δ M + 1 = s 1 T C αα s a s 1 T C αα s 1 + σ m 1 2 s a T C αα s a which is equivalent to Eq. 14

If D 2 contains two δ vectors ( δ 1 and δ 2 ) corresponding to the first two experiments, then the j _k value is calculated as: j k ( D 2 , δ M + 1 ) = ( q 1 T v ) 2 + ( q 2 T v ) 2 where q 2 is defined by Gram-Schmidt process such as: q 2 = ( δ 2 − δ 1 T δ 2 δ 1 T δ 1 δ 1 ) δ 2 T δ 2 ( 1 − δ 1 T δ 2 δ 1 T δ 2 δ 1 T δ 1 δ 2 T δ 2   )

Thus, the j _k value is: j k = ( δ 1 T δ M + 1 ) 2 δ 1 T δ 1 δ M + 1 T δ M + 1 + ( δ 2 γ a − δ 1 T δ 2 δ 1 T δ 1 δ 1 δ M + 1 ) 2 ( δ 2 T δ 2 − δ 2 T δ 1 δ 1 T δ 2 δ 1 T δ 1 ) δ M + 1 T δ M + 1 = ( s 1 T C αα s a ) 2 ( s 1 T C αα s 1 + σ m 1 2 ) ( s a T C αα s a ) + ( s 2 T C αα s a − s 2 T C αα s a s 1 T C αα s 2 s 1 T C αα s 1 + σ m 1 2 ) 2 ( s 2 T C αα s 2 + σ m 2 2 − s 1 T C αα s 2 s 1 T C αα s 2 s 1 T C αα s 1 + σ m 1 2 ) s a T C αα s a which is equivalent to Eq. 15

Consequently, the general expression for j _k value can be written as: j k = ∑ i = 1 L ( u i T δ M + 1 ) 2 u i T u i   δ M + 1 T δ M + 1 where u 1 = δ 1 u i = δ i − ∑ k = 1 i − 1 u k T δ i u k T u k u k ,     i = 2,3 , … , L which evaluates the same scalar quantity as that calculated by the matrix element notations in earlier this section.

Figure 6 shows the overall process for evaluating the similarity or relevance of an experiment to a given application. The similarity accounts for the correlation between two responses, e.g., one from the application and the other from the experiment, as calculated by a computer code. The ACCRUE index extends the concept of similarity to quantify the relevance, as measured by the added value of the experiment, taking into account both the experiment’s measurement uncertainties as well as past experiments. As shown in this figure, the first step is to check if an adjoint solver is available which allows one to employ sensitivity coefficients. If no experimental measurement uncertainty is available and only a single experiment is available, the ACCRUE index j _k reduces to the similarity index c _k . If the experimental uncertainty is available, Eq. 12 is employed to evaluate the j _k index. If additional experiments are available, then the most general expression for j _k is employed per Eq. 13.

Figure 9 shows representative results for the high relevance case study for three different applications with high, intermediate, and low biases. The x-axis explores the change in the GLLS-estimated biases when adding one experiment at a time, wherein the experiments are ordered according to their c _k values (top graphs), and j _k values (bottom graphs). The c _k -sorting is straightforward as each experiment is employed once in conjunction with the given application. Markedly different, the j _k value depends on the number and order of experiments included, hence each sorting is expected to give rise to different profiles for j _k and the associated bias and bias uncertainty. The goal is thus to identify the order that allows the analyst to reach a certain level of confidence in the calculated bias with minimal number of experiments. In the current study, the search for this optimal order is initiated with the experiment having the highest c _k value, with the second experiment selected to maximize the j _k value among all remaining experiments, and so on. In practice, one may start with any experiment, and adds experiments as they become available, or may employ the j _k value to quantify the value of new/planned experiments. For each added experiment, the GLLS bias and bias uncertainty are calculated to help compare the c _k and j _k -sorting. The goal is to achieve a stable prediction of the bias with minimal uncertainty.

These results show that the bias stabilizes quicker with the j _k sorting. More important, the c _k -sorting could show sudden changes after a period of stable behavior. For all considered applications using the c _k -sorting, the biases continue to experience sudden or gradual changes following a period of stable behavior. For example, for the low bias application, the estimated bias exhibits an upward trend after the 20th experiment. The implication is that the additional experiments continue to provide value to the GLLS procedure despite their lower relevance. With the j _k -sorting, a more explainable trend is achieved whereby the bias trend shows stable behavior after the 20th experiment.

Regarding the bias uncertainties, shown in Figure 9 as error bars, they are plotted in Figures 10, 11 as a function of the number of experiments using both the c _k and j _k -sorting for, respectively, one application with high relevance experiments and one application with low relevance experiments. The results highlight a key limitation of the c _k -sorting, that is the addition of low relevance experiments could change the trend of both the bias and the bias uncertainty. The j _k -sorting however does not suffer from this limitation, implying that one can select the minimal number of experiments required to reach a certain pre-determined level of confidence for the calculated bias. Comparison of the uncertainty values in both figures indicate that the impact is much less pronounced when highly relevant experiments are available. This highlights the value of the j _k -sorting when limited number of experiments are available, as is the case with first-of-a-kind designs.

These results are compared in Figure 12 with randomized orders (shown as multi-color solid lines) for one of the low relevance applications at different levels of measurement uncertainty, specifically 10 pcm (representing a highly accurate measurement), 100 pcm (a realistic measurement), and 500 pcm (a low confidence measurement) for the measured k _eff values. The results show that pure random sampling could be superior to c _k -sorting, with the j _k sorting still exhibiting the best behavior in terms of reducing the bias uncertainty with minimal number of experiments. When the measurement uncertainty is too high, the ordering of the experiments is no longer important. This is a key observation highlighting the value of ordering experiments as experimentalists continue to improve their measurements in support of model validation.

To help understand the changes in the bias and bias uncertainty associated with the c _k -sorting, the left plot in Figure 13 orders the experiments according to their c _k values, and the middle plot calculates the corresponding j _k values using the c _k -sorting. Notice that although the first few experiments, i.e., #15, #13, #12, …, to #4 have higher relevance than later experiments, they do not change the j _k value, and hence the bias and bias uncertainty as shown in the earlier figures. The j _k values start to show larger increase when additional lower relevance experiments are added, explaining the sudden or gradual change in the bias and its uncertainty. However, when the experiments are ordered according to their j _k values, as shown in the right plot, a smoother j _k profile is obtained which is consistent with the bias and bias uncertainty profiles obtained using the GLLS procedure. The implication is that one can employ the j _k profile to develop insight into the amount of experimental effort necessary to reach a target confidence in the calculated bias, even before the actual measurements are recorded. This follows because j _k only requires access to the prior uncertainties and the expected measurement uncertainty, not the actual measured bias.

As mentioned earlier, the analytical expressions for the similarity or relevance metrics such as c _k and j _k require access to derivatives which may not be readily available. To address this challenge, earlier work has developed an alternative to the estimation of c _k value using a non-intrusive forward-based stochastic method (Huang et al., 2020). In this section, we present numerical results comparing the results of the analytically-calculated j _k value to that estimated by the noted stochastic method. This will serve two purposes: the first is to provide a simple approach for the calculation of the j _k value when derivatives are unavailable, and the second to help verify the calculated j _k value using the stochastic method by comparing it to the analytically-calculated value. To achieve that, two benchmark experiments (HMF-005-001 and 005-002) are selected to calculate the j _k value. Their calculated response uncertainties are 1,492 pcm and 1721 pcm respectively, and measurement uncertainties are 360 pcm. A total of 1,000 samples are generated to examine the convergence of forward-based j _k value, whose corresponding analytical value is 0.9445 as given by Eq. 15, shown in Figure 14 as a horizontal line. The results indicate that the forward-based approach produces acceptable approximation of the analytical value using few hundred simulations, which is consistent with the results reported previously for the c _k value (Huang et al., 2020).