In TAS experiments, the concept of a three-dimensional reciprocal space Q is essential. For a given sample (or material), an element of Q is denoted by q = (h,k,l) ^⊤ . An element of the energy transfer space E is denoted by ω. Furthermore, we call (q,ω) ^⊤ ∈ R ^r , r = 4, the Q - E variables. For details, we refer to Shirane et al. (2002).

Since TAS experiments are carried out only along n ≤ r, n ∈ N, predefined directions in Q-E space, we introduce experiment variables x = ( x 1 , … , x n ) ⊤ ∈ R n and a corresponding affine transformation T : R ⁿ → R ^r to Q-E variables, i.e., q ω = W x + b =: T x (8) for a full rank matrix W ∈ R ^r×n and an offset b ∈ R ^r . It follows that x = W ⊤ W − 1 W ⊤ q ω − b =: T − 1 q , ω (9) for (q,ω) ^⊤ ∈ R ^r . Note that T ⁻¹(T(x)) = x for each x ∈ R ⁿ , but T(T ⁻¹ (q, ω)) = (q,ω) ^⊤ only for (q,ω) ^⊤ ∈ T(R ⁿ ).

Each experiment variable x _k only ranges between respective limits of investigation x k ± ∈ R , i.e., we have that x k − ≤ x k ≤ x k + . (10)

The corresponding domain of interest X ⊆ R n is defined as X : = x 1 − , x 1 + × ⋯ × x n − , x n + . (11)

The representation of a TAS instrument is divided into two structures, an instrument configuration and an instrument.

Definition (Instrument configuration). The tuple Π = sm , sc , se (12) for a scan mode sm ∈ {”constant k _i ”, ”constant k _f ”}, a scan constant (k _i or k _f ) sc > 0, and a vector of scattering senses se = ( se mono , se sample , se ana ) ⊤ ∈ { − 1 , + 1 } 3 is called an (TAS) instrument configuration.

To measure intensities at a certain point in Q-E space, a TAS instrument needs to steer its axes to six related angles. However, it is enough to regard only a subset of four angles due to dependencies.

Definition (Instrument). The tuple Ins = v , d mono , d ana (13) for a vector of angular velocities v = ( v 1 , … , v 4 ) ⊤ ∈ 0 , ∞ 4 , and lattice parameters of the monochromator, d _mono > 0, and the analyzer, d _ana > 0, is called an (TAS) instrument.

The lattice parameters of the monochromator and the analyzer are necessary for a well-defined translation of points in Q-E space to associated angles of instrument axes. The so-called angle map Ψ : dom ( Ψ ) → 0 , π 4 , q , ω ↦ Ψ 1 q , ω , … , Ψ 4 q , ω ⊤ (14) is induced by a sample, its orientation, and an instrument Ins and its configuration Π. Note that angular velocities v _k are related to angles Ψ _k (q, ω). The domain dom(Ψ) ⊆ Q × E denotes the set of points in Q-E space for which Ψ is well-defined, i.e., points that are reachable by the instrument Ins. Furthermore, we define X ∗ := T − 1 dom ( Ψ ) ∩ T X (15) as the set of points in X such that Ψ x := Ψ T x (16) is well-defined for x ∈ X ∗ . The limits of investigation for the domain of interest X have to be set such that Ψ   : X ∗ → 0 , π 4 becomes an injective function.

For a given configuration Π and instrument Ins, we can specify a resolution function (Shirane et al., 2002) φ Π , Ins : T X ∗ → T X ∗ → 0 , ∞ , (17) i.e., each point in T ( X ∗ ) ⊆ R r gives an individual resolution function defined over T ( X ∗ ) . Of course, the exact resolution function depends on additional parameters such as distances between instrument components or beam divergence allowed by collimators. However, these parameters are fixed during an experiment and hence can be omitted here. If Π and Ins are known from the context, we write φ = φ _Π,Ins. Also, for x ∈ X ∗ , we define φ Π , Ins x := φ Π , Ins T x . (18)

A sample and its orientation induce a so-called scattering function s   : T ( X ) → 0 , ∞ . Again, for x ∈ X , we define s x := s T x . (19)

Given a resolution function φ, we get an associated intensity function i : X ∗ → 0 , ∞ defined by a convolution of s with φ, i.e., i x := s ∗ φ x x . (20)

For benchmarking, we assume that i can be exactly evaluated. Of course, this is not possible in real experiments due to background and statistical noise.

For our suggestion of a benefit measure (Section 4.3), we additionally need an intensity threshold τ > 0.

The experimental time is defined as the sum of the cumulative counting time at measurement points in Q-E space and the cumulative time for moving the instrument axes. For simplicity, we assume here that the single counting time, i.e., the counting time at a single measurement point, denoted by T _count ≥ 0, is constant for each point.

Now, we can specify the notion of a TAS test case and a TAS experiment.

Definition (TAS test case). A tuple t = sample , orientation , W , b , X , Π , Ins , τ , T count (21) for a sample and its orientation, an affine transformation induced by a full rank matrix W ∈ R ^r×n and an offset b ∈ R ^r (Eq. 8), a domain of interest X ⊆ R n (Eq. 11), an instrument configuration Π, an instrument Ins, an intensity threshold τ > 0, and a single counting time T _count ≥ 0 is called a TAS test case.

Note that a TAS test case t induces an intensity function i : X ∗ → 0 , ∞ .

In the context of a certain test case, a TAS experiment is defined as a collection of intensities i(x) at locations x ∈ X ∗ .

Definition (TAS t-Experiment). Let t be a TAS test case. A TAS t -experiment A = A t is an N-tuple of location-intensity pairs A = x 1 , i 1 , … , x N , i N , (22) where x j ∈ X ∗ denote measurement locations and i _j = i(x _j ) ≥ 0 are corresponding values of the intensity function i induced by t.

We need to align our proposition of a cost measure with the limited experimental time, which is the critical quantity in a TAS experiment. Recall that the experimental time is defined as the sum of the cumulative counting time and the cumulative time for axes movement (Section 4.1).

For the cumulative counting time, we define c count A := ∑ j = 1 | A | T count = | A | ⋅ T count , (23) where T _count ≥ 0 denotes the constant single counting time.

The cost measure representing the cumulative time for moving the instrument axes is defined as c axes A := ∑ j = 1 | A | − 1 d x j , x j + 1 (24) for a metric d   : X ∗ × X ∗ → 0 , ∞ . The metric d = d _t measures the maximum time of axes movement between corresponding angles, i.e., d x , x ′ = max 1 ≤ k ≤ 4 Ψ k x − Ψ k x ′ v k , (25) where Ψ = ( Ψ 1 , … , Ψ 4 ) ⊤ denotes the angle map from Eq. 14 and v = ( v 1 , … , v 4 ) ⊤ is the vector of the instrument’s angular velocities (Eq. 13). Note that the metric d is fully determined by the TAS test case t. Also, it is indeed a metric (Encyclopedia of Mathematics, 1999) since the angle map Ψ was chosen to be injective.

The measures c _count and c _axes can be used either individually or additively to form a cost measure representing the entire experimental time. For the latter, we finally define c A := c count A + c axes A . (26)

In our opinion, this cost measure is the most suitable to reflect time costs in a general TAS experiment.

We propose a benefit measure that measures a type of weighted L ² approximation error between a benchmark intensity function i = i _t and an approximation i ̂ = i ̂ ( A ) resulting from an experiment A = A t . For example, i ̂ can be constructed with location-intensity pairs from A by linear interpolation or other approximation methods. To compare benefit values across experiments relating to different test cases, benefit measures should be ”normalized”, i.e., we regard relative errors.

Let us define μ A := ‖ i − i ̂ A ‖ ‖ i ‖ , (27) where ‖⋅‖ is a norm that enables to control the error measurement. Note that the notion of a “benefit” refers to an “error” in this case, i.e., reducing the benefit measure μ = μ _t leads to an increase in benefit.

A suitable error norm ‖⋅‖ needs to reflect that a TAS experimenter is more interested in regions of signal than in the background. This suggests that we use i itself in a suitable definition. However, an important constraint is that signal regions with different intensities are weighted equally since they might be equally interesting. For this, we use the intensity threshold τ > 0 from the TAS test case t (Eq. 21) and define i τ x := min { i x , τ } ≥ 0 (28) for x ∈ X ∗ , i.e., we cut i to a maximum intensity value of τ. As i _τ is a nonnegative function, its normalization ρ i , τ x := i τ x ∫ X ∗ i τ x ′ d x ′ (29) is a probability density function and can be used for weighting. Finally, we set ‖ ⋅ ‖ = ‖ ⋅ ‖ i , τ := ‖ ⋅ ‖ L 2 X ∗ , ρ i , τ , (30) where ‖ h ‖ L 2 X ∗ , ρ = ∫ X ∗ h x 2 ρ x d x 1 / 2 (31) for a function h and a density function ρ. Note that ‖ ⋅ ‖ L 2 ( X ∗ , ρ ) can be approximated by numerical quadrature rules or estimated by a Monte Carlo approach.

A useful set of test cases represents the variety of different scenarios that can occur in a TAS experiment. It is the set of intensity functions that is particularly important here. Therefore, we suggest to create test cases such that the corresponding induced intensity functions are composed of one or more of the following structures known in the field:

• non-dispersive structures (e.g., crystal field excitations),

Particular intensity functions can be created by mathematical expressions, physical simulations, or experimental data.

As an example for an outcome of our benchmarking procedure, Figure 1 displays a comparative benchmark result of two approaches for a test case including an intensity function that reflects a soft mode of the ferroelectric phase transition at ∼ 60 K of a transverse optical phonon measured on SnTe (Weber and Heid, 2021).

This figure contains exemplifications of each benchmark component: experimental time (x-axis in Figure 1C) as cost measure (Section 4.2), approximation error (y-axis in Figure 1C) as benefit measure (Section 4.3), a phonon-type intensity function (Figures 1A,B) as part of the test case (Section 4.4), and milestone values (ticks on x-axis in Figure 1C).

The TAS setting does not comprise each detail occurring in a real experiment. Indeed, we neglected some aspects which we, however, see as acceptable deviations.

Certainly the most prominent neglected aspect is background and statistical noise. We find it difficult to include them in the benchmarking procedure due to their non-deterministic nature since a comparison of approaches needs to be done in a deterministic setting. The benefit measure from Section 4.3, for instance, makes use of a “true” benchmark intensity function which would not be available in the presence of background or statistical noise.

We assumed that the single counting time is constant for each measurement point. However, since single counting times are determined by the physics of the sample and the requested statistics in real experiments, they might vary at different measurement locations. In our opinion, this variation is negligible in a first step, but can be taken into account in more complex benchmarking setups if desired.

Next, the cost measures from Section 4.2 are rather general and can be applied to any TAS instrument. They can, however, be arbitrarily extended by more details if benchmarking is to be done in the context of a specific instrument. For example, real TAS instruments may differ in more [PANDA (Schneidewind and Čermák, 2015)] or less [ThALES (Boehm et al., 2015)] time-consuming procedures for moving k _i values.

Furthermore, the intensity functions induced by the test cases described in Section 4.4 are derived in the context of fixed environmental parameters of the sample (such as temperature, external magnetic or electric field, etc.). Future work might extend the four-dimensional Q-E space with these parameters, i.e., r > 4, to allow for more complex intensity functions that would, however, require adjusted cost measures.

Finally, to compute all benefit values for Table 1, the benchmarking procedure requires an approach to perform an experiment that is large enough (Eq. 3). Hence, an autonomous stopping criterion is not tested although we consider it a crucial part of a fully autonomous approach.

The benchmarking procedure is formulated in a modular way and depends only on abstract components like cost measures, benefit measures, and test cases. For TAS experiments, we specified these components in Section 4, but feel that the overall procedure is also applicable for scattering methods other than TAS such as diffraction, reflectivity, SAS/GISAS, or TOF. Indeed, experimenters for each of these methods measure costs and benefits in their own way and investigate different kinds of intensity functions.

Diffraction experiments (Sivia, 2011), for example, may have a setting similar to TAS experiments as the experimenter is interested in intensities defined over Q variables and many diffractometers need to move their components (sample, detector). Therefore, the cost and benefit measures presented above might also be useful in this context. The set of test cases, however, would need to be composed differently since regions of signal are mainly separated and small in shape. Also, additional aspects (such as shadowing, overlapping reflections, missing knowledge of symmetry, etc.) are to be taken into account.

Since the benchmarking procedure has algorithmic structure, we decided to provide an implementation in the form of Python code that computes sequences of benefit values for given cost and benefit measures (cf. Table 1). Also, benchmark components for the TAS setting are already implemented. The repository along with instructions on how to run the code is publicly available (Teixeira Parente and Brandl, 2021). It also contains descriptions of test cases that can be complemented in the future.