HYPSTAR^® (Hyperspectral Pointable System for Terrestrial and Aquatic Radiometry; www.hypstar.eu) is an autonomous hyperspectral radiometer system dedicated to surface reflectance validation of all optical Copernicus satellite data products. For a complete description of the HYPSTAR^® system, we refer to Kuusk et al. (in this issue). HYPSTAR^® takes radiance and irradiance measurements using the same spectrometer (though with different optical paths for radiance and irradiance). Two slightly different instruments are used for water and land: (1) The Standard Range (SR) model provides visible and near-infrared (VNIR, 380–1,020 nm) data and is used on water sites, (2) the eXtended Range (XR) models have an additional shortwave-infrared (SWIR) spectrometer module which extends the spectral range up to 1680 nm. The XR models are meant to be used on the land sites. Pictures of the SR and XR models of the HYPSTAR^® are shown in Figure 1, together with some example field sites. The instrument characteristics are summarised in Table 1.

The spectral sampling of the HYPSTAR^® is 0.5 nm in the VNIR and 3 nm in the SWIR, and the spectral resolution full-width half-maximum (FWHM) is 3 nm in the VNIR and 10 nm in the SWIR. The SWIR sensor is actively cooled (typically to 0 °C, but this can be configured) to ensure the stability of the measurements. The field of view of the radiance measurements is ∼ 2 ° for the SR model and ∼ 5 ° for the XR model. The irradiance measurements observe the full hemisphere, i.e. 180°. The HYPSTAR^® system is mounted on a low-cost pointing system which allows the acquisition of autonomous, multi-angular measurements. A GPS, light and rain sensors as well as cameras to image the target and the sensor heads are also included. Data acquired by the HYPSTAR^® is saved in separate binary ‘.spe’ files for each viewing geometry. These acquisitions are sent to the central server for processing, together with some ‘rgb’ images taken by the camera within the system, a file with meteorological information (i.e., relative humidity, temperature and pressure measured within the system and illuminance measured by an external light sensor), log files made during the acquisition and data transfer, and a metadata file.

The HYPSTAR instruments are calibrated by Tartu University ¹ (See Kuusk et al. in this issue). Currently this includes radiometric calibration (gains) and non-linearity calibration (non-linearity coefficients). Measurements of the temperature responsivity and stray light are also planned/in progress, but these characterisations are not yet used in the processing of the data in the current processor version. The lab calibration also includes a wavelength calibration. The wavelengths for each instrument will thus be slightly different, and the radiance and irradiance measurements can also have slightly different wavelengths for each spectral pixel (see Section 3.2.4.1). The wavelength calibration assigns a wavelength to each spectral pixel in the raw data, but not all the raw data pixels will be used in the final product. This is either because these wavelengths (e.g., in the UV) cannot be calibrated due to the spectral limits of the radiometric calibration references used, or because these wavelengths are simply too noisy to be used. In the final publicly distributed data (L2B products) the wavelength range is limited to the ranges specified in Table 1 in order to only supply the user with the best quality data.

Radiometer measurements are taken in a defined set of geometries called a sequence. For the WATERHYPERNET network, the output of a single sequence is the resulting water-leaving reflectance (possibly measured at different relative azimuth angles between Sun and sensor) and associated uncertainty and quality flags. For the LANDHYPERNET network, it is a set of surface reflectance measurements for different viewing geometries with associated uncertainties and quality flags. The set of acquisitions at each viewing geometry in a sequence is called a series, and it is composed of a set of repeat measurements called scans (which will be averaged). Hence, one scan results in a single unique spectrum with a specific integration time. The number of repeat scans in a single series depends on the desired parameter and its natural variability, potentially constrained by the total duration and power consumption. For the land XR instruments, the SWIR sensors typically have larger integration times than the VNIR sensors (in order to optimally use the dynamic range of the sensor). Typically, a fixed number (the default is 10) of SWIR scans will be set, and the VNIR sensor will keep taking scans until the SWIR sensor has completed its scans (leading to a larger number of scans for VNIR than for SWIR).

The geometries for the HYPERNETS network are defined in Figure 2. Each of the angles are defined in the reference frame centred on the measurement location on the surface. The viewing angles are thus defined as the sensor ‘viewing from’ a particular direction. In the side view, the viewing zenith angles are computed from Nadir to Zenith, i.e., a viewing zenith angle, θ _v , of 180° is looking upward (e.g., for downwelling irradiance measurements). The solar zenith angle is measured from Zenith. In the top view, the viewing azimuth angle, φ _v , is the angle measured clockwise from the absolute North to the direction of the sensor (i.e., the line from the target to the sensor). This definition of the viewing azimuth angle matches that of most optical satellite geometries (such as those of Sentinel-2, Sentinel-3, Landsat 8). Two additional azimuth angles are defined, i.e., the ‘pointing-to’ azimuth angle, φ _p , measured clockwise from the absolute North to the pointing direction (i.e., the line from the sensor to the target, or, φ _v +180°), and, the relative azimuth angle, Δφ, measured clockwise from the Sun to the pointing direction (i.e., φ _p -φ _s ). In this definition, Δφ = 0 means the instrument is looking towards the Sun, and Δφ = 180 means the instrument is looking away from the Sun. Our definitions of φ _p and Δφ match the definitions in Ruddick et al. (2019) for their viewing azimuth and Δφ, respectively (see their Figure 1).

Figure 3 illustrates the measurement protocol for a WATERHYPERNET sequence. The current standard water protocol follows commonly used measurement protocols (Ruddick et al., 2019, and references therein). Upwelling above-water radiance, L _u , series are taken at θ _v = 40° and sun-sensor relative azimuths, Δφ, at ± 90° and/or ± 135° (to avoid Sun glint and high skylight reflectance within the sensor field of view). Each series of L _u is preceded and followed by a series of above-water downwelling (sky) radiance, L _d , in the specular reflection direction for the correction of the reflected skylight (i.e., θ _v for L _u = 180° - θ _v for L _d ). Downwelling irradiance, E _d , series are taken at the beginning and the end of each sequence. A standard water sequence (including only one single azimuth angle) lasts approximately 5 min and is executed every 15–30 min during daylight.

The LANDHYPERNET sequences also start and end with downwelling irradiance series, but have multiple upwelling radiance series covering a range of different viewing geometries (including a minimum of five view zenith and six view azimuth angles). The design of the land measurement protocol aims to optimise the viewing geometry during satellite’s overpass times and by its repeats through the day to obtain information about Bidirectional Reflectance Distribution Function (BRDF) properties of the site. A plot illustrating the different viewing geometries for a land sequence is illustrated in Figure 4. Land sequences do not measure the downwelling (sky) radiance. A standard land sequence typically lasts approximately 15 min (depending on illumination) and is executed every 30 min during daylight.

WATERHYPERNET and LANDHYPERNET sequences following the above protocols are called ‘standard’ sequences.

The hypernets_processor module is an open source Python package, with the code actively developed on GitHub ² . It has a modular design that enables simple development. The various processing steps (described in the next section) have been implemented within a flexible framework, so that rapid advances in instrumentation and systems can be easily accommodated. The software is run through a command-line interface (CLI) and ingests data and processes with 24/7/365 operation including automated monitoring of diagnostics both for the network operator and for the site operators. Installation instructions are available in the hypernets_processor documentation ³ .

The hypernets_processor has a number of different categories of usage or modes of operation, to provide context to the rest of the design, they are initially described here. The first distinction is between the LANDHYPERNET processor and the WATERHYPERNET processor.

• LANDHYPERNET processor - processing for data taken by the LANDHYPERNET Network.

Where possible, the processing of the LANDHYPERNET and WATERHYPERNET is done as consistently as possible, with deviations between them only in the L1C and L2A processing (Figure 5). Also, there is a distinction between whether the measured data are from a standard measurement sequence or custom measurement sequence.

• Standard Sequence - dataset containing the standard set of measurements defined by the LANDHYPERNET or WATERHYPERNET network (as described in Section 2.2).

Standard sequences are automatically processed to L2A, whereas custom sequences are only processed to L1B (flags and anomalies will be raised, triggering an anomaly that will stop the processing, see Section 3.3). Finally, there is the distinction between the two potential use cases, network processing or field use.

• Network Processing - automated processing to prepare standard sequences retrieved from network sites for distribution to users.

In the remainder of this paper, we will mainly focus on the network processing of standard sequences. For further information on the use case of field use, we refer the reader to the corresponding page in the hypernets_processor documentation ⁴ .

There are many options that can be fine-tuned in the running of the hypernets_processor, from controlling which measurement functions are used, where to store the output data and what plots and output files are produced, to details like number of Monte Carlo iterations used in the uncertainty propagation, thresholds of the quality checks or what ancillary data to use in the calculation of the air-water interface reflection factor (referred to as ‘rhof_option’). These options can be controlled through three config files. The processor.config file contains all the options that are in common between all sites (different processor.config files are used for the WATERHYPERNET and LANDHYPERNET network, though with most values in common). The job.config file controls all site-specific options. Finally, there is also a scheduler.config file which controls how often the hypernets_processor checks for new data and how many sequences can be processed in parallel.

Set up routines are available to set up the processor (and the processor.config file using default values), to set up a new job (i.e., the processing of a new site, including what directory to check for new data) and to start the scheduler. The hypernets_processor documentation provides more information on how to set up and run the automated processing ⁵ .

The version of the hypernets_processor described in this paper is v2.0. This is the first version that is intended for operational use. Previous versions include a beta version (as described in Goyens et al., 2021), and a v1.0 which was used to produce the first public dataset on Zenodo (see Section 4.4). A new version of the dataset on Zenodo will be released upon publication of this paper, using the v2.0 of the hypernets_processor.

The version number is made up of two numbers. The first indicates the major version number. Changes to this number indicate significant changes and improvements to the code, and major changes to the datafiles. The second number, called the minor version, is incremented when there are minor feature changes or notable fixes which do not significantly change how the data products should be used.

Compared to the v1.0, there are a few changes in the v2.0 hypernets_processor worth noting, so that any HYPERNETS results using the v1.0 data products can be understood.

• The first important difference is in the definition of the viewing azimuth angles, which have changed by 180° in order to be consistent with satellite viewing azimuth angles in v2.0 (see Section 2.2). The pointing azimuth angles in v2.0 are equal to the viewing azimuth angles in v1.0. Pointing azimuth angles are mainly of interest for the WATERHYPERNET network as it is commonly used (Ruddick et al., 2019, and references therein) to compute the relative azimuth angles.

The network processing is done centrally on the LANDHYPERNET and WATERHYPERNET servers, through a command line interface, which continuously checks for new data, and processes it as soon as it comes in. The hypernets_processor takes the data from acquisition (raw data) to application of calibration and quality controls, computation of correction factors (e.g., air-water interface reflectance correction for water processing), temporal interpolation to coincident timestamps, processing to surface reflectance and averaging per series. A diagram showing the design for the network processing is provided in Figure 5.

The inputs are the raw field acquisitions and the instrument characterisation and calibration database (ICCDB). The main outputs are the various L0A-L2B NetCDF datasets listed in Table 2. The hypernets_processor also produces various plots and SQL databases of successfully processed products and anomalies (Section 4.3). The different processing steps and the plots produced are described in the next section. As part of the processing, quality checks are performed which either add quality flags to the produced data processing, or in some cases raise anomalies and halt the processing. These quality checks are described in Section 3.3. Uncertainties are also propagated through each of the processing steps, as described in Section 3.4. Details on the produced products are provided in Section 4.

Throughout the different processing steps described in the previous section, a number of quality checks are applied. These quality checks are described below. When a given quality check fails, there are two possible outcomes.

• When the quality check is critical for having useful data, failure of the quality check results in halting of the processing, and an anomaly is raised and stored in the anomaly database (see Section 4.3). In Tables 3, 4, it is indicated which quality checks halt the processing and what anomaly is raised.

The main output files produced by the hypernets_processor are in NetCDF CF-convention version 1.8 format. There are also plots, typically produced in png format, and SQL databases (see Section 4.3).

The different NetCDF files contain a range of different variables and metadata. The main measurands, as well as their dimensions for the different levels of data files are described in Table 2. For these measurand variables, there are also uncertainty variables for each of the components described in Section 3.4, as well as error-correlation variables for the systematic uncertainty components.

In addition to these there are coordinate variables, wavelength and series/scans, as well as a number of common variables (i.e., present in each of the data products) that provide additional details about the measurement. Acquisition time, viewing zenith and azimuth angle, solar zenith and azimuth angle are examples of common variables with series or scans as dimension. Bandwidth is also a common variable which has the wavelength dimension. Then there are a few additional variables such as the quality flag variable and variables specifying the number of valid and total VNIR scans and specifying the number of valid and total SWIR scans (for the LANDHYPERNET network).

There are also a number of variables that are only present in some of the data products. For example, there is some additional information in the L0A files, such as integration times, values of the accelerometers, the requested and returned pan/tilt angles. This information is propagated to the L1A and L0B files, but not beyond.

The quality flag field consists of 32 bits. Every bit is related to the absence or presence of a flag as described in Section 3.3. The quality flag value given in each data level is the compound value of the specific bits of each raised flag. The specific flags associated with each bit are given in the quality flag field metadata. Some flags are left as placeholders for future updates. Tables 3, 4 present the flags used in the current version.

The are is also a range of metadata contained within the files. For each variable, there is metadata such as the standard name, long name, units and uncertainty components (where relevant). The uncertainty variables will have additional metadata describing their error correlation (see Section 3.4). Finally, there is also a range of global metadata, describing information about how, and when the data was processed, what data files it used, information about the site (e.g., latitude and longitude) etc.

The naming convention is intended to allow the unique identification of all product files and to summarise the contents. It is composed of a defined sequence of data fields, separated by an underscore. For the HYPERNETS measurement data, the file name is composed as follows:

SYSTEM NETWORK SITEID LEVEL TYPE ACQUISITIONDATETIME PROCESSINGDATETIME_version.nc.

where.

• System: “HYPERNETS”

For instance, for a L1B product processed by the hypernets_processor version two of WATERHYPERNET network acquired at Blankaart South at 11:30 UTC on 2023–10–04 and processed at 11:30 UTC on 2023–10–05, the filename should be:

HYPERNETS_W_BSBE_L1B_RAD_20231004T1130_20231005T1130_v2.0.nc,

and the related L2A files for standard measurement protocols taken at 90° and 135° Δϕ should be:

HYPERNETS_W_BSBE_L1B_RAD_20231004T1130_20231005T1130_v2.0.nc, and,

HYPERNETS_W_BSBE_L2A_REF_20231004T1130_20231005T1130_135_v2.0.nc.

Table 5 defines the abbreviated name convention applicable to the individual HYPERNETS sites. Site name convention is a 4 letter abbreviation [LLCC] with LL standing for the location and CC for the country.

The hypernets_processor produces SQL Database entries to keep track of successfully processed sequences and anomalies in the processing. These entries are combined into the following three databases, each stored in the SQLite format: ¹⁴

• Archive Database: SQLite database listing all successfully processed data products, together with auxiliary information to enable queries (e.g., product_name, datetime, sequence_name, site_id, latitude, longitude, solar and viewing angles, etc.).

These databases can be used to produce processing statistics, or to find a set of sequences/anomalies that matched a certain set of criteria using SQL queries.

To follow the Findable-Accessible-Interoperable and Reusable principles (FAIR), particular attention is given to the data format and metadata, and, data accessibility. Files are in the NetCDF CF-convention version 1.8 format. Common metadata (e.g., metadata section added with each data product) follow the INSPIRE directives ¹⁵ in accordance with the EN ISO 19115 for the metadata elements and the Dublin Core Metadata Initiative ¹⁶ . Instrument, component and system metadata are bound by a unique metadata key (i.e., system_id) allowing to trace the history of the system (e.g., replacement, maintenance, system updates or instrument setup improvements).

The distribution of Near Real-Time (NRT, 24 h between data acquisition and data availability) LANDHYPERNET and WATERHYPERNET data will happen through the data portals for the LANDHYPERNET (www.landhypernet.org.uk) and WATERHYPERNET (www.waterhypernet.org). However, during the current prototype phase, where improvement of the quality checks is still ongoing, and further site-specific quality checks are still being added by the site-owners, these data portals are restricted to consortium members. In addition, for the WATERHYPERNET network the distribution may be delayed to ensure that the NCEP/GDAS forecast data for wind speed are made available for the latest sequence (should be less than 24 h). Data transfer from the system to the server may also delay the NRT processing (e.g., due to poor 4G connections on the field). In the near future, these data portals will be opened to the public and will become the reference source of data for HYPERNETS.

In the meantime, a subset of the HYPERNETS data until 2023–04–31 is publicly available and can be found on Zenodo (Brando et al., 2023; Brando and Vilas, 2023; De Vis et al., 2023; Dogliotti et al., 2023; Doxaran and Corizzi, 2023a; Doxaran and Corizzi, 2023b; Piegari et al., 2023; Goyens and Gammaru, 2023; Morris et al., 2023; Saberioon et al., 2023a; Saberioon et al., 2023b; Sinclair et al., 2023). The initial datasets provided here in June 2023 were produced using the v1.0 of the hypernets_processor (see Section 2.5). A new version of the datasets on Zenodo will be released upon publication of this paper, using the v2.0 of the hypernets_processor.

In this section, we discuss the results of the processing of a few example land sequences. Two examples are included, one for the WWUK site on 2022–06–26 at 11:40 UTC and one for the GHNA site on 2022–08–04 at 10:00 UTC. Figure 12 shows a few of the plots created by the hypernets_processor for these two sequences.

The radiance plots show a lot more variability between different viewing zenith angles for the WWUK site than for the GHNA site, which is to be expected as GHNA is expected to be especially homogeneous and near-Lambertian, whereas the WWUK is a vegetated site, which has spatial heterogeneity and significant bidirectional reflectance distribution function (BRDF) effects. When the irradiances are inspected and compared to the clear sky models, we see that for each of these cases, the clear-sky model matches the observed radiances quite well. We note that this will not always be the case, as the clear sky model does not take into account the different surface reflectance, pressure or aerosol properties for each site.

The bottom two rows of Figure 12 show how the surface reflectances vary with viewing zenith and azimuth angles. Typical reflectances for vegetation (WWUK) and deserts (GHNA) are found. Again, much more variation is seen between the reflectances for the different geometries for WWUK than for GHNA, as expected. There are some blips in the reflectance around the absorption features (e.g., around 760 nm and 1,370 nm), which are likely caused by differences in the SRF of the radiance and irradiance sensors (See also Ruddick et al., 2023). Uncertainties have been increased to account for these SRF differences (see Sections 3.4; 5.3).

The last column in Figure 12 shows the plots provided by the WATERHYPERNET processor for a sequence at M1BE on 2022–06–19 at 08:02 UTC. Measurements were made at a relative azimuth angle, Δϕ, of 90° and the wind speed retrieved from NCEP/GDAS was around 6.2 m⁻¹. Top plot shows the averaged radiance scans for two series of L _d (two times three scans) and one series of L _u (six scans). It was a clear blue sky at M1BE on that day, and illumination during the sequence seemed to be constant. This is confirmed by the downwelling radiance (top right panel) and irradiance (second row right panel) series that are similar at the start and the end of the sequence. The bottom right panel show the final averaged reflectance product, i.e., the L2A reflectance. This figure shows that the water at M1BE was green and turbid on that day with a reflectance peak around 550 nm and relatively high red-NIR values.

We have illustrated the typical relative uncertainties of the LANDHYPERNET and WATERHYPERNET products in Figure 13, using the same three examples (GHNA, WWUK & M1BE) as discussed in the previous sections. For radiance (top row), the three uncertainty components are shown (one line for each series) for the three sites. The two systematic components are very similar between each of the sites, though for the water example (M1BE), the wavelength range is different compared to the land sites. This similar shape of the systematic uncertainties is expected as the instruments deployed at three sites are calibrated in the same way and thus have similar calibration uncertainties. These systematic components are also the same for each sequence for these sites, as the same calibration is applied for every sequence. The random uncertainty on the radiance is more variable, as it is dependent on illumination conditions, the nature of the target, and the stability of the instrument. Smaller relative uncertainties are found for brighter surfaces. Note the difference in wavelength range between the land and water sites with, subsequently, higher uncertainties in the blue and NIR spectral for the water site as it corresponds to the extremes of the wavelength range. The uncertainty plots for irradiance in Figure 13 are very similar than for radiance. The main difference, in particular for the land sites, is that there are fewer series (only two), and the relative random uncertainties are a bit smaller (due to higher signal in irradiance).

For reflectance, there are only two uncertainty components as the systematic component for which the errors in radiance and irradiance are correlated become negligible. For most wavelengths, the (independent) systematic component dominates over the random uncertainties. The random uncertainties will become even smaller when averaging measurements or integrating over the SRF of a satellite sensor (e.g., De Vis et al., in this issue), as opposed to the systematic uncertainties which will remain constant. For the water site, the random uncertainty is higher compared to the systematic uncertainty in the NIR spectral range. This is due to the usually very low water signal in this spectral range.

Finally, Figure 14 top row shows the error-correlation between the different wavelength channels of the systematic component(s) for the L1B radiances of the GHNA example, i.e., the error-correlation matrix for the independent systematic uncertainty (left) and the correlated (between radiance and irradiance) systematic uncertainty (right). The bottom plots show the error-correlation for the L2A (independent) systematic uncertainties on reflectance for the land (left) and water (right) examples (GHNA and M1BE, respectively). For the independent systematic error correlations for land, there is a strong correlation between the wavelengths within the VNIR and SWIR sensors, but not between VNIR and SWIR. The strong wavelength correlation comes from various components, with a strong contribution from the placeholder uncertainty. As this placeholder is replaced by more realistic contributions, this error-correlation structure might become more complex. For the water example, correlations remain high at all wavelengths between the blue and red wavelengths.

In this section we briefly provide some of the processing statistics for both the LANDHYPERNET and WATERHYPERNET networks until the end of April 2023. for the LANDHYPERNET, there were a total of 12,190 sequences acquired. 11,802 of these (97%) were successfully processed to L2A. The remainder had an anomaly which halted the processing (Section 3.3). Stringent site specific quality checks were applied to the L2A data, and a total of 4,256 sequences were provided on the Zenodo database.

For the WATERHYPERNET, there were 55,514 sequences acquired. 44,412 of these (80%) were succesfully processed to L2A. Stringent site specific quality checks were applied to the L2A data, and a total of 4,931 sequences were provided on the Zenodo database. Note that the data provided on Zenodo are still a subset of the entire dataset and that the number of processed and quality checked sequences is expected to increase since the first version of the processor has been considerably improved (e.g., additional and more appropriate flagging, processing issues and bugs have been solved, accounting for different relative azimuth angles, retrieving wind speed from a different data source, i.e., NCEP/GDAS) to the current v2.0 of the hypernets_processor. ¹⁷