The goal when designing the PlotCam was to make an instrument that could support several cameras and sensors at one time, was compact enough to fit on the back seat of a vehicle, and was easily and rapidly assembled once the operator got to the experimental location. The system needed to be light weight to minimize operator fatigue when it was carried from plot to plot. The unipod of the PlotCam, supporting the sensor arm, was constructed from two pieces of hollow, square stock carbon fiber (26 by 26 mm ID Clearwater composites, Deluth MN, United States). The top and bottom pieces, which came apart to facilitate transport, were 940 and 840 mm long, respectively (Figure 1A). The top portion of the unipod contained another piece of square, hollow carbon fiber (20 by 20 mm OD) that telescoped from the larger portion to adjust the height of the sensors. The sensor arm was a rectangular piece of carbon fiber (52 by 26 mm ID and 390 mm long) with a hinge at the joint which allowed the sensor arm to be folded back over the leg of the unipod. For ease of transport, the instrument was designed to conveniently fit into the trunk of a vehicle with the unipod disassembled into the top and bottom sections, and the sensor arm folded over the unipod leg (Figure 1B).

All of the joints connecting the pieces of carbon fiber were reinforced with square stock hollow aluminum that was milled to fit snugly inside the carbon fiber of the PlotCam unipod. The foot of the unipod was constructed from a piece of flat stock aluminum connected to a piece of square stock and fixed at a 60^o angle. When the sensor arm was unfolded and locked in place, and the angled foot of the unipod placed flat on the ground, the cameras were positioned at nadir over the plot (Figures 1A, C).

A short wooden picket, placed in the centre of a field plot adjacent to the range-way provided a marker for the operator to align the foot of the PlotCam, ensuring that the sample area remained consistent on a temporal basis (Figure 1C). When the aluminum foot was placed against the wooden picket and the sensor head placed at a nadir position over the plot, sensor height could be adjusted between 1.6 to 2.2 m above the plot.

To protect the cameras they were mounted in a removable camera head made from a milled piece of aluminum block that fit into the end of the rectangular carbon fiber sensor arm and was attached to a piece of flat stock aluminum (5 mm thick × 70 w × 180 L mm) with cut outs and mounting brackets that kept the cameras level with the sensor arm and in the nadir position (Figure 1D). The camera head was held to the sensor arm with a clip pin that went through the arm and the aluminum block. To shade and protect the instruments, a 3 mm thick Plexiglas housing (78 w × 180 L × 160 h mm) fit over the camera mounting plate attached with four thumb screws. The housing was covered in 5 mm thick aluminum bubble heat shield (Cool Shield, Granger, Can.). A hinged access door on the side of the housing allowed the operator access to the cameras.

An instrument box mounted on the unipod, housed an Intel^® I5 NUC computer (NUC5i5RYK, Intel, Santa Clara, CA United States), with a 524 GB solid state drive which ran Windows 10 OS and the PlotCam software (Figure 2). The computer had 4 USB ports and an HDMI port. To increase the number of USB ports, two 4-way USB hubs were connected to the computer. USB devices included a wireless handheld trackball mouse (EIGIS 2.4 G Air Mouse, China), a type-K thermocouple mounted on a USB board (Phidget 1051, Calgary, Canada), and an accelerometer (Phidget 1041). The other USB hub was used to power the 18 MP RGB camera (Sony DSC-HX90V, FOV 70^o x 50^o, Sony, Canada) an external USB port mounted on the outside of the instrument box, a 0.3 MP stereo depth camera (Intel RealSense D415 FOV, 69^o × 42^o, Intel, Santa Clara, CA, United States) and a pyranometer (Apogee SP215SS, Apogee Instruments, Logan, UT, United States) interfaced via a USB interface board (Phidget Interface Kit 2/2/2). The 0.3 MP infrared camera (ICI 9640, FOV 50^o x 40^o, ICI - Beaumont, TX United States) was connected to a USB port on the computer. The RGB, infrared and depth cameras were positioned in the head so that the fields of view overlapped facilitating segmentation (Figure 1E). When the height of the sensor arm was changed to accommodate different plot widths the images scaled without changing the overall relative layout.

A 12.8 V 5Ah lithium iron phosphate battery (LiFePO₄, RELi³ON relionbattery.com, Rock Hill United States) carried in a utility belt by the operator powered the PlotCam (Figure 1C). An HDMI headset was used to view the computer display, interact with the custom PlotCam software as well as view a live video feed through the RGB camera. Other sensors and plot metadata were managed using the wireless mouse and an on-screen keyboard. The operator could see enough through the bottom of the headset to navigate around the field plots and place the camera in the plot with the unipod foot against the wooden picket. At 6.8 kg in total weight, the PlotCam was optimised to reduce operator fatigue.

The PlotCam operating software was written in Visual Basic.net (VB, Microsoft, WA United States). We chose Visual Basic primarily because of the availability of application program interfaces (APIs) and software development kits (SDK) written for the RGB, depth and infrared (IR) cameras and the Phidget devices as well as the ease of creating a graphical user interface in VB.net. The Sony RGB “Camera Remote API” the Intel D415 depth camera (Realsense SDK 2.0) and the ICI 9640 IR camera (ICIx64 SDK) were incorporated into our software and facilitated camera control and data collection. Software development was an evolutionary process with modules added as the camera grew in sensor number and complexity.

The PlotCam software had a main graphic user interface window, two additional windows and two dialog boxes. On start-up, the main window showed connection status of the peripheral devices including; the accelerometer, the light sensor, the air temperature (AT) sensor, the battery voltage, and the zoom percentage of the RGB camera (Figure 3). Live feed was available from the AT and light sensors, the D415 depth camera, and a live video feed through the RGB camera. The operator used this information to check all of the sensor and camera status on start-up and throughout operation.

The D415 depth camera initiated at start-up. The operator could set the distance to the ground over a flat surface by activating a depth camera dialog box using the “Plant Height Zeroing” button. This process was done in the lab over a flat surface, based on the width dimension of the plots used for imaging. The telephoto zoom on the Sony camera could be adjusted with the “Zoom Adjust” dialog box and the air temperature sensor was calibrated with the “Calibrate Temp Sensor” dialog box.

A check box on the main window activated the IR camera and brought up the IR camera popup reminding the operator to conduct a non-uniformity correction. To ensure the IR camera was in focus, the “View IR Imaging” button opened a window to view a live video feed, permitting manual focus of the camera as necessary.

The “FileName” button activated an onscreen keyboard window used to enter the metadata required to identify the plot details including the experiment name, day of the year, and plot number. The accelerometer, represented on the main screen by a green “bullseye” floating in blue target rings (Figure 3), was mounted in the sensor arm.

Field topology is a necessary consideration when choosing research sites for phenomics. While the integrated accelerometer and design of the PlotCam ensured the that the cameras were maintained at the nadir angle, in cases of extreme topography, such as hills, additional post-processing may be required.

To collect phenomic data on a plot, the operator placed the unipod foot in line with the wooden stake bisecting the plot (Figure 1C), used the RGB video image of the plot displayed on the HDMI headset to centre the image within the plot, and by slight adjustments of the angle of the unipod leg aligned the green circle in the middle of the blue accelerometer target (Figure 3). Once the plot was properly aligned, and the bullseye centered on the accelerometer circles, the operator clicked “Take Picture.” The software required the accelerometer to be centred prior to initiating data capture. After data capture had been initiated the Sony 18 MP RGB image was taken and stored to the camera secure data (SD) card in JPEG format, the D415 image (0.32 MP) was stored to the computer solid state drive in JPEG format, and the D415 depth and IR camera T profiles were each stored in linear arrays in a tab separated text file format. Simultaneously, air T, incoming light, the time stamp, an instantaneous estimate of average height and the camera meta data were stored to a text file along with the image name on the memory card and the plot identifiers entered by the operator. Once all of the data was stored (approximately 2 s), an audible beep informed the operator to move to the next plot. The plot number in the metadata was automatically increased by one. In the case of mistakes the operator could simply re-enter the plot identifier using the “FileName” button, retake the photo and continue.

Naming structure was optimized within the system to limit the possibility of data being overwritten or lost. The Sony HX90V camera has limited flexibility in naming structure, so the images were named in series on the memory card. The name as stored on the memory card was obtained from the camera and written to the corresponding line of the tab-delimited text file from that image. The D415 RGB image, the depth array file, and the IR T array file were named using a letter to define the data type (V for RGB, D for depth, and I for infrared), were named using a letter prefix to define the data type (v for RGB, D for depth, and I for infrared), the name of the Sony HX90V image in the memory card, the experiment, the date and the plot number. Files from the PlotCam computer were transferred to working drives using external hard drives, while the images from the Sony RGB camera were transferred by direct USB connection. The downstream analysis of image and sensor data was done using scripts written in Python v.3.8 [15].

Plot RGB images were segmented using a multi-feature machine learning model [16], which output the vegetative cover (VC) (plant pixels/total pixels) for each image based on distinguishing crop pixels from background pixels in the image. Plot IR arrays were segmented using a script based on this algorithm. The IR arrays were overlaid on the RGB images and the non-crop T were removed from the array. The physical location of the cameras on the sensor head (Figure 1D) was such that the IR camera field of view fell within the Sony RGB image field of view to facilitate segmentation as can be seen in Figures 4A, E. The remaining canopy temperatures (CT) were saved as a CSV file that was imported into a Python script to calculate average CT. The air T (AT) was measured every time an IR T was saved. The average CT per plot were normalized by subtracting AT from each plot (CT-AT).

To calculate plant height (HT) we used the methodology described in Morrison et al. [17]. Briefly, depth arrays from the D415 camera were filtered for extreme values, and to remove sensor noise from the matrix. The depths were converted into canopy HTs by subtracting the height of the D415 camera above the ground from each depth. From the canopy HT arrays we calculated the average canopy HT, the maximum HT, the averages of the maximum 1% and 5% of canopy HT of the plot, and the canopy HT variance.

The PlotCam was tested on soybean moisture stress tolerance trials at the Ottawa Research and Development Centre (75^o43″W, 45^o22′N), Agriculture and Agri-Food Canada (75^o43″W, 45^o22′N) in 2020 and 2021. In each year, the experiment screened 32 potential cultivars grown in a split-plot design with two moisture treatments and 4 replications, for a total of 256 plots. Cultivars differed between sampling years. Each plot had 4 rows of soybean spaced 40 cm apart seeded at a density of 55 plants per m² in plots 1.6 m wide and 3 m long. Seed was inoculated with Bradyrhizobium japonicum. The different moisture treatments were natural precipitation and natural precipitation augmented with drip tape irrigation (Toro Aquatraxx Drip Tape, El Cajon, CA, United States). Seeding was done with a disc seeder equipped with a draw behind drip tape applicator that placed the drip tape 11 cm below the soil surface. An irrigation tape was placed between two soybean rows resulting in two tapes per plot. Each tape emitter delivered 1.14 L h^-1 and on a per plot basis 30 min of irrigation resulted in an equivalent delivery of 2.4 mm of precipitation. Irrigation began in the early vegetative stage, was turned off during the growing season temporarily when daily precipitation exceeded 20 mm, and was turned off permanently when the seeds began to ripen. Plots were seeded on May 29 and 21 in 2020 and 2021, respectively, into Matilda sandy loam (Cryochrepts, Eutrochrepts, Hapludolls Can. classification). Pursuit herbicide (2-[4,5-dihydro-4-methyl-4-(1-methylethyl)-5-oxo-1H-imidazol-2-yl]-5-ethyl-3-pyridinecarboxylic acid) was applied pre-plant incorporated at commercial rates to control broadleaf and grass weeds and during the growing season the weeds were removed manually.

Phenomic data was collected on the soybean plots weekly from emergence until ripe when cloud cover was consistent; either complete cloud or full sun. Data was collected after solar noon to achieve the best CT-AT differential. The sensor head was set at a height of 1820 mm above the ground which resulted in a resolution of 362 pixels cm^-2 for the Sony RGB camera, 11 pixels cm^-2 for the D415 RGB camera, 12 canopy depth measurements cm^-2 and 14 T measurements cm^-2 for the Intel RGB and IR cameras, respectively. This working distance allowed imaging of all 4 plot rows.

Crop Heat Units [18] were calculated from seeding to harvest and used as a measurement of thermal time. Using accumulated thermal time instead of days allowed for a greater comparison among years for phenomic traits. At harvest, the plots were combine harvested, the seed cleaned, weighed and adjusted to 13% moisture by weight.