The implementation of optical monitoring is the process of setting up a camera or multiple cameras in stable locations looking at a “target” contained in the structure and deriving the structural motion information through tracking the target motions in image sequences. “Target” here could be either artificial feature (pre-installed marker, LED lamp, or planar panel with special patterns) or existing structural feature (e.g., bolts or holes).

The implementation comprises hardware and video-processing components. The hardware comprises camera(s), a computer with video-processing package, and accessories such as camera mount, camera lens, and optional artificial targets. The role of the video-processing package is acquiring the video frames covering the target regions, tracking the target locations in image sequences, and finally transforming the target location information in image into the time history of structural displacement.

For various vision-based systems in the literature, the hardware is almost the same [except the mixed systems, e.g., combing the camera system with total station (Ehrhart and Lienhart, 2015)] while the main difference is in the video-processing packages. A typical video-processing package can be decomposed into three components according to the procedures of measurement goal realization.

The first step is camera calibration to obtain transformation metrics using methods such as “scaling factor,” “planar homography,” and “full projection matrix.” It is aimed at determining the transformation metric between the image natural units (pixels) and the real world units (e.g., millimeters).

The scaling factor method (Stephen et al., 1993) is the simplest. It links the image motion to the structural displacement via a coefficient estimated by camera-to-target distance or one dimension correspondence in the image plane and in the structural coordinate system. This method, however, is based on the assumption that the camera principal axis is perpendicular to the target surface plane, which sets constraints on the camera-to-target geometry relation on site. The planar homography method (Xu et al., 2016) calculates the transformation relationship between the 2D image plane and the 2D target surface plane in the structure. This method considers projection distortion under non-perpendicularity of the camera principal axis but neglects out-of-plane motion of the target. The planar homography matrix could be determined by direct linear transformation (2D DLT) based on at least four sets of 2D-to-2D point correspondences (Hartley and Zisserman, 2003). The full projection matrix (Chang and Ji, 2007) is the general form to build the projection from the 3D structural coordinate system to the 2D image plane with no assumption, and supports the reconstruction of 3D structural displacement. The full projection matrix is usually calculated from multiplication of the two camera matrices that are determined separately (Chang and Ji, 2007; Martins et al., 2015): (i) laboratory calibration to determine the camera intrinsic matrix and (ii) site calibration to determine the camera extrinsic matrix.

The second step of target tracking is aimed at determining the target locations in a frame sequence in a video record using target matching or motion estimation techniques in computer vision. Template matching (Stephen et al., 1993) is a classical technique for target tracking through matching a rectangular subset region between two images using a similarity metric. Optical flow estimation detects motion or flows of each pixel within the target region based on one temporal constraint about image properties [e.g., brightness constancy in Lucas–Kanade method (Tomasi and Kanade, 1991) or local phase constancy in phase-based method (Fleet and Jepson, 1990)]. Feature point matching involves matching salient feature points in two images based on their local appearance [i.e., local feature descriptor (Szeliski, 2010)], e.g., Fast Retina Keypoint matching (Khuc and Necati Catbas, 2016).

The last step of structural displacement calculation delivers the structural displacement sequence using single or multiple cameras. It aims to derive the structural displacement given the image location sequences of the targets (which are the output of the target tracking step) and a transformation metric (the output of the camera calibration step).

For single camera applications using scaling factor or planar homography matrix as the transformation metric, the 2D structural displacement is derived uniquely, whereas for multiple camera applications using full projection matrix as the transformation metric, a least squares estimate is required to extract the 3D structural displacement by solving the underdetermined linear equations. Some researchers attempted to extract more information about target motion (up to 6 DOF) from single camera using the pose estimation technique (Chang and Xiao, 2010; Greenbaum et al., 2016) in which target position and orientation in structure is estimated by tracking multiple target points (at least four) in a rigid target.

The Vision System originating from research at the University of Bristol (Stephen et al., 1993; Macdonald et al., 1997) led to the “Video Gauge” software that was commercialized via the university spin-out Imetrum formed in 2003. The implementation of the Video Gauge in a hardware platform is called the Dynamic Monitoring Station (DMS) which includes one or more GigE high performance cameras.

A typical implementation of the DMS for bridge monitoring is shown in Figure 1. High-resolution cameras equipped with long focal length lens are connected to the controller (computer) via Ethernet cables and a group of up to four cameras are available for time-synchronized recording and real-time video processing. Cameras used for case studies in this paper have a resolution of 2,048 × 1,088 pixels and sensor size of 11.26 mm × 5.98 mm.

In the video-processing package, target tracking algorithms used are correlation-based template matching and super resolution techniques (Potter and Setchell, 2014) which enable better than 1/100 pixel resolution at sample rates beyond 100 Hz in field applications. The tracking objects could be either artificial targets or an existing feature (i.e., bolts or holes) on the bridge surface. The system supports either 2D structural displacement measurement by single camera or 3D structural displacement measurement by multiple cameras. In this study, single camera configuration was used to extract the 2D bridge displacement along the vertical and transverse directions at bridge mid-span and planar homography method based on coplanar dimension correspondences as used in camera calibration.

The system used by University of Exeter has been trialed in several 1-day field campaigns on a number of bridges in the UK, particularly Humber and Tamar Bridges. Such a 1-day field campaign on the Humber Bridge is described in the next section.

The Humber Bridge (Brownjohn et al., 2015), opened in 1981, has a main span of 1410 m and is known to experience mid-span deformations around 1 m in strong winds (Brownjohn et al., 2015).

A long-term monitoring system has been operating at Humber Bridge since 2010 (Brownjohn et al., 2015) and includes a base station at the bridge tower, two GPS rover receivers (Leica GMX902) mounted on the main cables at mid-span and three QA750 accelerometers mounted inside the steel box girder at mid-span (two vertical and one lateral/horizontal) (Brownjohn et al., 2015). The sample rates of GPS and QA accelerometers in the monitoring system are 1 and 20 Hz, respectively.

A single day of field testing using the DMS was performed to measure the lateral and vertical displacement at mid-span on a clear mid-summer day with low winds and normal traffic load. The DMS performance was evaluated in time and frequency domains by comparing with a GPS “reference sensor.”

In the field test, the camera and controller along with battery power supply were located near the foundation of the North (Hessle) tower shown in Figure 2A, to the East of the pylon. A concrete plinth built between the pylons for the 1990s Vision System deployment (Stephen et al., 1993) was not used due to poor sightline to the target attached to the bridge parapet. A 300 mm f/2.8 lens was attached to both camera and tripod via a rigid double-support translation stage (described in a later section of the paper). A custom-made 1 m square steel frame holding an artificial target was mounted on the parapet at the mid-span 710 m from the lens as shown in Figure 2B. The pattern of the target is a set of concentric rings with a gradual blend from black to white at the edges.

The frequency range of interest containing the majority of significant vibration modes was less than 1 Hz, so the frame rate of the DMS system was chosen as 10 Hz. To save storage space, the image size of each frame was saved as 850 × 400 pixels although the default image size is 2,048 × 1,088 pixels.

Both the custom-made target and a natural feature target at mid-span were tracked. Figure 3 shows a single captured video frame. The red dashed boxes in the figure include the custom-made target and the natural target. The latter comprises ribs of the box deck on the bridge soffit and while it is judged to be close to the artificial target it could be at a spanwise location differing by a few meters. To transform the image natural units (pixels) to the real world units (e.g., millimeters), a transformation metric reflecting the geometric relation between the 2D image plane and the 3D structural coordinate system is required, and planar homography method for camera calibration is used in this application. With the knowledge (Brownjohn et al., 2015) that the out-of-plane motion along the longitudinal direction of the bridge is negligible, the planar homogrpahy matrix was estimated based on four coplanar line correspondences between the 2D image plane and the 2D target surface plane. The lines with known dimensions came from the edge and diagonal of the installed artificial target frame.

During the test, a D-SLR camera was adapted to video-record traffic on the bridge. Figure 4A shows one captured frame from the video recorded when two heavy good vehicles (trucks) approached each other at mid-span from opposite directions. Figure 4B shows the corresponding measurement using the DMS in the vertical direction, with vertical deflection at mid-span caused by the two vehicles reaching 221 mm. In general, the measurements from tracking each of the two targets agree well; the DMS demonstrates similar performance for tracking either target.

Displacement from tracking the custom-made target was used for comparison with the displacement data from one of the mid-span GPS receivers. Figure 5A shows the vertical displacement measurement by DMS and the GPS receiver; the cross correlation coefficient of two signals is calculated to be 98.8%. Figure 5B shows a zoom-in view of 1-min of data. Consistent with reported GPS standard accuracy of 35 mm in vertical direction (Nickitopoulou et al., 2006), the accuracy of the GPS observation at Humber was at the centimeter level. For vision-based systems, it is hard to quantify the measurement accuracy on site since the true bridge motion is unknown. One approximate way to estimate the measurement accuracy is from target tracking accuracy using the scaling factor method (1)IdispSdisp=fpixelD where I_disp and S_disp are the target motions in the image plane (e.g., pixel) and structural system (e.g., millimeter); f_pixel is the focal length of camera lens in terms of pixel units corresponding the focal length in terms of millimeters scaled by the camera sensor resolution; and D denotes the distance between the camera optical center and the target surface plane.

The nominal resolution of I_disp in target tracking can be better than 0.01 pixel while the reported accuracy varies from 0.5 to 0.01 pixel (Bing et al., 2006) which is related to target pattern (texture contrast) (Busca et al., 2013) and illumination condition (reported in Section “Change of Target Pattern Due to Shadow and Illumination”). In this application, given the focal length of 300 mm, the camera sensor resolution at 0.0055 mm/pixel, and the camera-to-target distance at approximately 710 m, the accuracy of 0.1 pixel (artificial target of high-contrast pattern) in image plane corresponds to an accuracy (or rather resolution) of 1.3 mm in the structural system.

The power spectral densities of the DMS displacement signal, the GPS displacement signal, and the acceleration data were obtained by Welch’s method as shown in Figure 5C. From the previous studies on the bridge, it is known that (vertical) modal frequencies exist at 0.117, 0.31, and 0.46 Hz (Rahbari et al., 2015). The DMS captured the first and second vertical modal frequencies while the GPS captured only the first one. In theory, the DMS and GPS measurement sampled at 10 and 1 Hz, respectively, have the chance to capture modal frequencies in the range 0–0.5 Hz, but in fact they have failed. This is because the displacement induced by vehicle loads is always dominated by the static and quasi-static components while the dynamic component of displacement is relatively small (i.e., the root mean square of de-trended acceleration signal during this time interval is only 0.0016 g) and easily contaminated by the measurement noise. It indicates that the displacement sensor (either DMS or GPS) has the capacity to capture the dominant frequency component but might fail to capture some frequency components lower than the Nyquist frequency.

The Imetrum DMS is validated to be a mature, accurate, and stable optical system for bridge deformation measurement over long ranges and over several hours within a day, but it is a proprietary solution and there remain open various research routes to wider applications and lower costs in non-contact sensing. This section considers these routes in terms of concept, procedures, and the potential for improvement.

In the field applications at Humber Bridge and Tamar Bridge, one single GigE camera with 300 mm lens was used for recording. During the video processing, correlation-based template matching (Potter and Setchell, 2014) was used for target tracking and planar homography method was used for camera calibration to determine the transform relation between the image plane and the structural system. The final output was the two-dimensional structural displacement along the vertical and transverse directions at bridge mid-span.

Regarding hardware component of an optical system:

Professional high-resolution cameras equipped with long focal length lens used in the DMS are necessary for long-range monitoring. For short-distance monitoring, consumer-grade cameras or smartphones could be alternatives reducing system cost. Consumer-grade cameras have been validated as feasible for displacement measurement and system identification in laboratory testing (Yoon et al., 2016), but reports of field implementations are hard to find.

Regarding the video-processing methodologies: (1)

Camera calibration is aimed at determining the transformation metric between the image natural units (pixels) and the real world units (e.g., millimeter). The scaling factor method using the camera-to-target distance (Khuc and Necati Catbas, 2016; Yoon et al., 2016) or merging optical system with a total station (Charalampous et al., 2015; Ehrhart and Lienhart, 2015) has no requirement about known geometric information. However, these applications are based on the prerequisite that the camera principal axis is perpendicular to the target surface plane and are thus not suggested. The general form of transformation metric (i.e., planar homography matrix or full projection matrix) is related to camera internal features (i.e., focal length, principal point locations, etc.) as well as the camera-to-target geometric relation (i.e., position and orientation of camera in structural coordinate system). Parameters describing camera internal features (i.e., camera intrinsic matrix) can be determined in the laboratory ahead of a field test, i.e., using the camera to observe a planar calibration object in a few different views (Zhang, 2000). However, determination of camera position and orientation (i.e., camera extrinsic matrix) requires some geometric information from the structure (at least four coplanar point locations). The points with known locations are usually provided by attached artificial targets, e.g., planar chessboard target (Chang and Ji, 2007), planar T-shape wand with active markers attached (Park et al., 2015), or 3D calibration object with four non-planar active targets (Martins et al., 2015). To achieve complete contactless sensing, efforts should be spent on alternative means to obtain dimension information in structure, e.g., through surveying or structural design drawings.

Based on optical system concepts, structural motion monitoring is possible with a single consumer-grade camera and a custom-developed video-processing package. This section shows a simple field implementation of a non-proprietary optical system for cable vibration monitoring to Miller’s Crossing cable-stayed footbridge in Exeter, UK (Figure 14).

One of the shorter footbridge cables is known to vibrate due to pedestrian traffic, so pedestrian excitation (jumping at the 2.4 Hz vertical natural frequency) was used to obtain the dynamic parameters of the cable.

Contacting sensors for cable vibration measurement, e.g., accelerometers and strain gauges require troublesome direct access to the cable at height whereas a non-contact optical system offers the possibility of quick, easy, and economic measurement. In this application, a consumer-grade camera was used to record the video that was post-processed using video-processing code custom-developed in MATLAB. A Gopro Hero4 Session video camera was mounted on a tripod 30 m away from the bridge for recording; a sample frame is shown in Figure 14A. Since the monitoring purpose is only for system identification, specifically estimating modal parameters of cable vibration, and exact vibration values are not required (Kim and Kim, 2013; Chen et al., 2015b), the video-processing package includes only the target tracking step to extract the cable motion projected in image. The image includes salient distortion due to the wide angle lens, thus the distortion was removed ahead of target tracking using offline camera calibration (Chang and Ji, 2007) and the corrected image is shown in Figure 14B.

The target tracking algorithm is based on edge detection using the Sobel–Zernike moment operator (Ghosal and Mehrotra, 1993; Ying-Dong et al., 2005) with a region of interest including a small cable segment selected for tracking. An arbitrary direction, e.g., normal to an identified edge was assumed as the cable motion direction and the distance between identified edges within two frames corresponded to the cable motion projected in the image which inherently has subpixel resolution. Even if motion direction is not transverse to line of sight there is no influence on the identification of cable natural frequencies.

Figures 15A,B show time histories of cable motion, with the maximum motion in the image estimated to be 0.583 pixel. The corresponding power spectrum density is presented in Figure 15C identifying peak frequencies at 2.53, 5.03, and 7.6 Hz. These values could be used with known length and mass properties to estimate cable tension.

This example of a simple non-contact optical system for cable vibration monitoring shows the potential for vibration measurement of other slender line-shape structural components, e.g., transmission towers using consumer-grade cameras.