A novel cross-domain identification method for bridge damage based on recurrence plot and convolutional neural networks

2.1. Fundamentals of recurrence plots

The recurrence plot (RP) is a visualization tool used to transform a time series data into a 2-D matrix that compares each point in a two-dimensional space to the others. Each point in the matrix represents the degree of similarity between two moments in time, typically depicted using binary or grayscale values. RP helps intuitively understand the repetitive patterns and dynamic features hides in the time series, making it extensively applicable in fields of signal processing, time series analysis, and dynamical system modeling.

For time series $x=\{x_1,x_2,...,x_n\}$, the first step in constructing the RP is to reconstruct the equivalent phase space $X$ based on the time delay $\tau$ and embedding dimension $m$. The time delay $\tau$ and embedding dimension $m$ can be determined through embedding dimension theory and time delay methods:

where, $N$ is the trajectory length of the reconstructed phase space, and $n$ is the number of time series

In phase space $X$, the recurrence plot of time series x is constructed according to Eq. (2):

where, $R_{i,j}$ is the value at the position $\left(i,j\right)$, $\epsilon$ is a predefined threshold distance, $‖\cdot ‖$ indicates the Euclidean distance between phase points in the reconstructed phase space. $\mathrm{\Theta }(\cdot )$ denotes the Heaviside function, which can be expressed below:

where, $S_{i,j}$ represents the distance calculated between points $X_i$ and $X_j$ in the reconstructed phase space. $R_{i,j}=1$ represents a black point at the position $\left(i,j\right)$ and $R_{i,j}=0$ represents a white point at the position $\left(i,j\right)$.

The time delay $\tau$ can be determined by the mutual information method [30, 31]:

where, $p\left[x_i\right]$, $p\left[x_j\left(\tau \right)\right]$ are the single probability densities associated with the time series $x_i$ and $x_j\left(\tau \right)$, $p\left[x_i,x_j\left(\tau \right)\right]$ is the joint probability density associated with the time series $x_i$ and $x_j\left(\tau \right)$. The first local minimum of the mutual information curve is the optimal choice of $\tau$.

The minimum embedding dimension $m$ can be determined by the false nearest neighbors [32]. With the time delay $\tau$ and the time series $x=\{x_1,x_2,...x_n\}$, an $m$-dimensional state space vector $X=\left\{X_1,X_2,\dots ,X_i,\dots ,X_N|X_i=\left(x_i,x_{i+\tau },\dots ,x_{i+\left(m-1\right)\tau }\right)\right\}$ is constructed. $X^{\text{'}}$ is denoted as the $r$th neighbor of $X$, the distance between them is defined as:

As the dimension of the embedding space increases from $m$ to $m+1$, the distance between $X^{\text{'}}$ and $X$ can be expressed as:

The relative increment of the distance between $X^{\text{'}}$ and $X$ is calculated as:

where $R_{tol}$ is the error threshold. When Eq. (8) is satisfied, points that were previously considered as neighbors now exhibit their true distances upon increasing the embedding dimension, suggesting that these points are not actually close in the higher-dimensional space. Such points are referred to as false nearest neighbors. The false neighboring point is easier to identify when $R_{tol}\ge$10 during the calculation process. When $r=$1, it is called the nearest neighbor point. With the increase of the embedding dimension, when the number of false nearest neighbor points tends to 0, the embedding dimension currently is determined as the minimum embedding dimension $m$.

To avoid the influence of the selection of thresholds and enhance the ability to handle multidimensional data, the traditional RP is modified.

The joint RP considers the recurrences of multiple variables’ trajectories within their respective phase spaces. The joint RP compares the similarity of recurrence natures among multiple dynamical systems. The joint RP of two signals is taken as an example, $X$ and $Y$ are the reconstructed phase spaces, joint RP is described as:

The un-threshold RP calculates recurrences directly based on the similarity of time series data without presetting a threshold. The construction of the un-threshold RP is as follows:

where $D_{i,j}$ represents the distance between points $X_i$ and $X_j$ in the time series, which is typically calculated using the Euclidean distance.

In un-threshold RPs, each $D_{i,j}$ value directly affects the brightness or color of the corresponding pixel in the plot. Typically, smaller distances result in brighter pixels, indicating greater similarity between the states at two time points. The un-threshold RPs allow the intuitively repeating patterns and dynamic structures in time series data easily to be identified without the requirement of selecting an appropriate threshold.

2.2. Convolutional neural networks

Convolutional Neural Networks (CNNs) are deep learning models inspired by the biological visual cortex. Aiming at simulating the human visual system for feature extraction, CNNs are primarily used for processing image data. As shown in Fig. 2, the deep learning network framework consists of an input layer, hidden layer and output layer. The hidden layer includes convolution layer, pooling layer and fully connected layer. The architecture of CNNs allows for parallel computing which can enhance computing efficiency.

CNNs are a type of multilayer feedforward neural network named for their use of convolution operations. CNNs extract and abstract image features effectively, making them suitable for tasks like image classification and object detection. Through convolution and pooling operations, CNNs achieve invariance to the positioning of targets within images. CNNs are widely applied and perform excellently in image processing and computer vision.

To ensure the network can capture the nonlinearities and complex dependencies in the input data, and to perform a variety of complex prediction and classification tasks, activation functions are set up in CNNs. The activation function used in this study is the Leaky ReLU, which addresses some issues associated with the standard ReLU function, such as failing to activate neurons from negative inputs and non-centered outputs. Leaky ReLU introduces a leaky value for negative values, ensuring that the count backward is never zero and thus resolving the problem of neurons failing to activate with negative inputs. The construction of the ReLU is as follows:

Non-linear activation functions add non-linearity to convolutional neural networks, enabling them to approximate virtually any function.

Fig. 2Architecture of deep learning network

2.3. Damage identification method

The differences between the numerical model of a suspension bridge and its physical model exist, despite the modal modifications to the numerical model. A CNN trained only on data from the numerical model may perform suboptimal when recognizing data obtained from physical model experiments, resulting in insufficient accuracy in cross-domain structural damage identification. To address this problem, this study proposes a cross-domain CNNs fusion (2D-CNNFs) method based on transfer learning [33] to release the single-domain data training and difficult cross-domain identification.

The domain $D$ in transfer learning, the subject space of knowledge learning, can be defined as:

where, $\chi$ and $P\left(X\right)$ are the feature space and edge probability distribution, respectively.

The task of knowledge learning $G$ in transfer learning can be defined as:

where, $f(\cdot )$ and $Y$ are the implicit characteristic function and label space, respectively. There are two types of agent space, active domain and target domain, and the probability distribution at the edge of agent space is different, which is expressed as $P^s\left(X\right)\ne P^t\left(X\right)$.

The source task $G^s$ is a labeled source domain data $D^s$, which can be represented as:

where, $x_i^s$ is the $i$th data in the source domain feature space, $y_i^s$ is the $i$th label in the source domain space, and $n^s$ is the number of samples in the source domain space.

The target task $G^t$ is an unlabeled target domain data $D^t$, which can be expressed as:

Due to the similar features in the source domain and the target domain, transfer learning makes full use of the samples in the source domain $D_i^s\left|{\mathrm{}}_{i=1}^{n^s}\right.$ to fit a decision function $f(\cdot )$. Therefore, the $f(\cdot )$ can not only represent the mapping relationship between data and labels in the source domain, but also can be transferred to the target domain to express such mapping relationship and predict labels.

By comparing data across domains, common or similar features are found and thereby the transfer learning to enhance the network model can be applied. The trained network can generalize and apply the learned insights in different contexts to solve cross-domain structural damage identification problems [34, 35]. Based on the 2D-CNNs model and the idea of transfer learning, a cross-domain data feature cross transfer learning network model is constructed to achieve cross-domain structural damage identification. The flowchart of this method is illustrated in Fig. 3, and the detailed steps are drawn below.

Step 1: collecting accelerations of the physical model and the modified numerical model forced by frequency-swept excitations to obtain data set. The collected accelerations are extracted valid data points during the excitation period to form an 8-column by 1000-row numerical matrix.

Step 2: constructing the un-threshold multivariate RPs samples based on both the physical and numerical models.

Step 3: training the 2D-CNNs model using the numerical model-based RPs samples until the loss function stabilizes while the trained 2D-CNNs model is validated using 1/9 of the RPs samples.

Step 4: the set of physical model-based RPs samples is split into training and validation sets in a 1:1 ratio. The initial layers of the trained 2D-CNNs are frozen, and the end layers with the physical model-based samples are trained and validated to form the2D-CNNFs.

Step 5: inputting the un-threshold multivariate RPs from the physical model into the 2D-CNNFs for realizing the cross-domain structural damage identification.

Fig. 3Flowchart of damage identification method based on RPs and 2DS-CNNFs

3.1. Physical model

The load-bearing process of a suspension bridge is to transfer the external load from the deck to the slings and then to the main cables. The main cables, towers, and anchors constitute the primary load-bearing structure of a suspension bridge. The main cables distribute the shear forces to the bridge tower and transmit the axial forces to the foundation through the anchors connected to the ground. Other components of a suspension bridge include the stiffening girders of the bridge deck, slings, and saddles.

As shown in Fig. 3(c), the scaled model of a suspension bridge is prepared. The primary components selected for manufacturing the model include perforated aluminum plates, steel ropes, steel towers, clamps, and buckles. The main cables of the model are connected to the ground using buckles. The towers of the model and the work platform, as well as between the bridge deck and the main towers, are bolted together. The main cables are anchored at the top of the towers in notches to secure them. Clamps are used at the connections between the slings, main cables, and the bridge deck. The detailed dimensions of the physical model are listed in Table 1.

3.2. Numerical models

An equal-size finite element model of the suspension bridge was created using Abaqus software. This finite element model consists of four main components: the main cables, bridge towers, slings, and the bridge deck. Fixed constraints are set up at the bottom of the towers, between the bridge deck and the towers, between the tower and the main cable, and between the main cables and slings. Coupling interactions are established between the bridge deck and the slings. Translational displacement is fixed between the main cables and the foundation, while rotational directions are simulated with spring constraints with a spring stiffness of 1×10⁷N/m to mimic the physical model’s buckle. As the main cables and slings are the primary load-bearing components of the suspension bridge, they initially suffer tensile forces when subjected to constant loads. To accurately model the bridge’s operational state, axial prestress with $F_2=$ 20 N and $F_1=$1 N are assigned to main cables and slings, respectively. The material parameters of the numerical model are outlined in Table 2.

During the discretization the finite element model, the bridge towers and deck are simulated using eight-node 3-D solid elements, specifically the C3D8R element in Abaqus software. The main cables and slings are simulated using truss elements T3D2, which is good at simulating the tensioned structural characteristics of components. The diameters of the main cables and the slings are set as 4 mm and 2 mm, respectively. The deck thickness is divided into four layers of mesh, and the mesh density is increased at various connection points and at the grooves at the top of the towers. The discretized model comprises a total of 5376 elements and 8591 nodes.

Fig. 4Numerical models of the suspension bridge: a) geometric model, b) discretized model

a)

b)

3.3. Model modification

Seven frequencies of the physical model are identified from the frequency response functions derived by the modal testing data. The obtained seven frequencies are checked as the vertical bending and torsion modes of the model. For consistency, the modal order in the previous 50 orders of modal analysis of the numerical model is extracted and compared. The model modification based on parameter identification is employed to improve the fit between the finite element model and the physical model of the suspension bridge. A surrogate model is used to approximate the mechanical behavior of the numerical model while enhancing computational efficiency.

Among many optical algorithms, support vector regression (SVR) offers better handling of nonlinear relationships and noisy data, exhibiting significant generalization capabilities and largely avoiding data overfitting. Therefore, this study utilizes an SVR-driven Bayesian inference optimization algorithm for parameter identification to adjust the numerical model. The accuracy of the surrogate model is evaluated using the R-Squared (R²) error as follows:

where, $Y_i$ is the actual observed value, $\widehat{Y_i}$ is the value predicted by the model, $\overline{Y}$ is the mean of the actual observed value, and $N$ is the number of samples.

As shown in Fig. 5(a), the frequency curves constructed from the numerical natural frequencies has been revised. Moreover, the measured natural frequency data points fit better with the updated computational frequency curve after the model modifications. The parameter identification method effectively modifies the numerical model of the suspension bridge, making it more closely align with the actual physical model. As shown in Fig. 5(b), the 99.9 % validation accuracy of SVR proxy model demonstrates the numerical model of suspension bridge is well modified and can be used for cross-domain learning with the physical model.

3.4. Damage settings

Slings are critical load-bearing components of suspension bridges which suffer substantial loads. Slings are susceptible to aging and damage due to oxidation and corrosion from long-term exposure to natural elements. Moreover, significant wind loads, and bridge vibrations can cause instability to sling. Thus, slings are the most damage-prone components among the components of suspension bridges.

Fig. 5The results of model modification: a) measured frequency before and after modification, b) scatter plot of suspension bridge real response and surrogate model predicted response

a)

b)

This study focuses on the damage of the slings in suspension bridges. Half of the bridge is investigated considering the symmetrical structure of the suspension bridge. As shown in Fig. 6(a), four distinct positions are considered with two severity levels, 9 cases are formed in total with the comparison of intact case. The damage locations are strategically set to cover a broad range of potential damage locations on the slings. Position $a$ nears the bridge tower, position $b$ locates at one-quarter of the bridge span from the tower, position $c$ locates at two-thirds of the span from the same side, and position $d$ locates at the midpoint of the span. When dealing with the physical model, damage is simulated by changing the slings to modify their elastic modulus. The severity of damage is simulated by using aluminum wire and nylon wire with different materials to replace the original steel wire. The damage using the aluminum wire and nylon wire are labeled as Type 1 and Type 2, respectively. As shown in Fig. 6(b), 8 accelerometers numbered from 1 to 8 are attached along the horizontal axis for measuring the vertical accelerations. The vertical accelerations of the corresponding nodes in the modified numerical model are extracted.

Fig. 6Damage settings: a) damage locations, b) accelerometers layout

a)

b)

4.1. Data collection and processing

As shown in Fig. 7 (a), a linear frequency-swept excitation is applied on the bridge physical model using an electromagnetic shaker. The frequency of the excitation increments from 0 to 100 Hz in 10 seconds, while the sampling frequency of the signal acquisition is set to 1000 Hz. Each damage case is reloaded 20 times, totally 180 sets of vertical accelerations are collected. Fig. 7(b) shows one of the accelerations sets.

Fig. 7a) linear frequency-swept excitation, b) acceleration responses

a)

b)

For the modified numerical model, a same linear frequency-swept excitation ranging from 0 to 100 Hz is also applied at the center of the bridge deck. The transient analysis is conducted on the modified numerical model, with a sampling frequency of 1000 Hz and a sampling duration of 10 seconds. Each of the 9 damage cases is subjected to 200 computational analyses, and a total of 1800 sets of vertical accelerations are collected. Fig. 8 displays the computational results for different damage cases under the same excitation. Various cases show similar vibration characteristics under the same excitation, it is difficult to find the differences directly from the time-domain accelerations. Fig. 9 shows the computational results of different sensors for the type 1 damage-a damage case under excitations-1 and excitations-2 The amplitudes of accelerations induced by different excitations show some difference, and the frequency variations remain highly consistent. Regarding the different damage cases and measuring positions, the variations of accelerations are weak, and it is not enough to deconstruct and observe this difference in 1-D signals.

Fig. 8Accelerations forced by a same excitation for different damage cases

a)

b)

c)

d)

e)

f)

g)

h)

i)

Fig. 9Accelerations of different measuring positions for same damage case

a)

b)

c)

d)

e)

f)

g)

h)

4.2. Sample spectra of un-threshold RPs

To minimize the loss of signal characteristics caused by parameter settings, the un-threshold multivariate RPs have been employed to extract structural damage characteristics. The un-threshold multivariate RPs allow a more nuanced representation of the damage by capturing a broader range of interactions between multiple variables without the requirement of predefined thresholds. An 8-dimensional un-threshold multivariate RPs using accelerations from each set of experiments are constructed according to Eq. (10) with the following steps.

Step 1: data preparation. Standardizing the data and transforming it into a standard normal distribution with the same mean and standard deviation as:

where, $x$ is the original data point, $\mu$ is the mean of the dataset, and $\sigma$ is the standard deviation of the dataset. The resulting Z-value indicates the deviation of the original data point from the mean, expressed in units of standard deviation.

Step 2: RP construction. Calculating the number of RPs that can be generated based on the size of the input matrix, the dimensions of the RPs, and the step size.

Step 3: distance calculation. Creating a three-dimensional zero matrix to store the distances for each RP. Calculating the Euclidean distance between each block and storing these distances in the three-dimensional zero matrix.

Step 4: image conversion and saving. Converting each distance matrix within the three-dimensional zero matrix into a grayscale image, thereby constructing the un-threshold multivariate RPs.

Fig. 10 shows the un-threshold multivariate RPs based on the modified numerical model of the suspension bridge, which depict the same locations with different severities of damage. The primary differences in damage depiction are concentrated in the upper right corner of the plots. In Fig. 10(b) and Fig. 10(c), there is a consistent “cross” feature at the positions indicated by the rectangle, although this feature exists in different spatial dimensions. This specific feature indicates the potential areas of structural damage and could be used to identify and quantify damage severity.

Fig. 10Un-threshold multivariate RPs for different cases based on the modified numerical model

a) Health state

b) Type 1 damage-c

c) Type 2 damage-c

Fig. 11 shows un-threshold multivariate RPs based on the modified numerical model of the suspension bridge, which depict different locations with the same severity of damage. A “cross” feature is noticeable at the position indicated by the rectangle in the upper right corner of the RPs. However, variations in the image’s grayscale and spatial dimensions can be observed. These differences reflect the influence of the position of damage on the structural dynamics captured by the RPs. This suggests that the “cross” feature is a consistent marker of damage and it changes depending on the location of the damage within the structure.

Fig. 11Un-threshold multivariate RPs for the different locations based on the modified numerical model

a) Type 2 damage-a

b) Type 2 damage-d

Fig. 12 shows the un-threshold multivariate RPs constructed from the measured data of different cases using the physical model. The differences in grayscale can be used to distinguish between the severities and specific characteristics of the damage. The visual diversity in the RPs highlights the utility of this method in capturing subtle nuances in bridge’s behavior, which can be critical for accurate damage detection.

Fig. 12Un-threshold multivariate RPs for different cases based on the physical model

a) Health state

b) Damage state

In summary, the two-dimensional RPs release varying damage characteristics, which confirms the effectiveness for depicting structural damage in single-span suspension bridges. Fig. 11 and Fig. 12 illustrate some differences for different damage scenarios, but the distinctness of these features is not easily discernible to the naked eye. To address this issue, it is proposed to utilize the powerful classification capabilities of CNNs for image classification, aiming to enhance damage identification in structural assessments.

1800 un-threshold multivariate RPs based on the modified numerical model and 180 RPs based on the physical model are used to construct two pre-training sample sets.

The first set comprises only the RPs from the modified numerical model and is intended for a 2DS-CNN network model. The RPs are classified into 9 categories according to their scenarios, with each training set containing 180 plots and each validation set containing 20 plots, labeled from 0 to 8 as listed in Table 4. The second set, intended for a 2DS-CNNF network model, includes RPs from both the physical model and the modified numerical model. Similarly, the RPs are classified into 9 categories, each containing 190 training plots and 30 validation plots.

4.3. 2D-CNNFs structure and parameter settings

The structure of the CNN model adapted in this study, which incorporates the convolutional layer, batch normalization layer, pooling layer, and fully connected layer, is depicted in Fig. 13. The batch normalization layer helps standardize the inputs across the network, facilitating faster and more stable training. This configuration is aimed at optimizing the network's performance in recognizing and interpreting the complex patterns represented in the RPs. To prevent overfitting due to excessive complexity while ensuring robust feature extraction capabilities, three convolutional layers are chosen. The batch normalization layers, placed after the convolutional layers, address the issue of vanishing gradients during training, which can slow network convergence, thereby enhancing the robustness of the CNN. Additionally, the Leaky ReLU function is used as the activation function, enhancing the network's capability for nonlinear modeling.

Fig. 13Structure of the CNN modal

The CNNs are trained using un-threshold multivariate RPs generated from data collected with the modified numerical model of the suspension bridge as input and validation samples. The network’s parameters are detailed in Table 5. The setup allows comprehensive testing of the CNN’s effectiveness in detecting and classifying structural damage based on complex patterns identified in the RPs.

5.1. Improvement of CNNs based on transfer learning

The data of various damages can be obtained easily from the numerical model, but not easily from the physical model or real bridge. To address the difficulty of obtaining extensive physical model damage data for input training, the transfer learning is utilized. Transfer learning approach allows a cross-domain data feature fusion for achieving an effective damage identification in the physical model. The transfer learning network model, abbreviated as CNNF, is established, and its framework is illustrated in Fig. 14. The changes to the model’s end-layer parameters are detailed in Table 6.

Several key steps of the construction of the CNNF involves:

Step A: source domain training.

A total of 1800 labeled RPs from the numerical model of is used to train the 2D-CNNs, among them 1/9 of the data is split as validation sets. This step aims to extract structural damage features and establish mapping relationships. The trained model provides a pretrained network for the transfer of learning from numerical features to physical features.

Step B: fine-tuning with mixed data.

The parameters of the front layers of the 2D-CNNs are frozen to retain the damage features from the numerical model. A small number of labeled physical RPs are input into the network's end layers for rapid training. This process fuses the features from both the numerical model and physical model and realizes the preparation of the 2D-CNNFs.

Step C: unlabeled data testing.

This step involves inputting unlabeled physical RPs into the trained 2D-CNNFs to observe prediction outcomes and validate the new network model’s accuracy and robustness.

Using the framework of the 2D-CNNF network, a large number of the modified numerical data, along and a small number of physical data are used for training and validating. This approach is robust and effective for structural damage identification with limited actual damage data.

Based on the trained 2D-CNNs, 180 un-threshold multivariate RPs constructed from physical model of the suspension bridge are used. The training and validation data were split equally for training in the end layers of the constructed 2D-CNNs. 10 RPs are randomly selected from each case, and a validation set is formed by a total of 90 RPs. As shown in Fig. 15, the loss function of the improved convolutional neural network approaches zero, demonstrating that the network effectively extracted image features from the un-threshold multivariate RPs. It also successfully integrated the features of the modified finite element model, resulting in a cross-domain 2DS-CNNFs.

Fig. 14Framework of 2D-CNNFs

Fig. 15Function plot of loss: a) train loss, b) validation loss

a)

b)

5.2. Identification results

The confusion matrix derived by the 2D-CNN based on the validation data is shown in Fig. 16. The main diagonal of the matrix shows an accuracy of 97.2 %, indicating that the trained 2D-CNN can effectively extract the features from the un-threshold multivariate RPs. The 2D-CNNFs generated the numerical data and physical data are compared with the 2D-CNNs generated by the numerical data. Both networks are used to identify damage using un-threshold multivariate RPs generated from physical data of the suspension bridge, with the results displayed in Fig. 16 and Fig. 17.

Fig. 17 reveals a damage identification accuracy of 83.3 %, indicating that the RPs generated from the modified numerical data indeed share similar features with those generated from physical data of the suspension bridge. However, the accuracy of the damage identification is defective during the cross-domain structural damage identification. Specifically, the identification accuracy of the intact case and type 1 case achieves only 77.8 %. The confusion between these two cases suggests that the 2D-CNNs without cross-domain feature learning struggles to distinguish between the two types of damage.

Fig. 16Confusion matrix for damage identification in modified finite element model

Fig. 17Confusion matrix for non-transfer learning group

In contrast, Fig. 18 shows a damage identification accuracy of 91.1 % for the 2D-CNNFs with cross-domain feature learning. This network significantly outperformed 2D-CNNs without cross-domain feature learning, with all the damage identification accuracy higher than 88.9 %. Observations from the 3rd and 7th columns of the main diagonal confirm a noticeable improvement of the 2D-CNNFs in distinguishing between the two cases. The effectiveness of the transfer learning in enhancing the ability of feature extraction of the 2D-CNNs is proved. The proposed damage identification method based on RPs and CNNs can effectively identify the structural damages in the suspension bridge.

Fig. 18Confusion matrix for transfer learning group