Research on intelligent detection of pavement damage based on CNN

2.1. Image acquisition of pavement damage mechanism

Currently, there are various types of road damage, such as cracks, subsidences, channels, pads, pits, disks, etc., sometimes each damage occurs alone, sometimes in several complex forms [17-20]. However, it can be found that there is a certain law: all kinds of damage phenomena are caused by the interaction between driving and natural factors as well as the road surface, and it varies with external influencing factors and road surface working characteristics. The ability of the subgrade and pavement as a whole to deform or fail under the influence of an external force is referred to as pavement strength, and it reflects the mechanical properties of the pavement. The stability of pavement denotes the variability of the internal structure of the pavement and the strength of the pavement under the influence of external factors and the range of the change. The common road damage phenomenon refers to the specific manifestation of insufficient strength and road strength as well as stability.

As a result, in the face of complex types of pavement damage, we broadly classify pavement damage into five types: potholes, depressions, one-way cracks, looseness, and cracks, which greatly enhance the identification efficiency during the identification process. Moreover, these damage types can provide maintenance personnel with a clearer maintenance direction. Pavement damage image classification is illustrated in Fig. 1.

Fig. 1Image of pavement damage types

a) Loose

b) Pit

c) Depression

d) Rut

e) Crack

2.2. Convolutional neural networks

A convolutional neural network is a very special kind of deep feedforward neural network. It is a multi-level non-fully connected neural network, which simulates the structure of the human brain. It can directly recognise visual patterns from original images via supervised deep learning [20-24]. The CNN model generally has four layers of stacking, including a convolutional layer, pooling layer, fully connected layer, and SoftMax classification layer. Its structure is shown in Fig. 2.

Each layer of the neural network is constructed on multiple planes composed of independently distributed neurons. The connection between layers is a non-fully connected convolution calculation. Each neuron represents a partial dimension value of the input unit weighting. The current relatively successful CNN models consist of the Alex network, Google network, and VGG-19 network, which are capable of accurately identifying generalised objects.

Fig. 2Structure of convolutional neural network for pavement damage image

2.3. The transfer learning structure of the VGG-16 model

In this section, a transfer learning method is put forward to apply the VGG-19 network to pavement damage detection. Transfer learning is a powerful deep learning technique. We can frequently employ a well-trained convolutional neural network such as the VGG-19 network when the dataset is not large enough to train a complete CNN network, which is shown in Fig. 3.

There are two types of VGG-19 model transfer learning: one is to freeze the weights trained prior to a fully connected layer of the model, only by changing the last fully connected layer, train its own data in the fully connected layer, and finally get the recognition. The other is to modify the weights in the intermediate layers and make the training results more accurate by continuously adjusting the weights. The first form of transfer learning is used in this experiment. First, the VGG-19 network model is introduced by transfer learning. Secondly, freeze the other layers with the exception of the fully connected layer. After importing the data, adjust the weight parameters of the data in the fully connected layer, and finally, train the data to achieve the result.

Fig. 3VGG-19 model structure diagram

3.1. Technical route

A pavement damage identification model based on a convolutional neural network is established, and the classification results of pavement damage identification are achieved. Fig. 4 depicts the steps of the research process.

The research is roughly divided into two categories: research on the collection and selection of pavement damage images by means of photography and the selection of network databases. The data set on the experimental subjects were processed by the study following the acquisition. For acquiring a more reasonable classification of pavement damage, the study makes use of the basic theoretical knowledge of pavement damage mechanism so as to reclassify the research object, that is, the pavement damage image. The results obtained from the classification types are displayed in Fig. 1. There are five types: pits, depressions, one-way cracks, looseness, and cracks. If these five forms of damage can be found and resolved immediately, more serious structural damage can be avoided on the road surface and the basic hazards of road surface damage can be mitigated. To ensure better outcomes when the images are fed into the model, the classified dataset is augmented by image processing. For the expansion and processing of the data set, a variety of image processing methods have been adopted in the research, so that the images can be extracted better.

Fig. 4Schematic diagram of identification process

For the well-processed images, the method of transfer learning is utilised to substitute the pavement damage image data set into the research selection model. After adjusting the details of the selected model, we were able to obtain an experimental model that could be adapted to this dataset, so that this model can be fully applied to the study of pavement damage images.

Since a number of existing network models of convolutional neural networks can be used in the experiments on pavement damage, for the purpose of verifying the rationality of the research model in the selection of pavement damage models, this study carried out models of the same data set for various models, training, and validation testing. The results produced can demonstrate that for the data set, the model selected in this study is more reasonable and can be applied in future research on artificial intelligence detection.

4.1. Data set processing

Prior to a model being trained, preparatory work or the establishment of the data set is performed. Due to the complexity of pavement conditions and multiple safety factors, pavement damage images cannot get a large number of samples, hence the transfer learning method resolves the difficulty of having a smaller sample library.

For the types of pavement damage we have categorised in this sample database, 20 samples of each type of sample data are gathered, and a total of 100 samples of five types of pavement damage images are collected such as potholes, cracks, and depressions. For these 100 images, a data set was established. We expanded the smaller data set using techniques such as rotation extension and edge extension in order to get better data results. Its rotation formula is as follows:

Once the database is expanded, a total of 3000 datasets are obtained, with 600 samples of each type after classification. Secondly, the data set is split into a test set and validation set with a ratio of 4:1, and finally, 2400 test set data as well as 600 validation set data are acquired. Next, we perform region-of-interest interception (ROI) on the classified dataset.

4.2. Image processing

The image is intercepted by the region of interest (ROI), and the image is initially cropped into a 256×256 two-dimensional image. The road damage image is then further preprocessed, the road damage image is flipped horizontally and diagonally, and the random parts are enlarged. Enhance image contrast is based on the histogram equalization method. And fill up the blank parts of the preprocessed image. As a result, the image can be extracted with more accuracy in the subsequent feature extraction process. A specific example picture is indicated in Fig. 5.

Fig. 5 shows a featured image of road damage and surface damage that will be put to the test. It is obtained by performing feature extraction on the region of interest, adjusting the contrast, normalising the image, and performing edge-filling processing on the periphery of the image. This series of operations is carried out on the data set, and a more reasonable sample library is ultimately obtained.

Fig. 5Image processing

4.3. Establishment of VGG-19 model structure

4.3.1. Model establishment

The processed data set is classified by label, and converted into an electronic data set, followed by inputting into the VGG-19 network that has been migrated. Freeze the convolutional layers and pooling layers with the exception of the fully connected layer, and re-establish the fully connected layer. VGG-19 consists of two parts, VGG19.features, and VGG19.classifier. VGG19.features has convolutional layers and pooling layers as well as three linear layers and finally a Softmax classifier. After loading VGG19 with Tensorflow 1.4 and setting the pretraining weights to True, mark the convolutional and pooling layers as frozen layers, which makes these layers untrainable. Subsequent pairs use CrossEntropyLoss for a multi-class loss function. The basic idea can be found in Fig. 6.

Fig. 6Basic idea of VGG-19 model based on pavement damage images

4.3.2. Image classification based on softmax classifier

Since the original VGG-19 network model trains 1000 types of images, but then there are only 5 types of training in this paper. Following the modification of the two linear layers of the fully connected layer, the Softmax classifier is likewise utilised for this experiment. The schematic of the Softmax classifier is depicted in Fig. 7.

The output and input image classification types are divided into 5 types. Due to the fact that the Softmax classifier can realise multi-classification of data, it can ensure that the accuracy of the classified data is more accurate. The formula is as follows:

3

$y_1=w_{11}{x}_1+w_{12}{x}_2+w_{13}{x}_3+w_{14}{x}_4+w_{15}{x}_5+b_1.$

Fig. 7Softmax classification structure diagram

The SoftMax classifier is an extension of the logistic regression model for multi-classification problems. In the study, the class label y can choose 5 different values. For the training set $\left\{\left(x\left(1\right),y\left(1\right)\right),\dots ,\left(x\left(n\right),y\left(n\right)\right)\right\}$, the class label is $y\left(n\right)\in \left\{\mathrm{1,2},\dots ,r\right\}$. In this paper, the pavement damage images are categorised into five types: potholes, depressions, ruts, cracks, and looseness.

4.4. Analysis of results

4.4.1. Model training results

In order to verify the accuracy and reliability of the VGG-19 model, the study employed the same transfer learning method for the purpose of training the VGG-16 model, the ResNet121 model, and the ResNet50 model on the same dataset. The training results are illustrated in Fig. 8. The accuracy of each model can be found in Table 1.

Fig. 8The line chart of the accuracy of road damage images trained by each model

a) VGG-19

b) VGG-16

c) Resnet 121

d) Resnet 50

Table 1Comparison of the accuracy rates of various models

Network model	Validation accuracy	Validate the loss rate
Custom model	80.0 %	NUA
VGG-19	86.2 %	0.4043
VGG-16	82.6 %	0.4562
Resnet121	65.7 %	1.3346
Resnet50	60.4 %	1.5826

The diamond dotted line in the figure represents the accuracy of the test set, and the square dotted line reflects the accuracy of the validation set.

The $X$-axis is the number of detections, while the $Y$-axis is the accuracy of each detection.