Application of AI intelligent vision detection technology using deep learning algorithm

3.1. Defect image pre-processing

Before visual inspection of SPSD can be performed, the visual image needs to be pre-processed first. The main technique used in the study is ROI detection. This technique aims to achieve ROI detection of steel plate surface images in order to filter out defective images from defect-free images and avoid unnecessary image processing. An example of the defect image is shown in Fig. 1.

Fig. 1Example of defect image

a) Example of defect free images

b) Example of defect images

The algorithm needs to have a high detection rate as well as a fast detection speed considering the fast production of steel plates. An algorithm is proposed by the projection gray mean, which performs ROI detection by calculating the maximum and minimum values (MMV) of the horizontal and vertical projection gray mean and obtaining the difference. In addition, the algorithm sets a preset threshold to compare with the difference value to determine whether there is a defect in this image, and if there is, the next image preprocessing step is performed. In the absence of defects, the algorithm clears the image to free up the memory and re-inputs the image to improve the detection efficiency. The specific flow is shown in Fig. 2.

As can be seen, first, the MMV of the horizontal and vertical projection gray mean values of the image are obtained, and then the difference between them is calculated and compared with the preset threshold value. If the difference value is greater than the threshold, the image is defective and the next image pre-processing step is required. Conversely, if the difference is less than or equal to the threshold, the image is not defective. In this case, the image needs to be cleared to free the memory, and the image is re-imported for detection. The horizontal projection gray mean value is shown in Eq. (1):

where, $x$ is row number, $y$ is column number, and $N$ represents the total number of pixels in each row. The mean vertical projection grayscale value is shown in Eq. (2):

where, $M$ is the total number of pixels in each column. The difference between MMV of the horizontal projection grayscale mean is shown in Eq. (3):

where, $f_{x\mathrm{m}\mathrm{a}\mathrm{x}}$ and $f_{x\mathrm{m}\mathrm{i}\mathrm{n}}$ means the MMV of the horizontal projection of the average gray value, respectively, and the difference between the MMV of the vertical projection of the average gray value is shown in Eq. (4):

where, $f_{y\mathrm{m}\mathrm{a}\mathrm{x}}$ and $f_{y\mathrm{m}\mathrm{i}\mathrm{n}}$ represent the MMV of the mean vertical projection gray value, respectively. The conditions for determining the presence of defective images are shown in Eq. (5):

where, $T_1$ and $T_2$ represent the detection thresholds for horizontal and vertical directions, respectively. In defect image processing, denoising is required to improve defect localization detection. For this problem, the study uses the Improved Adaptive Median Filtering Algorithm (IAMF) algorithm as the main aluminum filtering algorithm, which is able to remove noise well while retaining the edge and detail information of the defects, and its flow is shown below.

Fig. 2ROI detection algorithm

The processing steps of the algorithm include boundary filling, grayscale judgment, filter window size adjustment, and value extraction output. When using the IAMF algorithm to filter the defective image, the original image needs to be boundary-filled first, and then the window size is gradually adjusted according to the principle of the algorithm and the grayscale values of the corresponding pixel points are output. If the final output grayscale value still cannot remove the noise, calculating the average grayscale value of all pixel points is needed within the maximum filtering window except the noise point and output it as the grayscale value of the current pixel. If all pixels within the maximum filter window are noisy points, the weighted average of the minimum and maximum grayscale values within the current filter window needs to be taken as the grayscale value output for the current pixel. Finally, the filtered image can be obtained by removing the filled boundary. the IAMF algorithm can better remove all kinds of noise, including pretzel noise and Gaussian noise, etc., thus making the defect image processing more reliable.

Fig. 3IAMF algorithm

3.2. AlexNet network defect classification model design

The classification of SPSD is important in the steel production process. The performance of the traditional AlexNet structure in identifying steel surface defects is not satisfactory, so improvements are made to the traditional AlexNet network to improve steel surface defect classification. The main improvements include the attention mechanism module, depth-separable convolution, batch normalization layer, residual network structure, and the replacement of the activation function. First is the attention mechanism module. It is introduced to filter the important features in the feature map. The attention mechanism (Sequeeze-and-Excitation (SE) module is derived from SENet. The design idea of the SE module is to learn the feature weights of each channel through the loss function, so that the feature map weights of task-related channels are significant, and thus improve the model classification performance. In the traditional AlexNet structure, the SE module is added before pooling layer 1, so that the input feature map can retain the important features better. The improved structure is shown below.

Second, the depth-separable convolution is introduced. The introduction of it can make the model have a better classification effect. Depthwise Convolution is divided into Depthwise Convolution (DW) and Pointwise Convolution (PW), and convolution kernel size of DW is Dk×Dk and number of k. The convolution kernel size of PW convolution is 1×1×k and the number of N. Compared with the traditional convolution, the computation and the number of parameters of Depthwise Convolution are smaller. The depth-separable convolution is computationally smaller and the number of parameters is smaller, so it has higher computational efficiency. In the modified network, the 3×3 convolutional layers in the original AlexNet structure are replaced with deeply separable convolutions. Assume that the input feature map size is shown in Eq. (6):

Fig. 4Structure after introducing SE

a) Original structure

b) Structure after introducing SE

The output feature map dimensions are shown in Eq. (7):

Convolution kernel size is shown in Eq. (8):

Then the total calculation is shown in Eq. (9):

Then the total number of parameters is shown in Eq. (10):

The total computation of the depth-separable convolution is shown in Eq. (11):

Eq. (12) demonstrates the ratio of the total computation required for depth-separable convolution to standard convolution:

Next, the batch normalization layer is introduced, which is typically positioned after the activation function to ensure that the activation function input is at the requisite position where the nonlinear function is more sensitive to the input and the network gets a larger gradient. The introduction of the batch normalization layer can reduce model overfitting. Furthermore, the residual network structure is presented, a deep learning architecture designed to effectively address the issues of gradient disappearance and explosion. This model directly transfers the previous layer's output to the subsequent layers and adds it with the data obtained through convolution of the residual structure to serve as the input of the lower layer network. Lastly, the activation function is replaced. ReLU activation in the original AlexNet structure causes the neuron to fail to update the parameters when the input is negative. Therefore, the ReLU function is replaced by the Leaky ReLU function with leakage correction as the activation function, and the function output has a very small slope when the input is negative, which can ensure the response of the function input to negative information. In summary, by introducing improvements such as attention mechanism module, deep separable convolution, batch normalization, residual network structure, and activation function replacement, the improved AlexNet network can be better applied to SPSD classification. In particular, the introduction of an attention mechanism and deep separable convolution can help extract more key features and thus improve the classification performance. The batch normalization layer makes the data distribution more uniform, which helps to reduce the overfitting phenomenon. The residual network structure is achieved by introducing short-join connections to increase the depth and width of the model network while effectively avoiding gradient disappearance and gradient explosion problems. The replacement of the activation function avoids the degradation of learning ability due to the failure to update some neuron parameters. The improved AlexNet network structure package is shown in Fig. 5.

Fig. 5Improving the AlexNet network structure

The improved AlexNet network structure includes the convolutional, pooling, and fully connected layers from the original AlexNet network, as well as the new SE module, deep separable convolution, batch normalization layer, residual network structure, and Leaky ReLU activation function. These improvements enable the network to achieve better performance in steel plate surface defect classification tasks with enhanced feature extraction and generalization capabilities while keeping the number of parameters and computational effort low.

When modifying the standard AlexNet for surface defect classification of steel plates, the study first considers the network's parameter selection. Network parameters can be adjusted and optimized through experiments to achieve optimal performance. Regarding the learning rate, it is an essential parameter in optimization algorithms. A smaller learning rate may slow down convergence speed, while a larger one may lead to unstable training processes. The adaptive learning rate optimization algorithm, such as Adam, can be chosen to be used, which enables the automatic adjustment of the learning rate. It is suggested that the initial learning rate be set to 0.001. In terms of batch size, the number of samples employed in each training iteration is referred to. Better utilization of hardware acceleration is typically achieved with larger batch sizes, but performance limitations may arise. Different batch sizes, such as 32, 64, 128, etc., can be experimented with. Concerning activation function selection, the study adopts the method provided by Leaky ReLU to improve ReLU and selects the activation function with the best performance through cross-validation.

In terms of network architecture, research has introduced an attention mechanism module - by introducing the Sequence-and-Excitation (SE) module, important features in the feature map can be filtered and the weight of task related channels can be increased, thereby improving the classification performance of the model. In the improved structure, the SE module is added to the position before pooling layer 1 to better preserve important features in the input feature map. Simultaneously introducing deep separable convolution - deep separable convolution can improve the model's feature extraction ability, thereby improving classification performance. In the improved network, replace the original AlexNet structure with 3 × Replace 3 convolutional layers with deep separable convolutions. And introducing batch normalization layer (BN layer) - batch normalization can reduce model overfitting and make the distribution of input data in each layer more uniform. The BN layer is usually used before activating the function. Studying the use of residual network structure - residual network structure can effectively solve the problems of gradient vanishing and gradient explosion. By introducing a short circuit connection, the output of the previous layer is directly transmitted to the subsequent layers, and added to the convolutional data obtained by passing through the residual structure as the input of the lower layer network. And replace the activation function - the ReLU activation function used in the original AlexNet structure may cause neurons to be unable to update parameters when inputting negative numbers. In the improvement method, Leaky ReLU is used to replace ReLU, ensuring that the function can respond to negative information when inputting negative values. In summary, this study has improved the AlexNet network by introducing attention mechanism modules, deep separable convolutions, batch normalization layers, residual network structure, and replacement of activation functions. This has enabled the improved AlexNet network to have stronger feature extraction and generalization capabilities while maintaining a low parameter and computational complexity, thus achieving better performance in steel plate surface defect classification tasks. In terms of network architecture, the research has introduced an attention mechanism module known as the Sequence-and-Excitation (SE) module. This module filters important features in the feature map and increases the weight of task-related channels, thereby enhancing the classification performance of the model. In the enhanced structure, the SE module is incorporated before pooling layer 1 to better preserve crucial features in the input feature map.

Simultaneously, deep separable convolution has been introduced to improve the model's feature extraction ability, thereby enhancing classification performance. In the improved network, the original AlexNet structure is replaced with a 3×3 deep separable convolution for each of the three convolutional layers. Furthermore, a batch normalization layer (BN layer) has been introduced. Batch normalization reduces model overfitting and promotes a more uniform distribution of input data at each layer. The BN layer is typically utilized before activating the function. The study also explores the use of a residual network structure, which effectively addresses the issues of gradient vanishing and gradient explosion. By introducing short-circuit connections, the output of the previous layer is directly transmitted to subsequent layers and added to the convolutional data obtained through the residual structure as the input of the lower layer network. Additionally, the activation function has been replaced. The ReLU activation function used in the original AlexNet structure may hinder parameter updates when negative numbers are input. In the improvement method, Leaky ReLU is employed as a replacement to ensure that the function can appropriately respond to negative information. In summary, this study has enhanced the AlexNet network by introducing attention mechanism modules, deep separable convolutions, batch normalization layers, residual network structure, and replacement of activation functions. These enhancements equip the improved AlexNet network with stronger feature extraction and generalization capabilities while maintaining low parameter and computational complexity. Consequently, the network achieves superior performance in steel plate surface defect classification tasks.

3.3. Improved RCNN defect location detection model design

The aim of this study is to solve SPSD using a Faster Region-based Convolutional Neural Network (Faster RCNN) method for locating and detecting defects on steel plate surfaces. In achieving this goal, the differences between different target detection algorithms are discussed and optimization work for Faster RCNN is carried out to improve the accuracy in locating and detecting SPSD. SPSD detection is a key process to ensure steel quality, improve product value, and reduce production costs are crucial objectives. Target detection algorithms are categorized into two main groups: faster YOLO and SSD algorithms and the more accurate Faster RCNN algorithm. Through the statistics of the steel plate surface defect data set, it is found that a large proportion of small size and slender type defects characterize the steel plate. Therefore, to improve defect detection capability, a Faster RCNN algorithm with higher accuracy is selected for optimization in this study, and the overall structure is shown in Fig. 6.

Fig. 6Faster RCNN algorithm structure

First, the feature extraction part of Faster RCNN is improved for Faster RCNN. The traditional Faster RCNN extracts feature by VGG16 whose output feature map has a low resolution, which may decrease localization accuracy of small-sized defects. Therefore, this study adopts the improved AlexNet network to extract features, aiming to ensure the high resolution of the feature maps and improve the accuracy of global semantic information. Secondly, in order to overcome the effect of subjectivity generated by setting the Anchor manually, the IOU K-means clustering-based algorithm can optimize the Anchor generation method in the RPN network. By calculating the average intersection ratio between each real defective border and the k Anchors obtained by clustering, the appropriate k value is determined, and the number and size of Anchors are adjusted accordingly. The average cross-merge ratio can be expressed in the form of Eq. (13):

The average cross-merge ratio can be calculated using Eq. (14):

where, $\theta$ denotes the real edges of defects under labeling, while $\psi$ denotes the clustered edges, $m$ denotes total number of defect training sets, and $k$ denotes the amount of clustering. In addition, $IOU$ can be calculated using Eq. (15):

Finally, in order to avoid the errors brought by the ROI Pooling algorithm to the detection results and to improve small defects detection, ROI Align can replace ROI Pooling. It retains the floating point results and calculates the eigenvalues of the sampling points by bilinear interpolation, which can better maintain the accuracy of the defect feature size. The ROI Align algorithm implementation process is shown in Fig. 7.

Fig. 7ROI Align algorithm

After optimizing the Faster RCNN network, a more suitable feature extraction network and a more reasonable Anchor generation method for steel plate surface defect features, as well as a more accurate pooling algorithm, can be obtained. All these optimizations provide strong support for the detection task. During the training of the model, multiple iterations of training can be performed using methods such as cross-validation to evaluate the optimization effect of the Faster RCNN network and improve defect detection performance. This ensures that a more robust and robust network model is obtained.

4.1. Analysis of the detection effect

The study first analyzed the detection effect of ROI detection for the model, and the detection rate and detection time were mainly used as the main test indexes in the analysis process, and the specific results are shown below.

First, from the perspective of detection rate, the data showed the difference between different defect types. From a low of 88 % to a high of 100 %, there was a more significant change in the detection rate of the algorithm in detecting different defects. Among them, the detection rate for defect type Fr was the lowest, with only 88 % of the defects detected correctly, but the detection rates for the rest of the defect types reached 100 %. This indicated that the algorithm's detection effect could vary with different defect types. However, overall, the average detection rate of the algorithm was 98 %, and the overall test result was more satisfactory. Second, from the perspective of detection time, it can be found that the detection time was relatively short, with the shortest detection time being only 0.0011 seconds. Although the detection time under the Sc defect type reached 0.0083 seconds, this was not too obvious from the detection time of other defect types. This indicated that the algorithm had a fast detection speed and efficient processing capability, which can be effectively applied to practical application scenarios.

4.2. Defect classification effect analysis

For the performance analysis of steel plate defect classification, different algorithms for the image filtering task were used to compare two different algorithms' performance. For classification, traditional AlexNet network performance was evaluated by confusion matrix of the test set. The performance of the traditional AlexNet network was measured again after improving the network. The results of the peak signal-to-noise ratio analysis are shown in Fig. 8.

Fig. 8Peak signal-to-noise ratio analysis

a) PSNR

b) MSE

c) Filtering time

As can be seen from Fig. 8, with a noise density of 0.1, the peak signal-to-noise ratio (SNR) of the AMF algorithm was 14.448, while the peak SNR of the IAMF algorithm was 14.521. The difference between the two was not significant, but the IAMF algorithm was closer to the SNR level of the original image. In mean square error (MSE), the AMF algorithm’s error value was 2335.581, while the error value of the IAMF algorithm was 1197.881. The error value of the IAMF algorithm was not only lower, but also about 1/2 of the error value of the AMF algorithm, which means that the IAMF algorithm has stronger noise suppression ability than the AMF algorithm. As the noise density increased, performance gap between two algorithms became more obvious. At a noise density of 0.7, the PSNR of the IAMF algorithm was 6.096, while the PSNR of the AMF algorithm was only 5.781. In MSE, error value of the IAMF algorithm was 15971.058, compared to the error value of the AMF algorithm of 17197.951. The lower error value indicates that the IAMF algorithm was more suitable for processing images with high noise density defects. The confusion matrix is shown in Fig. 9.

Fig. 9Peak signal-to-noise ratio analysis

a) Traditional AlexNet network

b) Improving the AlexNet network

In Fig. 9, F1 values of the traditional AlexNet network for the classification of steel plate defects were relatively low, especially since the misclassification rate of pockmarks and inclusions was high. The specific data results obtained are shown in Fig. 9.

As can be seen in Fig. 10, the improved AlexNet network achieves a great improvement in F1 values for each defect type, especially for the recognition of scratches, patches, and cracks with 100 % accuracy. In addition, the improved network also succeeded in the classification of pockmarks, reaching an F1 value of 98.6 %. In addition, after comparison, it was found that the improved AlexNet network had great advantages in both F1 value and FPS. Compared with Goog Le Net with the highest F1 value among the four networks, the F1 value of the improved AlexNet network was improved by 0.29 %, and the FPS of the improved AlexNet network reached 237.5 f/s, which was the highest among the classical CNN classification networks. This indicated that the improved AlexNet network was better in classification performance compared to the conventional CNN network.

4.3. Defect location detection effect analysis

For the field of image defect detection, the Average Precision (AP) metric is one of the important performance evaluation parameters. The defect detection effectiveness of different algorithms can be evaluated by comparing their AP scores to classify and select the appropriate algorithm. This experiment compared the performance of the conventional Faster RCNN network with the optimized Faster RCNN network and the SSD network and YOLOv2 network for the detection of SPSD, as shown below.

Fig. 10Improving AlexNet performance analysis

a) F1 (before and after improvement)

b) F1 (different models)

c) FPS (f/s)

Fig. 11Defect detection performance

a) AP (before and after improvement)

b) AP (different models)

From Fig. 11, the traditional Faster RCNN network had an AP value below 0.9 for all kinds of defects detection, while the optimized Faster RCNN network had an AP value above 0.9 for all six defects detection, especially for small size defects detection. The APs of the SSD network were slightly higher than those of the YOLOv2 network, but still lower than those of the optimized Faster RCNN network. Therefore, the optimized Faster RCNN network was one of the best-performing algorithms for surface defect detection with high accuracy and robustness, which was ideal for practical applications. A comparison of network convergence status and metrics is shown in Fig. 12.

In Fig. 12, the optimized Faster RCNN network performed excellently. Compared with the conventional network, the mAP value of the optimized Faster RCNN network was improved by 0.110 and the FPS was not reduced. Compared with the SSD network and YOLOv2 network, the detection accuracy of the optimized Faster RCNN network was higher. In addition, the optimized Faster RCNN network performed better in terms of loss values and converged better. Therefore, the model designed in the study had more performance advantages, had higher detection accuracy and performance stability, and had wide application prospects.

Fig. 12Comparison of network convergence status and indicators

a) Training set loss value

b) mAP

c) Fps