Fault diagnosis method for hydro-power plants with Bi-LSTM knowledge graph aided by attention scheme

1. Introduction

Nowadays, hydro-power plants are highly automated with complex protection, automation, and monitoring systems that support safe and stable operation [1]. Furthermore, the increasing complexity of the hydro-power plants leads to a greater number of failures of the automation systems, which causes the reliability of the plants to fluctuate significantly. This can severely affect power generation, ranging from economic losses to serious and dangerous events. At the same time, automation equipment is often inadequately maintained. In addition, the equipment is exposed to electromagnetic interference, which blocks the reception of the information from the equipment and affects the related mechanical construction process. In this case, the useful life of the equipment is shortened, and the operation of the hydro-power plants is affected. Add to that, due to the influence of natural and other factors on the equipment, the working environment of the equipment has become more complex and severe [2]. To avoid such events, effective and timely fault diagnosis for hydro-power plants is an emergency research issue [3].

To solve the problem, the data-based approaches are initially investigated on faults diagnosis for hydro-power plants, i.e., mining information in data and training models [4]. The database-aided approaches can achieve high diagnosis accuracy without requiring efficient expert knowledge [5]. However, data-based approaches also have two problems. The first problem is that the machine learning model is a black box for users, which cannot provide explanations for the output diagnostic results [6]. This reduces the credibility of practical engineering applications [7]. Second, the data-based fault diagnosis methods cannot effectively utilize the existing prior knowledge, e.g., fault manuals and other unstructured knowledge, resulting in knowledge waste [8].

In this case, expert knowledge-based knowledge is used for fault diagnosis by extracting entities and relationships from unstructured knowledge [9]. Thus, the knowledge graph-based method can provide a unified and standardized representation of unstructured knowledge. Furthermore, the concepts and their relationships can be described in symbolic form. In this way, entities and their associated attribute-value pairs can be linked [10]. Depending on the scenario, knowledge graphs can be categorized into two aspects, i.e., general knowledge graphs and domain knowledge graphs [11]. First, the general knowledge graph is used in the general domain, assuming the breadth of knowledge. Second, the domain knowledge graph focuses on a specific domain, assuming the depth of knowledge [12]. Therefore, the domain knowledge graph is usually constructed based on industry databases and targets industry practitioners and potential industry insiders. The application of knowledge graphs to fault diagnosis focuses on fault detection, question and answer, inference, and recommendation [13, 14].

Recently, knowledge graphs have been intensively utilized in many fields, such as medical, social, and movie reviews. For example, Liu et. al. [15] showed that multiple railroad operation faults and hazards form a causal network due to their connections, where the knowledge graphs are used to explore railroad operation faults in a causal network of railroad operation faults. To intelligently diagnose faults, the authors proposed a knowledge graph-based method to determine the correlation between expert knowledge and products. Inspired by this idea, Ou et al. used the information and knowledge collected from the smart grid terminals to monitor and maintain the unattended network [16]. To improve the efficiency of diagnosis, a question-answer scheme with a knowledge graph is proposed to diagnose the faults caused by the massive access of a large number of heterogeneous terminals to the smart grid [17]. Accordingly, the application of the knowledge graph in the field of fault diagnosis can accurately and comprehensively describe the fault reason and provide a new solution to the fault diagnosis [18]. However, to the best of the author's knowledge, the automatic fault diagnosis on hydro-power plants is still a challenge. Moreover, the traditional fault diagnosis algorithm with a knowledge graph cannot output the relationship path, used to reveal the indirect reasons for the occurring faults [19].

In this paper, we propose a knowledge graph-based diagnosis method to diagnose the faults in hydro-power systems. The contributions are listed as follows.

1) We propose a knowledge graph model to express the relationships between different entities in hydro-power plants. Moreover, according to the designed knowledge graph model, the relationship path of the occurred faults can be constructed.

2) We develop the Bi-LSTM with conditional random field (CRF) to label the entities in the given documents that record the phenomenon and the reasons for the occurred faults. Moreover, the attention scheme in the Bi-LSTM with CRF is used to select the closer relations from the testing data set.

3) To improve the efficiency of diagnosis, an automatic diagnosis algorithm is developed to construct the relationship path. According to the relationship path, the algorithm can find the headstream for the occurred faults.

4) Simulation results reveal that the proposed knowledge graph can find the faults in the given document about faults faster and with higher accuracy in comparison with traditional methods.

The rest of this paper is organized as follows. In Section 2, we describe the knowledge graph model of the relation path. In Section 3, we design the proposed relation extraction method and the automatic diagnosis algorithm. and then the experimental results with the proposed fault diagnosis method are presented in Section 4. Finally, we conclude this paper in Section 5.

2. Knowledge graph model

The knowledge graphs can be constructed in two ways: top-down and bottom-up, the top-down approach refers to constructing ontology. Then extracting entities and relationships based on the ontology. The bottom-up approach refers to extracting entities and relationships. The abstraction of the ontology by inducing the cluster with the extraction results. The top-down construction method is suitable for professional domain knowledge graphs with clear knowledge scope. In addition, the bottom-up construction method is mainly used for general knowledge graphs with wide knowledge coverage. Since the fault diagnosis on hydro-power system is a professional, we employ top-down knowledge graph construction method, as shown in Fig. 1. A typical knowledge graph construction consists of four steps: ontology construction, entity extraction, relationship extraction, and graph construction.

Knowledge graphs can be created in two ways: Top-Down and Bottom-Up, where the Top-Down approach refers to the creation of the ontology [20]. Then, entities and relationships are extracted based on the ontology. The Bottom-Up approach refers to the extraction of entities and relationships. The abstraction of the ontology by inducing the cluster with the extraction results. The Top-Down construction method is suitable for professional knowledge graphs with clear knowledge scope. On the other hand, the Bottom-Up construction method is mainly used for general knowledge graphs with a large knowledge coverage [21]. Since the fault diagnosis in the hydro-power plant is a professional task, we employ the Top-Down construction method for knowledge graphs, as shown in Figure 1. A typical knowledge graph construction consists of four steps: ontology construction, entity extraction, relations extraction, and graph construction.

Fig. 1An example model of the knowledge graph

Ontology construction also means that entities, attributes, and relationships in the knowledge graph can be clearly defined. In the Top-Down knowledge graph construction process, ontology construction starts from the top-level concepts to build ontologies, which are refined into entities and relationships. Then, we use a six-dimension vector to express knowledge ontology (KO) expressed as:

where $C$ is the set of the conceptions, $R$ is the relationship set, $A^C$ is the set of the attribution of conceptions, $A^R$ is the attribution of Relationships, and $I$ is the entity of the relationships.

The conception set is defined as:

where $n$ is the number of the conceptions.

Similarly, the entity set can be defined as:

where $m$ is number of the entities.

Let $C_s$, $C_t$ be the $s$-th, $t$-th conceptions in the concept sets. For any $C_t$, there exists $C_s$ ($s\ne t$, $1\le s$, and $t\le n$), and it holds that:

where $R$ is the relation between $C_s$ and $C_t$. $R(C_s,C_t)$ is used to express the relationship of conception and $R(I_s,I_t)$ is the relationship between of $I_s$ and $I_t$. The definition of the attribution set is defined as:

where $id$ is the attribution of the KO and $T_l$ is the attribution of conceptions. The relationship path $P$ consists $R_1,R_2,...,R_n$ for any $i(1<i<n-1)$, and we have:

where $domain\left(R\right)$ is the definition domain of the $R_i$, $rang\left(R_i\right)$ is the value domain of relations. It is hold that:

The relation path $P=R_1,R_2,...,R_n$ is expressed as:

where $T_0$ is expressed as $T_0=dom\left(R_1\right)=dom\left(P\right)$, $T_1$ is expressed as $T_0=range\left(R_1\right)=dom\left(R_2\right)$, and $T_i$ can also be derived with the same method.

Fig. 2An example of the knowledge relation path

In Fig. 2, the entities from $T_0$ to $T_6$ can be expressed as:

where $T_2$ is the $R_1$ with value domain $range\left(R_1\right)$, $T_3$ is the $R_2$ with value domain $range\left(R_2\right)$, and $T_6$ is the $R_3$ with value domain $range\left(R_3\right)$.

3. Entity and relation extraction

In this model, each word of the original document is treated as a node, and each relation is treated as an edge connecting its head entity and tail entity. The relation refers to the connection between entities, and after the entity extraction is completed, the relation extraction is performed according to the entities. Relation extraction can be used to automatically detect the relationship between entities from the original documents, and the accuracy of relation extraction directly affects the quality of the knowledge graph.

3.1. Entity extraction with Bi-LSTM aided by CRF

Bidirectional Long Short-Term Memory Network (Bi-LSTM) is a commonly used entity extraction algorithm. The principle is to connect two LSTMs with opposite timing directions to the same output. When performing entity extraction, it need to use both the information above and below. Conventional LSTMs can only use the information above but not below, which makes it difficult to be used to achieve accurate entity extraction. Bidirectional LSTM achieves the utilization of contextual information by connecting two LSTMs with opposite timing directions to the same output, solving the problem that LSTMs are unable to use the following information and improving the effectiveness of entity extraction.

The core of the bi-directional LSTM network structure is the merging of two LSTMs, where one LSTM is forward and the other is reverse. The forward LSTM can utilize its historical data, i.e., the antecedent information; while the historical data that the reverse LSTM can utilize is the posterior information. The bidirectional LSTM combines the outputs of the forward LSTM and the reverse LSTM so that its output at the current time node can then utilize both the forward and backward information in both directions, unlike the normal LSTM which can only use the forward information. There is no state sharing between the two different directions of the LSTMs that constitute a bidirectional LSTM, i.e., the output state of the forward LSTM only have an effect on the forward LSTM. The output state of the reverse LSTM only have an effect on the reverse LSTM and there is no direct connection between them and they will not affect each other.

The inputs of each time node are passed to the forward and reverse LSTMs respectively. These LSTMs have no effect on each other according to their respective inputs. The forward LSTM can produce outputs according to their respective states. These two outputs are connected together to the output node of the bidirectional LSTM, and together combine to form the final output. Although the two directions of the LSTM network LSTMs basically do not intersect, their contribution to the output of the current time node and the resulting loss can then be computed during training, and their parameters are optimized based on the gradient.

Fig. 3The Bi-LSTM scheme with CRF

In this paper, we use the Bi-LSTM aided by CRF to extract the entities, since BI-LSTM is able to handle the bidirectional long-term dependency with a large span in distance between entities. Since the total content of the constructed knowledge graph is very large, some of the contents of the knowledge graph are preferentially selected by using the attention scheme. In the following, we first focus on entity extraction with CRF from the given document, as shown in Fig. 3.

Since the CRF layer can learn the transition probability among different labels, the probability of the derived entities can be guaranteed. In this case, the CRF can be computed with the state of the entities. The entity as $X$ with the label $y$ can be calculated by:

11

$\mathrm{s}\left(X,y\right)=\sum _{i=0}^n{A}_{y_i,y_{i+1}}+\sum _{i=1}^n{P}_{i,y_i},$

where $A$ is the transmitting probability matrix, from $y_i$ to $y+1$, $P$ is the output matrix of the Bi-LSTM. We denote the $i$-th label as $y_i$. Given $X$, the probability of the label $y$ is:

12

$\mathrm{P}\mathrm{r}\left(y|X\right)=\frac{e^{s\left(X,y\right)}}{{\sum }_{y^{\mathrm{\text{'}}}\in Y_x}{e}^{s\left(X,y^{\mathrm{\text{'}}}\right)}},$

where $Y_x$ is the label for $X$ and $y\mathrm{\text{'}}$ is the real label. Thus, the training target is to maximize $\mathrm{P}\mathrm{r}\left(y\right|X)$. Thus, we adopt the loss function as:

13

$\mathrm{l}\mathrm{o}\mathrm{g}\left(p\left(y\right|X)\right)=s\left(X,y\right)-\mathrm{l}\mathrm{o}\mathrm{g}\left({\sum }_{y^{\mathrm{\text{'}}}\in Y_x}{e}^{s\left(X,y^{\mathrm{\text{'}}}\right)}\right).$

Accordingly, the maximal probability of $X$ labelled with y can be given by:

14

$y^{\mathrm{*}}=\mathrm{a}\mathrm{r}{\mathrm{g}}_{y^{\mathrm{\text{'}}}\in Y_x}\mathrm{m}\mathrm{a}\mathrm{x}s\left(X,y\right).$

3.2. Relation exaction with Bi-LSTM aided by attention

In this subsection, we use the attention model to reduce the number of relations. Since the attention model can assign more attention to important parts, we add an attention layer between the output layer and the hidden layer of the Bi-LSTM to adjust the weight of the output, as shown in Fig. 4.

Fig. 4The LSTM with attention model

In the Fig. 4, $X_t^s$ is the input of the of the $s$-th visiting point, $H_t^s$ is the output of the Bi-LSTM of the $t$-th time slot of the $s$-th visiting point, ${\beta }_t$ is the output of normalization module for the $t$th time slot, and $H_t^{\alpha }$ is the output of the Bi-LSTM with attention mechanism with $\alpha$ is the weight obtained by attention mechanism.

The proposed models based on the attention mechanism generally contain two computational processes, one is the computational process about the probability distribution, and the other is the computational process of the final features with the attention distribution. Let be the attention probability of the output data for the final state at the $n$-th time slots, which can be given by:

15

$a_n=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(h_n^{\text{'}}\right)}{\sum _{i=1}^N\mathrm{e}\mathrm{x}\mathrm{p}\left(h^{\text{'}}\right)},$

where $N$ denotes the number of input sequence elements, $h_n^{\mathrm{\text{'}}}$ is the sum of the hidden layer state values in both directions at the $n$-th time slot, expressed as:

16

$h_n^{\text{'}}=h_n^{T}UF,$

where $T$ is the number of category labels, $U$ is the weight matrix, $F$ denotes the sum of the final hidden layer state values in each independent direction in the Bi-LSTM. Let $F_{att}$ be the feature of attention distribution, which can be expressed as:

17

$F_{att}=\sum _{n=1}^N{a}_n{h}_n.$

With $F_{att}$, the probability distribution of the classification labels is calculated with a soft-max function in the output layer. Thus, the calculation process is given by:

18

$y=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(F_{att\left(i\right)}^{\mathrm{\text{'}}}\right)}{\sum _{j=1}^T\mathrm{e}\mathrm{x}\mathrm{p}\left(F_{att\left(j\right)}^{\mathrm{\text{'}}}\right)},$

where $F_{att}^{\mathrm{\text{'}}}=V{F}_{att}$ with $V$ being the weight matrix of the output layer, which represents the value of the $i$-th component in the vector. The vector length is equal to the number of classification labels.

By using the softmax function, the probability distribution of $y$ under the attention mechanism can be obtained. Note that, the loss entropy can be calculated with:

19

$E\left(Y,y\right)=-Y\mathrm{l}\mathrm{g}\left(y\right),$

where $Y$ denotes the probability distribution of the true category.

3.3. Automatically diagnose algorithm for rational path

In this paper, the training data is taken from the fault manual of a hydro-power plant. This document contains the possible failure modes (failures that occur), failure manifestations, failure causes, and resolution measures for the hydro-power plants. In this document, parts of the faults can be described in Table 1.

In this example, we can obtain the relations. (1) R1 = Located (C11, C13) represents that the wrong operation place is located at the operator. (2) R2 = Relative (C4, C9) represents that the transform failure relates with mechanical improper. (3) R3 = Suggest (C4, C10) represent the advice to replace. (4) R4 = HasP (C3, C9) the false appearance. According to relations, a knowledge graph can be given as Fig. 5.

With the obtained knowledge graph, we design the algorithm to diagnose the fault of the devices as shown in Table 2.

In Algorithm 1, the operation follows these steps. In step 1, the input set denoted as $P=\left\{p_1,p_2,...,p_n\right\}$ is the phenomenon set. visit the false phenomenon set $P$, to judge whether $P\subseteq G$, where $G$ is the entity set. If $P\subseteq G$ holds, the entity set can be used to diagnose the reason, and thus $P_d$ is updated by $P$. Otherwise, the entities are deleted from $P$.

In step 2, we use the knowledge mat to obtain the relations. By consulting the Entity-Relation table, the reverse relation $R^{-1}$ to create the related faults reason set $F_d$$P_d\stackrel{R_1}{\to }{F}_0$. Create fault relation value $M=range\left(R^{-1}\left(C_i,C_j\right)\right)$.

Table 1Entity relation example

Entity 1	Relation	Entity 2
Insulation fault (C1)	Cause by	Generator abnormal (C6)
Circuit breaker (C2)	HasP	Relay malfunction (C7)
Electrical equipment fault (C3)	HasP	Smoke (C9)
Transformer failure (C4)	Relative	Mechanical improper (C5)
Transformer failure (C4)	Suggestion	Replace (C10)
Mechanical improper (C5)	Caused_by	Wrong operation (C11)
Generator fault (C6)	Is_a	Mechanical improper (C5)
Instrumentation failure (C8)	Is_a	Mechanical improper (C5)
Transformer failure (C4)	Is_a	Electrical equipment fault (C3)
Wrong operation (C11)	Suggestion	Cut-off power (C12)
Wrong operation (C11)	Located	Operator pointer (C13)

Fig. 5Knowledge graph visualization example for hydro-power systems

Table 2Algorithm 1: automatically diagnose algorithm

Input: false phenomenon set $P$ Output: All Relation Path $F_R$ 1: Visit false phenomenon set 2: if ($P\subseteq G$) {$P_d=P$} 3: Else $P_d=\left\{p_i\right|p_i\in G\text{and}{p}_i\in P\}$ 4: while (The $F_R$ is not empty) { 5: Create a Relation $R_i$ 6: $S=S\cup \left\{R_i\right\}$ 7: i++} 8: repeat { 9: If ($P_d=R\_i\left(S\right)$) Append ($P_{th}$, $R_i$, Entity $P_i$) 10: Else $R^{-1}\{S，C_i\}=NULL$} 11: until ($P_I$ is not in Left $R$) 12: Order the Fault Reason $P$th

In step 3, verify that there exists a relationship path $R\left(C_i,C_j\right)$ between the set of associated structural entities $S$ and the set of fault phenomena $P_d$. If the relationship exists, the set of indirectly associated failure phenomenon entities $P_i=range\left(R\left(C_i,C_j\right)\right)$ is introduced by $S$ according to the relationship path $R\left(C_i,C_j\right)$. If it does not exist, the search failures and the set of indirectly associated failure causes, denoted as $F_i=\varnothing$.

In step 4, Iteratively derive the rational path according to the ration set. Create indirect relation according to $S$ until $R^{-1}\{S,C_i\}=NULL$. Finally, output the diagnose reason path, with the form as $T_0\stackrel{R_1}{\to }{T}_2\stackrel{R_2}{\to }{T}_4\stackrel{R_3}{\to }{T}_6$.

4. Experimental result and analysis

In this section, we present the experimental results of the proposed fault diagnosis method aided by Bi-LSTM with attention. Neo4j was used to build knowledge graphs, since Neo4j has many advantages, including high performance, design flexibility, and development agility. In addition, building the data model with Neo4j is intuitive and clear. To extract the entities and their relations from unstructured knowledge, the training parameters of the LSTM scheme are listed in Table 3.

In addition, the number of data for the attention scheme is listed in Table 4.

In this paper, we employ three indicators to evaluate the quality of relation extraction, i.e., accuracy and recall, and F1. First, the accuracy of the relation extraction is given by:

where $T_P$ is the number of characters actually extracted for a label and manually labeled for that label, $F_P$ is the number of characters manually labeled for a label and not actually extracted for that label.

Second, the recall rate is given by:

where $F_N$ is the number of characters actually extracted for a label and not manually labeled for that label.

Third, F1 (F-measure) represents the average trade-off between recall rate and accuracy, which can more comprehensively evaluate the recall rate and accuracy. The calculation of F1 is given by:

The tested trained model is used to extract the entities in the remaining original corpus to complete the relation extraction based on the attention mechanism of the BI-LSTM algorithm. We use the sampling method to create the training set, testing set, and verification set. For the different types of fault occurred faults, the accuracy of the extraction results is shown in Fig. 6.

Fig. 6The testing results with proposed diagnose algorithm

Fig. 7The testing results with proposed diagnose algorithm

As shown in Fig. 6, the proposed method uses knowledge map technology to exploit the relationships between the device structures and incorporate them into the diagnostic reasons. The experimental results show that the accuracy of the test cases for this method is 90.6 % to achieve effective utilization of the relationship between device structures. Second, we can observe that the recognition is much better than that of fault location and state. The reason is due to the fact that the description of repair suggestions and fault phenomena is highly structured and consists of two or three words, e.g., oil spill, insufficient power. The accuracy of fault location and fault status is lower, since the description of the positions of fault location and fault status in the sentence is more random, which leads to lower detection accuracy. In addition, the context information of fault location and fault status is highly uncertain, and thus cannot be normally trained and tested with small data sets.

From Table 5, we can observe that the recognition result obtained by the Bi-LSTM with CRF model labeling scheme, with F1 is 90.25 %. The fault diagnosis accuracy of this method is 92.1 %, which is much higher than that of the traditional rule-based fault diagnosis methods. Therefore, the proposed method can effectively solve the problem of inaccurate fault diagnosis in hydro-power plants.

To evaluate the efficiency of the automatic fault diagnosis algorithm, we compare the proposed diagnosis algorithm with two benchmark methods, i.e., the diagnosis algorithm with Petri net and Bayesian net, as shown in Fig. 7. First, we can observe that the average diagnosis time with the proposed algorithm is much shorter than that with the Petri net and Bayesian network. The reason is that the Petri net and Bayesian net cannot automatically create the relationship paths, and the online detector of the Petri net and Bayesian net needs much time to find the final reason for the faults. Second, the Petri-net-based diagnosis method is much faster than the Bayesian net algorithm, since the calculation of the transmission probability of the different events.

5. Conclusions

In this paper, we propose a novel method based on knowledge graph to diagnose faults for hydro-power plants, where the knowledge graph can be used to intelligently search for the reasons for occurred faults. We first construct the knowledge graph for typical hydro-power plants, and the formulated rational path is used to trace the headstream of faults. Second, we develop the Bi-LSTM with CRF to label the entities in the given documents that recorded the phenomenon and the reasons for the faults. In addition, a Bi-LSTM with an attention scheme was developed, which can be used to select the closer relationships between the entities. Finally, we design an automatic diagnosis algorithm to improve the efficiency of diagnosis by constructing relationship paths to trance the headstream for the occurred faults. Simulation results reveal that the intelligent search based on the knowledge graph has good applications in the field of fault detection.