CN117036665A - Knob switch state identification method based on twin neural network - Google Patents

Abstract

This application discloses a method for identifying the state of a knob switch based on a twin neural network, which relates to the field of electric power technology. The method first performs image preprocessing on the acquired original preset image, and extracts the knob switch in the state to be identified after image correction. Positive-angle images to reduce the impact of complex backgrounds and lighting as well as changing shooting angles, and then use a similarity calculation model based on twin neural networks to learn and express the status characteristics of the knob switch and the differences between each gear status It determines the image similarity between the image to be recognized and the standard image of each knob switch to achieve state recognition. It has good recognition accuracy and robustness. It is very suitable for automation control, smart substations and other fields, and is conducive to improving equipment. Monitoring and management efficiency and security.

Description

A method for identifying the status of knob switches based on twin neural networks

Technical field

This application relates to the field of electric power technology, in particular to a method for identifying the status of a knob switch based on twin neural networks.

Background technique

The substation is the connection point of the power grid lines in the power system. It plays a key role in maintaining the smooth operation of the power system and normal human production and life. It is of vital significance to ensure the normal operation of the substation. As an indispensable component of the substation, the rotary switch is widely used in various equipment and machines. The rotary switch realizes the distribution and dispatch of electric energy by turning on and off the circuit. When the circuit fails, the rotary switch can cut off the power supply in time. , to prevent fault expansion and ensure the stable and safe operation of the circuit. Therefore, the rotary switch plays an important role in the operation and maintenance of the substation. Determining the status of the rotary switch is very critical to ensure the normal operation of the substation.

The traditional approach is to conduct manual inspections of various equipment in the substation. Specific personnel are responsible for checking the status of knob switches and other equipment in designated areas every day and recording them. However, manual inspection methods have low automation and efficiency, and are affected by human factors, making it difficult to guarantee accuracy.

With the development of visual recognition technology, there are also some practices that will introduce image processing technology to automatically identify the status of knob switches. However, it is necessary to manually set the discrimination criteria in advance according to the unique characteristics of each knob switch. Therefore, as the number of knob switches increases, With the continuous increase of data and types, it is difficult to comprehensively cope with the real-time requirements of big data streams, and traditional image processing technology also has problems such as low recognition accuracy and sensitivity to environmental changes.

Therefore, further, deep learning-based methods are gradually introduced into the problem of state identification of knob switches. There are currently two mainstream methods: one method is to use classification networks such as VGG16, MobileNetV3, and ResNet to only include knob switches. The images are used for state recognition respectively. Another approach is to use target detection models such as YOLO series and R-CNN series to directly find the knob switch and identify its state in images containing a large amount of background. However, the environmental factors of the substation are complex, so the recognition effects of the above two methods are unstable, and the anti-interference ability is insufficient, and there is a high probability of false detection. For example, in the target detection model, the knob switch accounts for a very large proportion in the image containing the background. It is difficult to detect the knob switch in small situations, let alone identify the state. The classification network is prone to misclassification of states with small differences (such as the rotation angle of the knob switch is small or the gear is rotated 180°).

Contents of the invention

In response to the above problems and technical needs, this application proposes a knob switch state identification method based on twin neural networks. The technical solution of this application is as follows:

A method for identifying the status of a knob switch based on twin neural networks. The method for identifying the status of a knob switch includes:

Obtain the original preset image, which contains the knob image and background image of the area where the knob switch in the to-be-identified state is located;

Perform image preprocessing on the original preset position image, and extract the image-corrected knob image in the original preset position image as the image to be identified. The image to be identified is the positive angle image of the knob switch in the to-be-identified state after image correction;

Traverse each knob switch standard image in the knob switch standard library in sequence, input each knob switch standard image traversed and the image to be recognized into the pre-trained similarity calculation model at the same time, and obtain the difference between the knob switch standard image and the image to be recognized. Image similarity; among them, the rotary switch standard library includes rotary switch standard images of various types of rotary switches in various gear states, and each rotary switch standard image is obtained from the shooting angle facing the rotary switch. Positive angle image of the area where the knob switch is located; the similarity calculation model is built and trained in advance based on the twin neural network;

According to the image similarity between the image to be recognized and the standard image of the knob switch in different gear states, the gear state of the knob switch in the state to be recognized is obtained.

Its further technical solution is that the knob switch state identification method also includes:

Take original sample images of various types of rotary switches in various gear states from different shooting angles. Extract the image of the area where the rotary switch is located from each original sample image and mark the gear state of the corresponding rotary switch as Training samples are used to construct a training data set;

The network framework for constructing the similarity calculation model includes twin neural networks, fully connected layers and softmax layers. The twin neural network includes two feature extraction modules that are connected together and share network parameters. Each feature extraction module uses ResNet50 as the basic network structure and Introducing the CBAM module to learn the attention mechanism;

The network framework based on the similarity calculation model of the training data set is used for model training, and the trained similarity calculation model is obtained. The two feature extraction modules in the similarity calculation model independently obtain an input image and output a feature map. The twin neural network The network connects the feature maps output by the two feature extraction modules together to form the final feature representation, and then passes through the fully connected layer and the Softmax layer in sequence to calculate the Euclidean distance to obtain the image similarity between the two images.

The further technical solution is to obtain the gear position state of the knob switch in the state to be identified, including:

After obtaining the image similarity between the standard image of each knob switch and the image to be identified, the average value of the image similarity between the standard image of the knob switch and the image to be identified with the same gear status is calculated as the gear corresponding to the gear status. similarity;

The gear state with the highest corresponding gear similarity is used as the gear state of the rotary switch in the state to be identified.

The further technical solution is to extract the image-corrected knob image from the original preset image as the image to be recognized, including:

Determine the preset image template corresponding to the preset shooting position for capturing the original preset position image. Different preset shooting positions have different preset position image templates. The preset position image template corresponding to each preset shooting position indicates The area where the knob switch is located in the image taken at the preset shooting position;

Use the determined preset image template to perform image preprocessing on the original preset image, and extract the image to be recognized.

The further technical solution is to use the determined preset image template to perform image preprocessing on the original preset image, including:

Use the convolutional neural network to extract features from the preset image template and the original preset image to obtain respective feature maps;

Perform feature matching on the two feature maps to achieve image alignment between the original preset position image and the preset position image template, and combine the area where the knob switch in the preset position image template is extracted to obtain the state to be recognized in the original preset position image. The image of the knob in the area where the knob switch is located;

Perform image correction on the extracted knob image, and extract the image to be recognized.

The further technical solution is to extract the knob image in the area where the knob switch in the to-be-identified state is located in the original preset image, including:

Detect the image feature points of the preset image template based on the feature map of the preset image template, and detect the image feature points of the original preset image based on the feature map of the original preset image;

The random sampling consistency algorithm is used to filter the image feature points of the preset image template and the original preset image, and the four pairs of image feature points that best match are obtained. Each pair of image feature points includes the preset An image feature point in the bit image template and an image feature point in the matching original preset bit image;

according to The transformation matrix T ₁ is obtained by solving the coordinates of four pairs of image feature points. (x ₁₁ , y ₁₁ ) and (x ₂₁ , y ₂₁ ) are the coordinates of a pair of image feature points, (x ₁₂ , y ₁₂ ) and (x ₂₂ , y ₂₂ ) are the coordinates of a pair of image feature points, (x ₁₃ , y ₁₃ ) and (x ₂₃ , y ₂₃ ) are the coordinates of a pair of image feature points, (x ₁₄ , y ₁₄ ) and (x ₂₄ , y ₂₄ ) are the coordinates of a pair of image feature points, and (x ₁₁ , y ₁₁ ), (x ₁₂ , y ₁₂ ), (x ₁₃ , y ₁₃ ) and (x ₁₄ , y ₁₄ ) are all original preset images The image feature points in (x ₂₁ , y ₂₁ ), (x ₂₂ , y ₂₂ ), (x ₂₃ , y ₂₃ ), (x ₂₄ , y ₂₄ ) are all image feature points in the preset image template;

The transformation matrix T ₁ is used to perform coordinate transformation on the position coordinates of the area where the knob switch is located in the preset position image template, and the position coordinates of the area where the knob switch is to be identified in the original preset position image are obtained and the knob image is extracted.

The further technical solution is to perform image correction on the extracted knob image including:

Determine the coordinates A(X ₁ , Y ₁ ), B(X ₂ , Y ₂ ), C(X ₃ , Y ₃ ) and D(X ₄ , Y _{4 )} of the four vertices of the knob image in the original preset image );

According to the coordinates of the four vertices, the Euclidean distance W _a between vertex A and vertex B, the Euclidean distance Wb between vertex C and vertex D, the Euclidean distance Ha between vertex A and vertex C, vertex B and vertex D are calculated. The Euclidean distance Hb between;

Determine the coordinates of the four vertices of the transformed image as A′(0,0), B′(W,0), C′(0,H) and D′(W,H), where W is Wa The maximum value among and Wb, H is the maximum value among Ha and Hb;

according to The transformation matrix T ₂ is obtained by solving, and the transformation matrix T ₂ is used to perform image transformation on the knob image in the original preset image to obtain the image to be recognized.

The further technical solution is to use the transformation matrix T ₂ to perform image transformation on the knob image in the original preset image to obtain the image to be recognized, including:

Use the transformation matrix T ₂ to perform image transformation on the knob image in the original preset image to obtain an output image. The output image is the positive angle image of the knob switch in the state to be recognized;

Calculate the grayscale histogram of each pixel in the output image, and calculate the cumulative distribution function of each grayscale level in the grayscale histogram. The cumulative distribution function Sk(g) of any grayscale level g is equal to the grayscale level in [0 , g] The sum of the number of pixels within the range, min≤g≤G, min is the minimum value of the gray level, G is the maximum value of the gray level;

according to Convert the gray level of any pixel with gray level g in the output image to g′, N is the total number of pixels contained in the output image, Expresses yes/> Perform rounding.

The further technical solution is that detecting the image feature points of the preset position image template and detecting the image feature points of the original preset position image include inputting any one of the preset position image template and the original preset position image:

Extract each key pixel point and the descriptor of each key pixel point in the feature map of the input image. Each key pixel point not only has the largest spatial feature value within the predetermined local neighborhood range, but also has the largest channel value in the channel direction. Feature value; the descriptor of each key pixel is the channel direction vector of the location of the key pixel;

According to the extracted key pixel points and the descriptors of each key pixel point, each key pixel point is restored to the image size of the input image through bilinear interpolation, and the image feature points of the input image are extracted.

A further technical solution is to use a convolutional neural network to extract features from the preset image template and the original preset image to obtain respective feature maps, including for any of the preset image template and the original preset image. An input image:

Perform two convolutions and maximum pooling combined downsampling on the input image to obtain the deep feature map of the input image. The image size of the deep feature map is 1/4 of the image size of the input image;

Convolution, average pooling and atrous convolution are performed on the deep feature map again, and the feature map of the input image is obtained by fusion extraction.

The beneficial technical effects of this application are:

This application discloses a method for identifying the state of a knob switch based on a twin neural network. This method uses a similarity calculation model based on a twin neural network to learn and express the state characteristics of the knob switch and the differences between the states of each gear, and it also The image preprocessing process of the original preset image can effectively reduce the impact of complex background and lighting as well as changing shooting angles. After extracting the image to be identified, the image similarity between the knob switch standard image and the image to be identified is determined. It has good recognition accuracy and robustness. It is very suitable for automation control, smart substations and other fields, and is conducive to improving the efficiency and safety of equipment monitoring and management.

When constructing the twin neural network, the similarity calculation model used in this application uses ResNet50 as the basic network structure and adds the CBAM attention mechanism, so that the deep convolutional neural network can focus on the extraction of knob features and have better feature extraction effects. .

Description of the drawings

Figure 1 is a method flow chart of a knob switch state identification method according to an embodiment of the present application.

Figure 2 is a flow chart of a method for training a similarity calculation model in one embodiment of the present application.

Figure 3 is a flowchart of a method for performing image preprocessing and extraction on an original preset image to obtain an image to be recognized in one embodiment of the present application.

Figure 4 is a schematic diagram of the transformation of the output image obtained after transforming the knob image in the original preset image in one embodiment of the present application.

Detailed ways

The specific embodiments of the present application will be further described below in conjunction with the accompanying drawings.

This application discloses a method for identifying the status of a knob switch based on a twin neural network. Please refer to the flow chart shown in Figure 1. The method for identifying the status of a knob switch includes the following steps:

Step 1: Obtain the original preset image.

In the application scenario of the substation, there are several cameras fixed at multiple different locations in the substation. Each camera can be rotated to several different target angles to capture images within the field of view. Each camera at each target angle The shooting position is a preset shooting position, and the image obtained by the camera at a preset shooting position is the original preset position image in this application. Since the fixed positions of each camera are predetermined, and the target angles at which each camera can rotate are also predetermined, all preset shooting positions in the entire substation can be determined. Every image obtained in this application can be determined. The original preset position images were captured at one of the preset shooting positions in the substation, and the preset shooting position at which the images were captured can be specifically determined.

Due to the complex substation environment, the original preset image captured at each preset shooting position often includes not only the knob switch, but also other equipment in the substation, so each original preset image captured In addition to the knob image of the area containing the knob switch in the state to be recognized, a background image is also included.

Step 2: Perform image preprocessing on the original preset image, and extract the image-corrected knob image from the original preset image as the image to be recognized.

Since there are many types and quantities of knob switches in a substation, it is generally impossible to set up a camera for each knob switch to take pictures of it. Therefore, the original preset image obtained is often not in the state to be identified. Images captured using a knob switch are captured from various non-uniform tilt angles and are also affected by other environmental factors.

Therefore, after obtaining the original preset image, image preprocessing is first performed to compensate for the interference caused by the shooting angle and shooting environment. The extracted image to be recognized is the positive angle image of the knob switch in the state to be recognized after image correction.

Step 3: Traverse each knob switch standard image in the knob switch standard library in sequence, and input each knob switch standard image and the image to be recognized simultaneously into the pre-trained similarity calculation model to obtain the knob switch standard image and the image to be recognized. Image similarity between images.

The rotary switch standard library used here is pre-built. The rotary switch standard library includes rotary switch standard images of various types of rotary switches in various gear states, and each rotary switch standard image is facing the knob. The positive angle image of the area where the knob switch is located is obtained from the shooting angle of the switch. Each knob switch has a variety of different gear states according to different gear rotation angles. Each gear state covers a gear rotation angle range of the knob switch and is divided in advance according to the actual situation. For example, starting from 0°, every 30 degrees °The gear rotation angle range is divided to obtain multiple gear states, which can actually be customized and determined. When constructing the rotary switch standard library, for each type of rotary switch in the substation, adjust the rotary switch to different gear states, and use the camera to photograph the rotary switch in each gear state to obtain a rotary switch standard. Image, switch the gear state and shoot again. After obtaining the standard image of the rotary switch in all gear states of the rotary switch, do the same for other types of rotary switches, so as to obtain all types of rotary switches in the substation in all gear states. Knob switch standard image below.

This step also requires the use of a similarity calculation model, which is pre-constructed and trained based on the twin neural network. Its training method will be introduced later.

Step 4: According to the image similarity between the image to be recognized and the standard image of the knob switch in different gear states, the gear state of the knob switch in the state to be recognized is obtained.

In one embodiment, the gear position state of the standard image of the knob switch that has the highest image similarity to the image to be recognized is used as the gear position state of the knob switch in the state to be recognized.

However, in order to improve the recognition accuracy, in another embodiment, after obtaining the image similarity between the standard image of each knob switch and the image to be recognized, calculate the similarity between the standard images of multiple knob switches with the same gear status and the image to be recognized. The average image similarity of the image is used as the gear similarity corresponding to the gear state. After obtaining the gear similarity of each gear state, the gear state with the highest corresponding gear similarity is used as the gear state of the knob switch in the state to be identified.

When applying step 3 above, a similarity calculation model needs to be used. Therefore, this method also includes a method of pre-training the similarity calculation model, including the following steps. Please refer to the flow chart shown in Figure 2:

1. Take original sample images of various types of rotary switches in various gear states from different shooting angles, extract the image of the area where the rotary switch is located from each original sample image, and mark the corresponding gear position of the rotary switch. The status is used as a training sample to construct a training data set.

The method of constructing the training data set is similar to the method of constructing the rotary switch standard library mentioned above, and in actual application, the constructed training data set contains each rotary switch standard image in the rotary switch standard library, so the two parts share the same process. That is, while building the training data set, you can also build the knob switch standard library.

The difference from building a rotary switch standard library is that for each type of rotary switch in the substation, when the rotary switch is adjusted to each gear state, in addition to using the camera to shoot the rotary switch, it also uses The camera captures the knob switch at several other different angles.

A three-dimensional spatial virtual coordinate system is established with the rotary switch. The x0y plane of the spatial three-dimensional virtual coordinate system is parallel to the horizontal plane, the z-axis direction is perpendicular to the horizontal plane and upward, the y-axis direction is toward the front of the rotary switch, and the x-axis direction is perpendicular to the y-axis direction. Then multiple different shooting angles include horizontal shooting angles with different x-axis directions relative to the knob switch, vertical shooting angles with different z-axis directions relative to the knob switch, and different along-axis shooting angles relative to the knob switch. The front-to-back distance in the y-axis direction. In one embodiment, for each type of rotary switch in each gear state, the dome camera is used to sequentially move along the x-axis direction within the range of an angle of 30° to 150° with the positive x-axis direction. Move and shoot the knob switch in steps of 30° each time. And, use the dome camera to sequentially move and photograph the knob switch along the z-axis direction within the angle range of 30° to 150° in the positive direction of the z-axis, with each step of 30°. And, use the dome camera to move the dome camera sequentially along the y-axis direction within a range of 0.5m to 2m from the front and back of the knob switch and photograph the knob switch, with a step length of 0.5m.

After each original sample image is captured, the image of the area where the knob switch is located is extracted from the original sample image and the background image is removed, and then the corresponding gear status is marked.

2. The network framework for constructing the similarity calculation model includes twin neural networks, fully connected layers and softmax layers. The twin neural network includes two feature extraction modules that are connected together and share network parameters. The shared network parameters of the two feature extraction modules can be increased. network efficiency and reduce the number of parameters. Each feature extraction module uses ResNet50 as the basic network structure and introduces the CBAM module to learn the attention mechanism.

3. Use the network framework of the training data set based on the similarity calculation model to conduct model training, and obtain the trained similarity calculation model.

The two feature extraction modules in the similarity calculation model independently obtain an input image and output a feature map. The twin neural network connects the feature maps output by the two feature extraction modules together to form the final feature representation, which is sequentially passed through the fully connected layer and After the Softmax layer, the Euclidean distance is calculated to obtain the image similarity between the two images.

Cross-matching training of the similarity calculation model using training samples of different types of knob switches in different gear states and different shooting angles in the training data set can make the similarity calculation model have stronger feature extraction capabilities and robustness.

After the similarity calculation model in step 3 above obtains the standard image of the knob switch and the image to be recognized, it first standardizes the standard image of the knob switch and the image to be recognized into the same image shape and image size, and then sends the two images into Forward propagation is performed in the feature extraction module. The two images will pass through the shared layer and independent layer of the twin neural network respectively to obtain their respective feature maps. The feature vector at this stage is a high-level representation of the image and is used to describe the characteristics of the image. and structure. Finally, the feature maps of the two images are connected together to form the final feature representation of the twin neural network, and the Euclidean distance is calculated through subsequent processing modules such as the fully connected layer and Softmax layer to obtain the standard image of the knob switch and the image of the image to be recognized. Similarity.

In one embodiment, when performing image preprocessing on the original preset position image in step 2 above, first determine the preset position image template corresponding to the preset shooting position of the original preset position image, and then use the determined preset position image template. The positioning image template performs image preprocessing on the original pre-positioning image and extracts the image to be recognized. Different preset shooting positions have different preset position image templates. The preset position image template corresponding to each preset shooting position indicates the area where the knob switch is located in the image taken at the preset shooting position. Each preset shooting position The corresponding preset image template is predetermined.

Using the determined preset image template to perform image preprocessing on the original preset image includes the following steps, please refer to the flow chart shown in Figure 3:

1. Use the convolutional neural network to extract features from the preset image template and the original preset image to obtain respective feature maps.

In order to capture the key information in the image, for any input image in the preset image template and the original preset image, extracting the feature map of the input image includes: (1) Convolving the input image twice and maximizing Pooling is combined with downsampling to obtain a deep feature map of the input image. The image size of the deep feature map is 1/4 of the image size of the input image. Doing so prevents excessive spatial features from being lost during the convolutional feature extraction process. (2) Perform convolution, average pooling and atrous convolution operations again on the deep feature map to fuse and extract sparse features and obtain a larger receptive field, thereby extracting the feature map of the input image, so that more information can be obtained at the same time. Global image information and more identifiable features.

2. Perform feature matching on the two feature maps to achieve image alignment between the original preset image and the preset image template, and extract the area to be identified in the original preset image based on the area of the knob switch in the preset image template. The image of the knob in the area where the status knob switch is located. include:

(1) Detect the image feature points of the preset image template based on the feature map of the preset image template, and detect the image feature points of the original preset image based on the feature map of the original preset image.

In one embodiment, for any input image in the preset image template and the original preset image, detecting the image feature points of the input image based on the feature map of the input image includes:

Extract each key pixel point and the descriptor of each key pixel point in the feature map of the input image. Each key pixel point not only has the largest spatial feature value within the predetermined local neighborhood range, but also has the largest channel value in the channel direction. Eigenvalues. The descriptor of each key pixel is the channel direction vector of the location of the key pixel. These descriptors have richer and more comprehensive information. Compared with traditional feature point extraction algorithms, they can better resist scale, rotation, and lighting. , perspective and non-rigid transformation and other interferences.

The extracted key pixels keypoints can be expressed as keypoints=space∩channel, where space={max(fp(i;j;:) _(i,j)∈Nb )} represents any pixel point (i, The set of pixels corresponding to the maximum value of the spatial feature value within the predetermined local neighborhood range Nb of j). The predetermined local neighborhood range Nb can be customized. For example, the common setting is 3*3 neighborhood, fp(i ;j;:) represents all the pixels of the output feature map fp at the (i, j) position. channel={max(fp(:;:;k))} represents the set of pixels corresponding to the maximum value of the channel feature value in the channel direction of the feature channel index k in the feature map, fp(:;:;k) represents Output all pixels of the feature map fp in the channel direction of the feature channel index k. The symbol: indicates parameter default.

Then, based on the extraction of each key pixel point and the descriptor of each key pixel point, each key pixel point is restored to the image size of the input image through bilinear interpolation, and the image feature points of the input image are extracted.

(2) Use the random sampling consistency algorithm to filter the image feature points of the preset image template and the original preset image, and screen out the best matching four pairs of image feature points. Each pair of image feature points includes An image feature point in the preset image template and an image feature point in the original preset image that it matches.

(3)According to The transformation matrix T ₁ is obtained by solving the coordinates of the four pairs of image feature points. The transformation matrix T ₁ is a 3*3 matrix.

Among them, (x ₁₁ , y ₁₁ ) and (x ₂₁ , y ₂₁ ) are the coordinates of a pair of image feature points, (x ₁₂ , y ₁₂ ) and (x ₂₂ , y ₂₂ ) are the coordinates of a pair of image feature points, (x ₁₃ , y ₁₁ ) and (x ₂₃ , y ₂₃ ) are the coordinates of a pair of image feature points, (x ₁₄ , y ₁₄ ) and (x ₂₄ , y ₂₄ ) are the coordinates of a pair of image feature points, and ( x ₁₁ , y ₁₁ ), (x ₁₂ , y ₁₂ ), (x ₁₃ , y ₁₃ ) and (x ₁₄ , y ₁₄ ) are all image feature points in the original preset image, (x ₂₁ , y ₂₁ ) , (x ₂₂ , y ₂₂ ), (x ₂₃ , y ₂₃ ), (x ₂₄ , y ₂₄ ) are all image feature points in the preset image template.

(4) Use the transformation matrix T ₁ to perform coordinate transformation on the position coordinates of the area where the knob switch is located in the preset image template to complete the image alignment between the original preset image and the preset image template.

After completing the image alignment between the original preset position image and the preset position image template, since the area of the knob switch in the preset position image template is known, the knob switch in the state to be identified in the original preset position image can be correspondingly obtained The position coordinates of the area are extracted and the knob image is obtained.

A common approach is that the coordinates of the four vertices of the rectangular area where the knob switch is located in the preset image template are known, and the original preset image can be obtained by performing coordinate transformation on these four vertex coordinates using the transformation matrix T ₁ The coordinates of the four vertices in the area where the rotary switch in the state to be identified are located, and then the coordinates of the four vertices are connected in sequence to determine the area where the rotary switch in the state to be identified is located. The image intercepted in this area is the knob image.

3. Perform image correction on the extracted knob image, and extract the image to be recognized.

When the original preset image is captured from the knob switch in the state to be recognized, the boundary of the knob image extracted here forms a rectangular structure. However, due to the shooting angle, the boundary of the knob image extracted here is often not a rectangle. It is usually For an irregular quadrilateral, when performing image correction, it is first necessary to transform the irregular boundary of the knob image into a rectangle, so as to restore the knob image to the form taken at a positive angle as much as possible and reduce the impact of the shooting angle on recognition. Includes the following steps:

(1) Determine the coordinates A (X ₁ , Y ₁ ), B (X ₂ , Y ₂ ), C (X ₃ , Y ₃ ) and D (X ₄ ) of the four vertices of the knob image in the original preset image. , Y ₄ ). As shown in Figure 4, the quadrilateral formed by the four vertices of the knob image is usually irregular. Then the Euclidean distance Wa between vertex A and vertex B, the Euclidean distance Wb between vertex C and vertex D, the Euclidean distance Ha between vertex A and vertex C, vertex B and The Euclidean distance Hb between vertices D.

(2) Based on the Euclidean distance between two adjacent vertices of the knob image, determine the coordinates of the four vertices of the transformed image as A′(0,0), B′(W,0), C′(0 , H) and D'(W, H), where W is the maximum value among Wa and Wb, and H is the maximum value among Ha and Hb.

(3)According to The transformation matrix T ₂ is obtained by solving. The transformation matrix T ₂ is a 3*3 matrix, and the value range of each element in the transformation matrix T ₂ is [1, 3].

(4) Use the transformation matrix T ₂ to perform image transformation on the knob image in the original preset image to obtain the image to be recognized. This step can be completed using the OpenCV library function warpPerspective().

In this step (4), the output image obtained by using the transformation matrix T ₂ to transform the knob image in the original preset image is already a positive angle image of the knob switch in the state to be identified, that is, the original image has been transformed as much as possible. The knob image is restored to the form taken at a positive angle, thereby effectively reducing the impact of the shooting angle on status recognition. However, considering that the complex environment of the substation often has the problem of uneven illumination, further efforts are made to reduce the impact of uneven illumination on status recognition. , in one embodiment, the output image is not directly used as the image to be recognized, but also includes a remapping process of the image to be recognized, including:

(5) Calculate the grayscale histogram of each pixel in the output image, and calculate the cumulative distribution function of each grayscale level in the grayscale histogram. The cumulative distribution function Sk(g) of any grayscale level g is equal to the grayscale level The sum of the number of pixels in the range [0, g], min≤g≤G, min is the minimum value of the gray level, and G is the maximum value of the gray level.

(5)According to Convert the gray level of any pixel with gray level g in the output image to g′, N is the total number of pixels contained in the output image, Expresses yes/> Perform rounding.

From this, it is possible to extract the corrected positive angle image of the knob switch in the state to be identified, that is, to obtain the image to be identified that reduces the influence of the shooting angle and uneven lighting.

The above are only preferred embodiments of the present application, and the present application is not limited to the above embodiments. It can be understood that other improvements and changes directly derived or thought of by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims (10)

1. A method for identifying the status of a knob switch based on twin neural networks, characterized in that the method for identifying the status of a knob switch includes: Obtain an original preset image, which includes a knob image and a background image of the area where the knob switch in the to-be-identified state is located; Perform image preprocessing on the original preset image, and extract the image-corrected knob image in the original preset image as the image to be identified. The image to be identified is the process of the knob switch in the state to be identified. Positive angle image after image correction; Traverse each knob switch standard image in the knob switch standard library in sequence, input each knob switch standard image traversed and the image to be recognized into the pre-trained similarity calculation model at the same time, and obtain the knob switch standard image and all the knob switch standard images. The image similarity between the images to be recognized is described; wherein, the rotary switch standard library includes rotary switch standard images of various types of rotary switches in various gear states, and each rotary switch standard image is based on the normal The positive angle image of the area where the knob switch is located is obtained from the shooting angle of the knob switch; the similarity calculation model is constructed and trained in advance based on the twin neural network; According to the image similarity between the image to be recognized and the standard image of the rotary switch in different gear states, the gear state of the rotary switch in the state to be recognized is obtained. 2. The method for identifying the status of the knob switch according to claim 1, characterized in that the method for identifying the status of the knob switch further includes: Take original sample images of various types of rotary switches in various gear states from different shooting angles. Extract the image of the area where the rotary switch is located from each original sample image and mark the gear state of the corresponding rotary switch as Training samples are used to construct a training data set; The network framework for constructing the similarity calculation model includes a twin neural network, a fully connected layer and a softmax layer. The twin neural network includes two feature extraction modules that are connected together and share network parameters. Each feature extraction module uses ResNet50 as the basic network. Structure and introduce the CBAM module to learn the attention mechanism; The training data set is used to perform model training based on the network framework of the similarity calculation model to obtain the trained similarity calculation model. The two feature extraction modules in the similarity calculation model independently obtain one of the inputs. image and output feature maps. The twin neural network connects the feature maps output by the two feature extraction modules to form the final feature representation, and successively passes through the fully connected layer and the Softmax layer to calculate the Euclidean distance to obtain the image similarity between the two images. Spend. 3. The method for identifying the status of a knob switch according to claim 1, wherein said obtaining the gear position state of the knob switch in the state to be identified includes: After obtaining the image similarity between each rotary switch standard image and the image to be identified, the average value of the image similarity between the rotary switch standard image and the image to be identified with the same gear position is calculated as the gear position. The gear similarity corresponding to the gear status; The gear state with the highest corresponding gear similarity is used as the gear state of the rotary switch in the state to be identified. 4. The knob switch state identification method according to claim 1, wherein the extraction of the image-corrected knob image in the original preset position image as the image to be identified includes: Determine the preset image template corresponding to the preset shooting position of the original preset position image. Different preset shooting positions have different preset position image templates. The preset position image corresponding to each preset shooting position The template indicates the area where the knob switch is located in the image taken at the preset shooting position; The determined preset position image template is used to perform image preprocessing on the original preset position image, and the image to be recognized is extracted. 5. The knob switch state identification method according to claim 4, wherein the image preprocessing of the original preset position image using the determined preset position image template includes: Using a convolutional neural network to perform feature extraction on the preset image template and the original preset image to obtain respective feature maps; Feature matching is performed on the two feature maps to achieve image alignment between the original preset position image and the preset position image template, and the original preset position image is obtained by extracting the area where the knob switch is located in the preset position image template. Set the knob image in the area where the knob switch in the state to be recognized is located in the position image; Image correction is performed on the extracted knob image, and the image to be identified is extracted. 6. The method for identifying the status of a knob switch according to claim 5, wherein the extraction of the knob image of the area where the knob switch in the to-be-identified state is located in the original preset image includes: Detect the image feature points of the preset position image template based on the feature map of the preset position image template, and detect the image feature points of the original preset position image based on the feature map of the original preset position image; The random sampling consistency algorithm is used to filter the image feature points of the preset position image template and the image feature points of the original preset position image, and four pairs of image feature points with the best matching are obtained. Each pair of image feature points Points include an image feature point in the preset image template and an image feature point in the original preset image that matches it; according to The transformation matrix T ₁ is obtained by solving the coordinates of four pairs of image feature points. (x ₁₁ , y ₁₁ ) and (x ₂₁ , y ₂₁ ) are the coordinates of a pair of image feature points, (x ₁₂ , y ₁₂ ) and (x ₂₂ , y ₂₂ ) are the coordinates of a pair of image feature points, (x ₁₃ , y ₁₃ ) and (x ₂₃ , y ₂₃ ) are the coordinates of a pair of image feature points, (x ₁₄ , y ₁₄ ) and (x ₂₄ , y ₂₄ ) are the coordinates of a pair of image feature points, and (x ₁₁ , y ₁₁ ), (x ₁₂ , y ₁₂ ), (x ₁₃ , y ₁₃ ) and (x ₁₄ , y ₁₄ ) are all the original presets The image feature points in the bit image, (x ₂₁ , y ₂₁ ), (x ₂₂ , y ₂₂ ), (x ₂₃ , y ₂₃ ), (x ₂₄ , y ₂₄ ) are all in the preset bit image template Image feature points; The transformation matrix T ₁ is used to perform coordinate transformation on the position coordinates of the area where the knob switch is located in the preset position image template, and the position coordinates of the area where the knob switch is in the to-be-identified state in the original preset position image are obtained and extracted. Get the knob image. 7. The knob switch state identification method according to claim 5, wherein the image correction of the extracted knob image includes: Determine the coordinates A(X ₁ , Y ₁ ), B(X ₂ , Y ₂ ), C(X ₃ , Y ₃ ) and D(X ₄ , of the four vertices of the knob image in the original preset image, _Y4 ); According to the coordinates of the four vertices, the Euclidean distance Wa between vertex A and vertex B, the Euclidean distance Wb between vertex C and vertex D, the Euclidean distance Ha between vertex A and vertex C, and the distance between vertex B and vertex D are calculated. The Euclidean distance Hb2 Determine the coordinates of the four vertices of the transformed image as A′(0,0), B′(W,0), C′(0,H) and D′(W,H), where W is W The maximum value among _a and W _b , H is the maximum value among Ha and Hb; according to The transformation matrix T ₂ is obtained by solving, and the transformation matrix T ₂ is used to perform image transformation on the knob image in the original preset image to obtain the image to be recognized. 8. The method for identifying the status of a knob switch according to claim 7, characterized in that, using the transformation matrix _T2 to perform image transformation on the knob image in the original preset image to obtain the image to be identified includes: : Use the transformation matrix T ₂ to perform image transformation on the knob image in the original preset image to obtain an output image, where the output image is a positive angle image of the knob switch in the state to be identified; Calculate the grayscale histogram of each pixel in the output image, and calculate the cumulative distribution function of each grayscale level in the grayscale histogram. The cumulative distribution function Sk(g) of any grayscale level g is equal to the grayscale The sum of the number of pixels with levels in the range [0, g], min≤g≤G, min is the minimum value of the gray level, and G is the maximum value of the gray level; according to Convert the gray level of any pixel with gray level g in the output image to g′, N is the total number of pixels contained in the output image, Expresses yes/> Perform rounding. 9. The knob switch state identification method according to claim 6, wherein detecting the image feature points of the preset position image template and detecting the image feature points of the original preset position image includes: Either input image of the bit image template and the original preset bit image: Extract each key pixel point and the descriptor of each key pixel point in the feature map of the input image. Each key pixel point not only has the largest spatial feature value within the predetermined local neighborhood range, but also has the largest spatial feature value in the channel direction. Has the largest channel feature value; the descriptor of each key pixel is the channel direction vector of the location of the key pixel; According to the extracted key pixel points and the descriptors of each key pixel point, each key pixel point is restored to the image size of the input image through bilinear interpolation, and the image feature points of the input image are extracted. 10. The knob switch state identification method according to claim 5, characterized in that the use of a convolutional neural network is used to extract features of the preset position image template and the original preset position image to obtain respective features. Figure, including the input image for any one of the preset image template and the original preset image: Perform two convolutions and maximum pooling combined downsampling on the input image to obtain a deep feature map of the input image, where the image size of the deep feature map is 1/4 of the image size of the input image; The operations of convolution, average pooling and atrous convolution are again performed on the deep feature map, and the feature map of the input image is obtained through fusion extraction.