JP7589911B2 - Indication value reading program, indication value reading device, and indication value reading method - Google Patents

Description

The present invention relates to an indication value reading program, an indication value reading device, and an indication value reading method.

For example, factories that manufacture semiconductors typically have many analog meters for measuring the temperature, pressure, and other parameters of each device.

In such factories, workers, for example, patrol the factory at regular intervals and visually check the values indicated by each analog meter. Then, for example, the workers perform necessary calculations using the values of each analog meter to determine whether or not an abnormality (e.g., equipment failure, etc.) has occurred within the factory. As a result, if it is determined that an abnormality has occurred within the factory, the workers, for example, make arrangements to replace the equipment with the abnormality (see Patent Documents 1 and 2 and Non-Patent Documents 1 and 2).

JP 2004-133560 A JP 2017-126187 A

Yixiao Fang, Yan Dai, Guoli He and Donglian Qi, "A Mask RCNN based Automatic Reading Method for Pointer Meter", 38th Chinese Control Conference, 8466-8471. HaoWen Lai, Qi Kang, Le pan and Can Cui, "A Novel Scale Recognition Method for Pointer Meters Adapted to Different Types and Shape", 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), 374-379.

The calculations performed using the values of the analog meters as described above are performed manually by an operator, and may take a certain amount of time. Furthermore, such calculations may need to be performed after moving to a location outside the factory (such as the operator's office).

As a result, workers may not be able to quickly detect abnormalities occurring within the factory, and it may take time to resolve the abnormalities.

The object of the present invention is to provide an indication value reading program, an indication value reading device, and an indication value reading method that enable rapid detection of abnormalities that occur in a factory.

The indication value reading program of the present invention for achieving the above object is characterized in that it causes a computer to execute the following process: extracting partial image data for a display having a pointer from inspection target image data that includes the display; identifying a first orientation indicated by the pointer included in the partial image data based on the extracted partial image data; identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and the range of values that the pointer can indicate on the display; and outputting the identified value indicated by the pointer.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it generates multiple pieces of learning data by adding position information of a display having a pointer, which is included in each piece of learning image data, to each piece of learning image data, and generates a learning model by performing machine learning using the multiple pieces of learning data generated, obtains a value output from the learning model in response to inputting the inspection object image data, and extracts the partial image data corresponding to the obtained value from the inspection object image data.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it generates multiple pieces of learning data by adding information indicating the orientation of a pointer contained in each piece of learning image data for a display device having a pointer, generates a learning model by performing machine learning using the multiple pieces of learning data generated, obtains a value output from the learning model as the partial image data is input, and identifies the orientation of the pointer from the obtained value.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it calculates the length of the pointer from the value, and if the calculated length is within a predetermined range, performs a process to identify the orientation of the pointer from the value.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that, if the length is not within the predetermined range, the process of extracting the partial image data and the process of obtaining the value are performed again before the process of identifying the orientation of the pointer from the value is performed.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it identifies the value indicated by the pointer from the minimum value included in the range of values that the pointer can indicate, the maximum value included in the range of values that the pointer can indicate, the first orientation, the second orientation indicated by the pointer when the pointer indicates the minimum value, and the third orientation indicated by the pointer when the pointer indicates the maximum value.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it calculates a first value by subtracting an angle corresponding to the second orientation from an angle corresponding to the first orientation, calculates a second value by subtracting an angle corresponding to the second orientation from an angle corresponding to the third orientation, calculates a third value by subtracting the minimum value from the maximum value, calculates a fourth value by dividing the first value by the second value, calculates a fifth value by multiplying the fourth value and the third value, and calculates a value indicated by the pointer by adding the minimum value to the fifth value.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it acquires the inspection object image data of the display captured by an imaging device, performs homography transformation on the acquired inspection object image data, and extracts the partial image data from the inspection object image data that has been subjected to homography transformation.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it identifies one or more markers on the display device and performs a homography transformation on the inspection object image data so that the identified one or more markers move to a pre-specified position.

In one embodiment, the indication value reading program of the present invention for achieving the above object is characterized in that it calculates the average value of the values indicated by the pointer and outputs the calculated average value.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it calculates the standard deviation of the value indicated by the pointer identified in the process of identifying the value indicated by the pointer, and outputs the average value if the calculated standard deviation is equal to or greater than a predetermined threshold value.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it calculates the standard deviation of the value indicated by the pointer identified in the process of identifying the value indicated by the pointer, and outputs the standard deviation in addition to the average value.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it calculates the standard deviation of the value indicated by the pointer identified in the process of identifying the value indicated by the pointer, and if the calculated standard deviation is equal to or greater than a predetermined threshold, outputs information indicating that the pointer is vibrating in addition to the average value.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it calculates the difference between the maximum and minimum values indicated by the pointer identified in the process of identifying the value indicated by the pointer, and outputs the average value if the calculated difference is equal to or greater than a predetermined threshold value.

In one aspect, the indication value reading program of the present invention for achieving the above object is characterized in that it calculates the difference between the maximum and minimum values indicated by the pointer identified in the process of identifying the value indicated by the pointer, and if the calculated difference is equal to or greater than a predetermined threshold, outputs information indicating that the pointer is vibrating in addition to the average value.

The indication value reading device of the present invention for achieving the above object is characterized by having a partial image extraction unit that extracts partial image data for a display having a pointer from inspection target image data that includes the display, a direction identification unit that identifies a first orientation indicated by the pointer included in the partial image data based on the extracted partial image data, a value identification unit that identifies a value indicated by the pointer included in the partial image data based on the identified first orientation and the range of values that the pointer can indicate on the display, and a value output unit that outputs the identified value indicated by the pointer.

The method for reading indicated values according to the present invention for achieving the above object is characterized in that it causes a computer to execute the following processes: extracting partial image data for a display having a pointer from inspection target image data that includes the display; identifying a first orientation indicated by the pointer included in the partial image data based on the extracted partial image data; identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and the range of values that the pointer can indicate on the display; and outputting the identified value indicated by the pointer.

The indication value reading program, indication value reading device, and indication value reading method of the present invention make it possible to quickly detect abnormalities occurring within a factory.

FIG. 1 is a diagram showing an example of a configuration of an information processing device 1 according to the first embodiment. FIG. 2 is a diagram for explaining an outline of the indication value reading process in the first embodiment. FIG. 3 is a diagram for explaining an outline of the indication value reading process in the first embodiment. FIG. 4 is a flow chart illustrating details of the indication value reading process in the first embodiment. FIG. 5 is a flow chart illustrating details of the indication value reading process in the first embodiment. FIG. 6 is a flow chart illustrating details of the indication value reading process in the first embodiment. FIG. 7 is a flow chart illustrating details of the indication value reading process in the first embodiment. FIG. 8 is a diagram for explaining details of the indication value reading process in the first embodiment. FIG. 9 is a diagram for explaining details of the indication value reading process in the first embodiment. FIG. 10 is a diagram for explaining details of the indication value reading process in the first embodiment. FIG. 11 is a diagram for explaining details of the indication value reading process in the first embodiment. FIG. 12 is a diagram for explaining details of the indication value reading process in the first embodiment. FIG. 13 is a diagram for explaining details of the indication value reading process in the first embodiment. FIG. 14 is a diagram for explaining details of the indication value reading process in the first embodiment. FIG. 15 is a diagram for explaining details of the indication value reading process in the first embodiment. FIG. 16 is a diagram for explaining details of the indication value reading process in the first embodiment.

Below, an embodiment of the present invention will be described with reference to the drawings. However, such an embodiment does not limit the technical scope of the present invention.

First, a configuration example of an information processing device 1 (hereinafter also referred to as an indication value reading device 1) in the first embodiment will be described. FIG. 1 is a diagram showing a configuration example of an information processing device 1 in the first embodiment.

The information processing device 1 is a computer device, for example, a general-purpose PC (Personal Computer). The information processing device 1 performs a process (hereinafter also referred to as an indicated value reading process) for reading a value indicated by a needle of an analog meter (not shown) (hereinafter also referred to as a display).

The information processing device 1 has the hardware configuration of a general-purpose computer device, and for example, as shown in FIG. 1, has a processor, a CPU 101, a memory 102, a communication interface 103, and a storage medium 104. Each part is connected to each other via a bus 105.

The storage medium 104 has, for example, a program storage area (not shown) that stores a program (not shown) for performing the indication value reading process.

The storage medium 104 also has a storage unit 110 (hereinafter also referred to as storage area 110) that stores information used when performing the indication value reading process. The storage medium 104 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

The CPU 101 executes the program loaded from the storage medium 104 to the memory 102 to perform the indication value reading process.

The communication interface 103 communicates with an operation terminal 2 such as a smartphone via a network NW such as the Internet.

[Outline of the first embodiment]
2 and 3 are diagrams for explaining an outline of the indication value reading process in the first embodiment. Specifically, Fig. 2 is a diagram for explaining the learning stage of the indication value reading process in the first embodiment. Also, Fig. 3 is a diagram for explaining the inference stage of the indication value reading process in the first embodiment.

First, we will explain the learning stage of the indication value reading process in the first embodiment.

As shown in FIG. 2, the first data generation unit 111 of the information processing device 1 generates learning data (hereinafter also referred to as first learning data) by adding information about the position and size of the analog meter in the first learning image data (hereinafter collectively referred to as position information) to learning image data (hereinafter also referred to as first learning image data) that partially shows an analog meter.

Then, the first model generation unit 112 of the information processing device 1 generates a learning model (hereinafter also referred to as the first learning model) by performing machine learning using the multiple first learning data generated by the first data generation unit 111. After that, the first model generation unit 112 stores the generated first learning model in the memory area 110.

In other words, the first model generation unit 112 uses multiple first learning data to generate a first learning model that is used when extracting partial image data of an analog meter from the inspection target image data.

The second data generation unit 113 of the information processing device 1 generates learning data (hereinafter also referred to as second learning data) by adding information indicating the orientation of the pointer of the analog meter in the partial image data extracted by the first learning model generated by the first model generation unit 112 to learning image data (hereinafter also referred to as second learning image data) for an analog meter, for example.

Then, the second model generation unit 114 of the information processing device 1 generates a learning model (hereinafter also referred to as the second learning model) by performing machine learning using the multiple second learning data generated by the second data generation unit 113. After that, the second model generation unit 114 stores the generated second learning model in the memory area 110.

That is, the second model generation unit 114 uses multiple pieces of second learning data to generate a second learning model that is used to identify (estimate) the direction indicated by the needle of an analog meter.

Next, we will explain the inference stage of the indication value reading process in the first embodiment.

The partial image extraction unit 121 of the information processing device 1 extracts partial image data about the analog meter from the verification target image data that partially includes the analog meter.

Specifically, the partial image extraction unit 121 identifies position information indicated by a value output in response to input of the verification target image data to the first learning model stored in the memory area 110. Then, the partial image extraction unit 121 extracts the partial image data corresponding to the identified position information from the verification target image data.

Then, the direction determination unit 122 of the information processing device 1 determines the direction indicated by the needle of the analog meter shown in the partial image data extracted by the partial image extraction unit 121 (hereinafter, also referred to as the first direction).

Specifically, the direction determination unit 122 identifies the first direction indicated by the value output in response to input of partial image data to the second learning model stored in the memory area 110.

Next, the value determination unit 123 of the information processing device 1 determines the value indicated by the needle of the analog meter included in the partial image data extracted by the partial image extraction unit 121 based on the first orientation determined by the direction determination unit 122 and the range of values that can be indicated by the needle of the analog meter.

Then, the value output unit 124 of the information processing device 1 outputs, for example, the value identified by the value identification unit 123 to the operation terminal 2.

In other words, the information processing device 1 in this embodiment automatically identifies the direction in which the analog meter's needle is pointing by using image data showing an analog meter placed in a factory as input to a learning model. Then, the information processing device 1 identifies the value indicated by the analog meter's needle based on the identified direction.

The information processing device 1 then calculates (estimates) the probability of abnormal values occurring in equipment within the factory, for example, from the current and past values indicated by the needles of each analog meter. Then, for example, if the calculated probability of abnormal values occurring exceeds a predetermined threshold, the information processing device 1 determines that there is a high possibility that an abnormality has occurred within the factory.

This enables the information processing device 1 to automatically and quickly determine whether or not there is a high possibility that an abnormality has occurred in the factory. Then, for example, if the information processing device 1 determines that there is a high possibility that an abnormality has occurred in the factory, it becomes possible to notify workers to perform the necessary work (operation).

In this regard, the occurrence of an abnormality in the factory can lead to a major accident in the factory. Therefore, workers need to take prompt action in response to any abnormality that occurs in the factory in order to prevent a major accident from occurring in the factory.

Therefore, for example, if the information processing device 1 determines that there is a high possibility that an abnormality has occurred in the factory, it instructs the workers to take appropriate measures. Also, for example, if the information processing device 1 determines that there is an extremely high possibility that an abnormality has occurred in the factory (if it determines that the situation is such that it can be determined that an abnormality has occurred), it instructs the workers to take emergency measures.

Furthermore, by automatically reading the values indicated by the needles of each analog meter, the information processing device 1 can prevent the occurrence of human errors (reading errors, value input errors, etc.) that occur when an operator performs this reading.

[Details of the First Embodiment]
Next, the details of the indication value reading process in the first embodiment will be described. Figures 4 to 7 are flow charts for explaining the details of the indication value reading process in the first embodiment. Figures 8 to 16 are diagrams for explaining the details of the indication value reading process in the first embodiment.

[Processing in the learning stage]
First, the details of the learning stage of the indication value reading process in the first embodiment will be described with reference to FIG.

As shown in FIG. 4, the first data generation unit 111 waits, for example, until the learning timing arrives (NO in S11). The learning timing may be, for example, the timing when the worker inputs the first learning image data via the operation terminal 2. The learning timing may also be, for example, the timing when the worker inputs information to generate each learning model.

Then, when it is time to learn (YES in S11), the first data generation unit 111 generates first learning data by adding the position information of the analog meter contained in each piece of first learning image data (S12).

Specifically, as shown in FIG. 7, the first data generation unit 111 adds position information of the analog meter to the first learning image data DT11 by using, for example, a bounding box BB11 (bounding box BB11 specified by the operator) that indicates the position and size of the analog meter.

More specifically, the worker prepares, for example, image data corresponding to each frame included in video data captured by an imaging device (not shown) as the first learning image data. The worker then adds analog meter position information to each piece of first learning image data by using a bounding box. After that, the first data generator 111 generates the first learning data by, for example, associating the first learning image data with the coordinates (X and Y coordinates) of the top left corner of the bounding box and the vertical and horizontal lengths of the bounding box.

Then, the first model generation unit 112 generates a first learning model by performing machine learning using the first learning data generated in the processing of S12 (S13).

Next, the second data generation unit 113 generates second learning data by adding information indicating the orientation of the needle in each piece of second learning image data (S14).

Specifically, as shown in FIG. 8, the second data generation unit 113 adds pointer position information to the second learning image data DT12 by using, for example, a bounding box BB12 (bounding box BB12 specified by the operator) indicating the position of the pointer of an analog meter.

More specifically, the second data generation unit 113 acquires, for example, image data output in response to the input of the first learning image data to the first learning model generated in the process of S13 as the second learning image data. Then, the worker adds position information of the pointer to each piece of second learning image data by using a bounding box. After that, the second data generation unit 113 generates the second learning data by, for example, associating the second learning image data with the coordinates (X and Y coordinates) of the top left corner of the bounding box and the vertical and horizontal lengths of the bounding box.

Here, because it is necessary to generate a second learning model with high estimation accuracy, the second data generation unit 113 must generate, for example, hundreds to thousands of second learning image data showing an analog meter with a needle pointing to each scale mark on the analog meter for each scale mark on the analog meter. Furthermore, for example, if multiple analog meters are installed in a factory, the second data generation unit 113 must generate, for example, second learning image data for each analog meter.

In this regard, when performing the indicated value reading process in this embodiment, the worker may prepare only the first learning image data corresponding to, for example, a quarter of the scale of the analog meter (for example, the area between "0.1" and "0.3" on the analog meter shown in FIG. 8) and input these to the first learning model to generate the second learning image data. Then, after generating the second learning data from the generated second learning image data, the worker may further generate new second learning data corresponding to other areas (for example, areas other than the area between "0.1" and "0.3" on the analog meter shown in FIG. 8) by inverting the generated second learning image data horizontally or vertically (horizontal inversion and vertical inversion), thereby generating second learning data corresponding to all areas of the scale of the analog meter.

Specifically, for example, if the value indicated by the needle of the analog meter displayed in the second learning image data DT12 shown in FIG. 8 is "0.178", the value indicated by the needle of the analog meter displayed in image data obtained by flipping the second learning image data DT12 horizontally is "0.422", the value indicated by the needle of the analog meter displayed in image data obtained by flipping the second learning image data DT12 vertically is "0.022", and the value indicated by the needle of the analog meter displayed in image data obtained by flipping the second learning image data DT12 horizontally and vertically is "0.578".

Therefore, in this case, the worker generates new second learning data by adding position information of the pointer when pointing to "0.422" to image data obtained by flipping the second learning image data DT12 horizontally, new second learning data by adding position information of the pointer when pointing to "0.022" to image data obtained by flipping the second learning image data DT12 vertically, and new second learning data by adding position information of the pointer when pointing to "0.578" to image data obtained by flipping the second learning image data DT12 horizontally and vertically.

In other words, the second learning model in this embodiment is not a learning model that recognizes verification target image data that shows an analog meter, but a learning model that detects the direction (angle) indicated by the needle of an analog meter. Therefore, when generating the second learning model, the worker can use, for example, image data in which the characters indicating the scale (such as "0.1") are inverted. Therefore, the worker can generate only second learning data that corresponds to a partial area of the scale of the analog meter, and further generate new second learning data by inverting various parts of the second learning image data included in the generated second learning data.

This allows the worker to reduce the burden required to generate the second learning data. In this case, the worker can also generate a second learning model that can identify (estimate) the orientation of the pointer of other analog meters that have different types of pointers, different numbers of scales, etc.

Then, the second model generation unit 114 generates a second learning model by performing machine learning using the second learning data generated in the processing of S14 (S15).

The worker may input new first learning image data to the information processing device 1 at any time. The information processing device 1 may generate new first learning data including the new first learning image data at any time (S12).

After that, for example, if the proportion of the newly added first learning image data exceeds a predetermined proportion of all the first learning image data, the information processing device 1 may generate the first learning model and the second learning model again (S13 to S15).

This allows the worker to continuously improve the judgment accuracy of the first learning model and the second learning model.

[Processing in the inference stage]
Next, the details of the processing in the inference stage among the details of the indication value reading processing in the first embodiment will be described below. Figures 5 and 6 are diagrams for explaining the details of the processing in the inference stage.

As shown in FIG. 5, the partial image extraction unit 121 waits until the judgment timing arrives (NO in S21). The judgment timing may be, for example, the timing when the worker inputs the verification target image data via the operation terminal 2. The judgment timing may also be, for example, the timing when the worker inputs information to the effect that a judgment will be made on the verification target learning data.

Then, when the determination timing arrives (YES in S21), the partial image extraction unit 121 acquires inspection target image data for the analog meter captured by an imaging device (not shown) (S22).

Specifically, as shown in FIG. 9, the partial image extraction unit 121 receives inspection target image data DT21 of an analog meter transmitted from an imaging device that an operator uses to take images. The imaging device may be, for example, a non-fixed camera. The imaging device may also be, for example, one that is built into the operation terminal 2.

Next, the partial image extraction unit 121 acquires values output from the first learning model in response to input of the inspection target image data acquired in the processing of S22 (S23).

Then, the partial image extraction unit 121 extracts the partial image data corresponding to the value obtained in the process of S23 from the inspection target image data obtained in the process of S22 (S24).

Specifically, as shown in FIG. 10, the partial image extraction unit 121 refers to the position information (coordinates) corresponding to the value acquired in the processing of S23, for example, and acquires the portion of the inspection object image data acquired in the processing of S21 that shows the analog meter as partial image data DT22.

Next, as shown in FIG. 6, the direction determination unit 122 obtains values output from the second learning model in response to input of the partial image data extracted in the processing of S24 (S31).

In this case, the direction determination unit 112 may calculate the length from the axis of rotation of the needle on the analog meter to the tip (the length of the diagonal of the bounding box corresponding to the needle's position) by using the value acquired in the process of S31. If the calculated diagonal length is not within a predetermined tolerance range (for example, within a range of ±5 (%)), the direction determination unit 112 may determine that an error (partial image data reading error) has occurred in the process of S24, and may perform the processes from S24 onwards again.

This enables the direction determination unit 112 to improve the reading accuracy of partial image data in the processing of S24.

Then, the direction determination unit 122 determines the first direction in which the needle of the analog meter is pointing from the value acquired in the process of S31 (S32).

Specifically, as shown in FIG. 11, the direction determination unit 122 identifies the direction DRp as the first direction in which the needle of the analog meter shown in the partial image data DT22 extracted in the processing of S24 is pointing, for example.

Next, the value determination unit 123 determines the value indicated by the needle included in the partial image data extracted in the process of S24 based on the first orientation determined in the process of S32 and the range of values indicated by the needle on the analog meter (S33).

Specifically, the value identification unit 123 identifies the value indicated by the pointer included in the partial image data extracted in the processing of S24, for example, from the minimum value in the range of values that the pointer can indicate (hereinafter also simply referred to as the minimum value), the maximum value in the range of values that the pointer can indicate (hereinafter also simply referred to as the maximum value), the first direction identified in the processing of S32, the direction indicated by the pointer when it indicates the minimum value (hereinafter also referred to as the second direction), and the direction indicated by the pointer when it indicates the maximum value (hereinafter also referred to as the third direction). A specific example of the processing of S33 will be described below.

[Specific example of the process of S33]
The value specification unit 123 specifies the value indicated by the pointer included in the partial image data extracted in the process of S24, for example, according to the following formula 1. Note that, hereinafter, as shown in Fig. 12, the second direction is also referred to as direction DRm, the third direction is also referred to as direction DRM, and the direction (X-axis direction) that is the reference when calculating each angle is also referred to as direction DRX.

In the above formula 1, θ _p indicates the angle when moving clockwise from the direction DRX to the direction DRp, θ _m indicates the angle when moving clockwise from the direction DRX to the direction DRm, and θ _M indicates the angle when moving clockwise from the direction DRX to the direction DRM. Also, in the above formula 1, v _p indicates a value corresponding to the direction DRp within the range of values that the needle can indicate, v _m indicates the minimum value (value corresponding to the direction DRm) among the values that the needle can indicate, and v _M indicates the maximum value (value corresponding to the direction DRM) among the values that the needle can indicate. Note that, when θ _p and θ _M are smaller than θ _m , the value specification unit 123 sets θ _M to the angle obtained by adding 360 degrees to the angle from the direction DRX to the direction DRM.

Specifically, as shown in FIG. 12, for example, when θ _m is 135 degrees, θ _M is 405 degrees, and θ _p is 215 degrees, the value specifying unit 123 calculates v _p to be approximately 0.178.

Returning to FIG. 6, the value output unit 124 outputs the value indicated by the indicator identified in the processing of S33 (S34).

Specifically, the value output unit 124 outputs, for example, the value indicated by the indicator identified in the processing of S33 to the operation terminal 2.

The value output unit 124 may perform a predetermined calculation using the value indicated by the pointer identified in the processing of S33, and output the calculation result to the operation terminal 2. That is, the value output unit 124 may perform a calculation to determine whether or not an abnormality has occurred in the factory, for example, by using the value indicated by the pointer identified in the processing of S33, and output the calculation result to the operation terminal 2.

In this manner, the information processing device 1 in this embodiment extracts partial image data for an analog meter having a pointer from the inspection target image data showing the analog meter. Then, based on the extracted partial image data, the information processing device 1 identifies the first orientation indicated by the pointer included in the partial image data.

Furthermore, the information processing device 1 identifies the value indicated by the needle included in the partial image data based on the identified first orientation and the range of values that the needle can indicate on the analog meter. Then, the information processing device 1 outputs the identified value indicated by the needle.

In other words, the information processing device 1 in this embodiment automatically identifies (reads) the values indicated by each analog meter by inputting image data of each analog meter in the factory.

This allows the information processing device 1 to, for example, calculate (estimate) the probability that an abnormality has occurred in the factory by using the values indicated by each analog meter. Then, for example, when the information processing device 1 determines that an abnormality has occurred in the factory based on the calculated probability, it becomes possible to notify workers to perform the necessary work (operation).

In addition, by automatically reading the values indicated by the needles of each analog meter, the information processing device 1 can prevent the occurrence of human errors (such as reading errors or value input errors) that would occur if an operator were to perform this reading.

Here, the information processing device 1 may acquire video data of an analog meter captured by an imaging device (not shown) in the process of S22. The information processing device 1 may then perform the processes of S23 to S34 on each of a plurality of pieces of inspection target image data (e.g., a plurality of pieces of inspection target image data corresponding to 10 consecutive frames) included in the video data.

Specifically, in this case, the information processing device 1 calculates, for example, the standard deviation of the values calculated in the process of S33 (the values calculated for each of the multiple test target image data). If it determines that the standard deviation exceeds a threshold value, the information processing device 1 determines that the needle of the analog meter shown in the video data acquired in the process of S22 is vibrating, and outputs the average value of the values calculated in the process of S33.

This allows the information processing device 1 to determine whether or not the needle of the analog meter is vibrating. Furthermore, even if the needle of the analog meter is vibrating, the information processing device 1 can output the value indicated by the needle with high accuracy.

In this case, the information processing device 1 may output, for example, in addition to the average value of the values calculated in the process of S33, the standard deviation of the values calculated in the process of S33, or information indicating that the needle of the analog meter shown in the video data acquired in the process of S22 is vibrating. Furthermore, instead of determining whether the standard deviation of the values calculated in the process of S33 exceeds a threshold value, the information processing device 1 may determine whether the difference between the maximum and minimum values of the values calculated in the process of S33 exceeds a threshold value.

Furthermore, the information processing device 1 may output the average value of the values calculated in the process of S33 even when it is determined that the standard deviation does not exceed the threshold value. This enables the information processing device 1 to improve the accuracy of estimating the value indicated by the needle of the analog meter.

[Homography Transformation]
Next, homography transformation of partial image data will be described.

The information processing device 1 may, for example, perform homography transformation on the partial image data extracted in the process of S24 before performing the process of S31. Specifically, the information processing device 1 may, for example, perform homography transformation so that one or more markers on an analog meter shown in the partial image data extracted in the process of S24 move to a pre-specified position. Then, in the process of S31, the information processing device 1 may input the partial image data that has been subjected to homography transformation to a second learning model.

This enables the information processing device to convert, for example, detection target image data showing a tilted analog meter into detection target image data showing the analog meter in the correct orientation (the same orientation as when photographed from directly in front). Therefore, even when the information processing device 1 acquires detection target image data showing a tilted analog meter, it is possible to suppress any decrease in the estimation accuracy of the value indicated by the needle. Specific examples of homography transformation are described below.

[Specific example of homography transformation]
13 and 14 are diagrams for explaining a specific example of homography transformation.

For example, as shown in FIG. 13, the worker attaches four markers MR31, MR32, MR33, and MR34, which are stickers of a color not used in the analog meter, to the outer frame FL of the analog meter that is to be imaged by an imaging device (not shown).

Specifically, as shown in Figure 13, the worker attaches marks MR31 and MR33 to the analog meter to be inspected so that the line connecting marks MR31 and MR33 (hereinafter also referred to as the first line) is perpendicular to the ground and the first line passes through the base end of the pointer (the axis of rotation of the pointer). The worker also attaches marks MR32 and MR34 to the analog meter to be inspected so that the line connecting marks MR32 and MR34 (hereinafter also referred to as the second line) is horizontal to the ground and the second line passes through the base end of the pointer.

As shown in Fig. 14(A), for example, when partial image data DT31 showing an analog meter tilted to the left is extracted in the processing of S24, the information processing device 1 performs homography transformation so that, for example, the first straight line is parallel to the horizontal side of the partial image data and the second straight line is parallel to the vertical side of the partial image data, as shown in Fig. 14(B), to generate partial image data DT31a. After that, the information processing device 1 inputs the partial image data DT31a to the second learning model in the processing of S31.

Similarly, as shown in FIG. 14(C), for example, when partial image data DT32 showing an analog meter tilted to the right is extracted in the processing of S24, the information processing device 1 performs homography transformation so that, for example, the first straight line is parallel to the horizontal side of the partial image data and the second straight line is parallel to the vertical side of the partial image data, as shown in FIG. 14(D), to generate partial image data DT32a. After that, the information processing device 1 inputs the partial image data DT32a to the second learning model in the processing of S31.

[Estimation results of the second learning model when homography transformation is performed]
Next, the estimation result of the second learning model when homography conversion is performed will be described. FIG. 15 and FIG. 16 are diagrams for explaining the estimation result of the second learning model when homography conversion is performed. Specifically, FIG. 15(A) is a scatter diagram showing the estimation result of the second learning model when homography conversion is not performed, and FIG. 15(B) is a histogram showing the estimation result of the second learning model when homography conversion is not performed. Also, FIG. 16(A) is a scatter diagram showing the estimation result of the second learning model when homography conversion is performed, and FIG. 16(B) is a histogram showing the estimation result of the second learning model when homography conversion is performed.

The horizontal axis of the scatter plots shown in Figures 15(A) and 16(A) indicates the value actually indicated by the analog meter, and the vertical axis indicates the error between the value identified in the processing of S33 and the value actually indicated by the analog meter. The horizontal axis of the histograms shown in Figures 15(B) and 16(B) indicates the error between the value identified in the processing of S33 and the value actually indicated by the analog meter, and the vertical axis indicates the number of partial image data input to the second learning model.

Specifically, when compared with the diagrams shown in FIG. 15, the diagrams shown in FIG. 16 show that, for example, the number of partial image data in which the error between the value identified in the processing of S33 and the value actually indicated by the analog meter was 0 has increased. Also, when compared with the diagrams shown in FIG. 15, the diagrams shown in FIG. 16 show that the errors corresponding to each of the partial image data have become smaller overall.

In other words, the examples shown in Figures 15 and 16 show that performing homography transformation on the partial image data extracted in the process of S24 improves the estimation accuracy of the second learning model.

1: Information processing device 2: Operation terminal 101: CPU
102: Memory 103: Communication interface 104: Storage medium 105: Bus

Claims (18)

generating a plurality of learning data by adding information indicating the orientation of a pointer included in each of a plurality of learning image data for a display device having a pointer;
generating a learning model by performing machine learning using the generated plurality of learning data;
Extracting partial image data of a display having a pointer from inspection object image data including the display;
Obtaining a value output from the learning model in response to inputting the extracted partial image data;
Calculating the length of the pointer included in the partial image data from the acquired value;
If the calculated length is within a predetermined range, a first direction indicated by the pointer included in the partial image data is identified from the acquired value ;
identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and a range of values that the pointer can indicate on the display included in the partial image data;
outputting a value indicated by the identified indicator;
An indication reading program for causing a computer to execute a process. In claim 1, further comprising:
generating a plurality of learning data by adding position information of a display having a pointer, the position information being included in each of the plurality of learning image data;
generating a learning model by performing machine learning using the generated plurality of learning data;
The process is executed by a computer,
In the process of extracting the partial image data,
Obtaining a value output from the learning model in response to input of the inspection object image data;
extracting the partial image data corresponding to the acquired value from the inspection object image data;
An indication value reading program comprising: In claim 1 ,
In the process of identifying the first orientation,
if the length is not within the predetermined range, the process of extracting the partial image data and the process of obtaining the value are performed again before the process of identifying the orientation of the pointer from the value is performed.
An indication value reading program comprising: In claim 1,
In the process of identifying the value indicated by the pointer, the value indicated by the pointer is identified from a minimum value included in a range of values that the pointer can indicate, a maximum value included in a range of values that the pointer can indicate, the first orientation, a second orientation indicated by the pointer when the pointer indicates the minimum value, and a third orientation indicated by the pointer when the pointer indicates the maximum value.
An indication value reading program comprising: In claim 4 ,
In the process of identifying the value indicated by the guideline,
calculating a first value by subtracting an angle corresponding to the second orientation from an angle corresponding to the first orientation;
calculating a second value by subtracting an angle corresponding to the second orientation from an angle corresponding to the third orientation;
calculating a third value by subtracting the minimum value from the maximum value;
calculating a fourth value by dividing the first value by the second value;
calculating a fifth value by multiplying the fourth value and the third value;
calculating a value indicated by the pointer by adding the minimum value to the fifth value;
An indication value reading program comprising: In claim 1, further comprising:
acquiring the inspection object image data of the display unit captured by an imaging device;
performing homography transformation on the acquired inspection object image data;
The process is executed by a computer,
In the process of extracting the partial image data, the partial image data is extracted from the inspection object image data that has been subjected to homography transformation.
An indication value reading program comprising: In claim 6 ,
In the process of performing the homography transformation,
Identifying one or more indicia on the indicator;
performing a homography transformation on the inspection object image data so that the one or more identified landmarks are moved to a pre-specified position;
An indication value reading program comprising: In claim 1, further comprising:
performing the process of extracting the partial image data, the process of identifying the first orientation, and the process of identifying the value indicated by the pointer for a plurality of the inspection object image data;
calculating an average value of the values indicated by the indicator identified in the process of identifying the values indicated by the indicator;
The process is executed by a computer,
In the process of outputting the value indicated by the pointer, the calculated average value is output.
An indication value reading program comprising: In claim 8 , further comprising:
Calculating a standard deviation of the value indicated by the indicator identified in the process of identifying the value indicated by the indicator.
The process is executed by a computer,
In the process of outputting the value indicated by the indicator, when the calculated standard deviation is equal to or greater than a predetermined threshold value, the average value is output.
An indication value reading program comprising: In claim 8 , further comprising:
Calculating a standard deviation of the value indicated by the indicator identified in the process of identifying the value indicated by the indicator.
The process is executed by a computer,
In the process of outputting the value indicated by the pointer, the standard deviation is output in addition to the average value.
An indication value reading program comprising: In claim 8 , further comprising:
Calculating a standard deviation of the value indicated by the indicator identified in the process of identifying the value indicated by the indicator.
The process is executed by a computer,
In the process of outputting the value indicated by the indicator,
When the calculated standard deviation is equal to or greater than a predetermined threshold, in addition to the average value, information indicating that the pointer is vibrating is output.
An indication value reading program comprising: In claim 8 , further comprising:
calculating a difference between a maximum value and a minimum value among the values indicated by the indicator identified in the process of identifying the value indicated by the indicator;
The process is executed by a computer,
In the process of outputting the value indicated by the indicator,
When the calculated difference is equal to or greater than a predetermined threshold, the average value is output.
An indication value reading program comprising: In claim 8 , further comprising:
calculating a difference between a maximum value and a minimum value among the values indicated by the indicator identified in the process of identifying the value indicated by the indicator;
The process is executed by a computer,
In the process of outputting the value indicated by the indicator,
When the calculated difference is equal to or greater than a predetermined threshold, in addition to the average value, information indicating that the pointer is vibrating is output.
An indication value reading program comprising: Extracting partial image data of a display having a pointer from inspection object image data including the display;
Identifying a first direction indicated by the pointer included in the partial image data based on the extracted partial image data;
identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and a range of values that the pointer can indicate on the display device;
performing the process of extracting the partial image data, the process of identifying the first orientation, and the process of identifying the value indicated by the pointer for a plurality of the inspection object image data;
Calculating the average value of the values indicated by the identified guidelines;
Calculate the standard deviation of the values indicated by the identified guidelines;
If the calculated standard deviation is equal to or greater than a predetermined threshold, output the calculated average value.
An indication reading program for causing a computer to execute a process. a data generating unit that generates a plurality of learning data by adding information indicating the orientation of a pointer included in each of a plurality of learning image data for a display device having a pointer;
a model generation unit that generates a learning model by performing machine learning using the generated plurality of learning data;
a partial image extraction unit that extracts partial image data of a display having a pointer from inspection object image data including the display;
a direction specification unit that obtains a value output from the learning model in response to input of the extracted partial image data, calculates a length of the pointer included in the partial image data from the obtained value, and specifies a first direction indicated by the pointer included in the partial image data from the obtained value if the calculated length is within a predetermined range ;
a value specifying unit that specifies a value indicated by the pointer included in the partial image data based on the specified first orientation and a range of values that the pointer can indicate on the display device;
A value output unit that outputs a value indicated by the specified pointer.
An indication reading device characterized by the above. a partial image extracting unit for extracting partial image data of a display having a pointer from each of a plurality of pieces of inspection object image data including the display;
a direction specifying unit that specifies a first direction indicated by the pointer included in the partial image data based on the extracted partial image data for each of the plurality of inspection object image data;
a value specifying unit that specifies, for each of a plurality of pieces of inspection object image data, a value indicated by the pointer included in the partial image data based on the specified first orientation and a range of values that the pointer can indicate on the display;
and a value output unit that calculates an average value of the values indicated by the identified indicator, calculates a standard deviation of the values indicated by the identified indicator, and outputs the calculated average value if the calculated standard deviation is equal to or greater than a predetermined threshold value.
An indication value reading device. generating a plurality of learning data by adding information indicating the orientation of a pointer included in each of a plurality of learning image data for a display device having a pointer;
generating a learning model by performing machine learning using the generated plurality of learning data;
Extracting partial image data of a display having a pointer from inspection object image data including the display;
Obtaining a value output from the learning model in response to inputting the extracted partial image data;
Calculating the length of the pointer included in the partial image data from the acquired value;
If the calculated length is within a predetermined range, a first direction indicated by the pointer included in the partial image data is identified from the acquired value ;
identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and a range of values that the pointer can indicate on the display included in the partial image data;
outputting a value indicated by the identified indicator;
A method for reading an indication value, the processing of which is executed by a computer. Extracting partial image data of a display having a pointer from inspection object image data including the display;
Identifying a first direction indicated by the pointer included in the partial image data based on the extracted partial image data;
identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and a range of values that the pointer can indicate on the display device;
performing the process of extracting the partial image data, the process of identifying the first orientation, and the process of identifying the value indicated by the pointer for a plurality of the inspection object image data;
Calculating the average value of the values indicated by the identified guidelines;
Calculate the standard deviation of the values indicated by the identified guidelines;
If the calculated standard deviation is equal to or greater than a predetermined threshold, output the calculated average value.
A method for reading an indication value, the processing of which is executed by a computer.

Images