JP2024126526A - State determination device and state determination method - Google Patents

Abstract

To provide a state determination device and method capable of detecting abnormality occurring in an object and identifying the abnormality and an erroneous detection factor such as a shadow by using a photographic image of the object.SOLUTION: A state determination device is used, including a camera mounted on a mobile body to acquire a photographic image in a travel space, an object image extraction part for extracting an object image of a small area including an object of state determination in the photographic image, and a determination part for performing image processing on the object image and outputting a state determination result showing the presence/absence of abnormality of the object and an erroneous detection factor determination result showing the presence/absence of an erroneous detection factor.SELECTED DRAWING: Figure 1

Description

The present invention relates to a state determination device and a state determination method.

JP 2021-124751 (Patent Document 1) is a background technology of this technical field. This publication states that "The imaging unit is provided on the railway vehicle and captures images of railway equipment installed along the route of the railway vehicle. ... The video captured by the imaging unit is associated with the vehicle position and the video is encoded. ... The motion vector between the first image included in the decoded first video and the second image included in the decoded second video is calculated, and an abnormality in the railway equipment is diagnosed based on the motion vector."

JP 2021-124751 A

The main applications of the status determination device and status determination method include, for example, wayside equipment inspection systems that aim to automatically determine the status (normal/abnormal) of wayside equipment in order to realize railway operation support and unmanned operation. For wayside equipment inspection systems, a useful method is to mount a camera on a railway vehicle and use an analysis device inside or outside the vehicle to determine the status from the captured images.

The above-mentioned Patent Document 1 describes that "the imaging unit is provided on the railway vehicle and captures railway equipment installed along the railway vehicle's travel route. The video captured by the imaging unit is associated with the vehicle position and the video is encoded. A motion vector between a first image included in the decoded first video and a second image included in the decoded second video is calculated, and an abnormality in the railway equipment is diagnosed based on the motion vector." However, it is difficult to detect an abnormality that does not appear in the motion vector, such as dirt on the surface of the equipment, and there is room for improvement. In particular, how to distinguish between a shadow or the like generated on the surface of the equipment and an abnormality is one of the important points. For example, if it is not possible to distinguish between a shadow generated on the surface of the equipment and dirt, and the shadow is determined to be an abnormality, unnecessary maintenance work will occur. In addition, if unnecessary alerts are frequently generated, it will cause a decrease in reliability of the alert.
An object of the present invention is to provide a state determination device and a state determination method that can detect abnormalities occurring in an object using a captured image of the object and distinguish between abnormalities and causes of erroneous detection such as shadows.

In order to achieve the above object, one representative status determination device of the present invention is characterized by comprising: a camera mounted on a moving body and acquiring captured images of the driving space; a target image extraction unit that extracts a target image of a small area including an object for status determination from the captured images; and a determination unit that performs image processing on the target image and outputs a status determination result indicating the presence or absence of an abnormality in the object and a false detection factor determination result indicating the presence or absence of a false detection factor.
Furthermore, one representative status determination method of the present invention is characterized in that it includes a video frame acquisition step in which a status determination device acquires a video frame of a traveling space of a moving body, a target image extraction step in which a target image of a small area including an object for status determination in the video frame is extracted, and a determination step in which image processing is performed on the target image, and an output of a status determination result indicating the presence or absence of an abnormality in the object and an erroneous detection factor determination result indicating the presence or absence of an erroneous detection factor.

According to the present invention, it is possible to provide a condition determination device and method that can use a captured image of an object to detect abnormalities occurring in the object and distinguish between abnormalities and causes of erroneous detection, such as shadows.

1 is a diagram illustrating an overall configuration of a railway line facility inspection system according to a first embodiment of the present invention. FIG. FIG. 2 is a schematic diagram showing an example of a captured image of the front of a railway vehicle. FIG. 2 is a schematic diagram showing an example of a target image. FIG. 2 is a schematic diagram showing an example of a captured image of the front of a railway vehicle. FIG. 2 is a schematic diagram showing an example of a target image. 2 is a diagram illustrating an internal configuration of a determination unit in the trackside facility inspection system according to the first embodiment of the present invention. FIG. 4 is a table showing an example of an internal operation of an integrated determination unit in the lineside facility inspection system according to the first embodiment of the present invention. 4 is a flowchart illustrating an example of internal processing in the trackside facility inspection system according to the first embodiment of the present invention. 5 is a flowchart illustrating an example of internal processing of a determination step in the lineside facility inspection system according to the first embodiment of the present invention. FIG. 11 is a diagram illustrating an overall configuration of a railway line facility inspection system according to a second embodiment of the present invention. 13 is a diagram illustrating an internal configuration of a determination unit in a wayside facility inspection system according to a second embodiment of the present invention. FIG. 13 is a schematic diagram showing an example of an anomaly score history when the state of an object is normal. FIG. 13 is a schematic diagram showing an example of an abnormality score history when the state of an object is abnormal. FIG. FIG. 13 is a diagram illustrating the overall configuration of a trackside facility inspection system according to a third embodiment of the present invention. FIG. 13 is a diagram illustrating the overall configuration of a trackside facility inspection system according to a fourth embodiment of the present invention. FIG. 2 is a schematic diagram showing an example of a captured image of the front of a railway vehicle. FIG. 2 is a schematic diagram showing an example of a captured image of the front of a railway vehicle.

Below, an example embodiment of the present invention will be described with reference to the drawings.

In this embodiment, we will explain a wayside equipment inspection system, which is the main application of the status determination device and method of the present invention. The wayside equipment inspection system uses captured video of the area in front of the railway vehicle to determine whether or not there is an abnormality in wayside equipment, and uses the information to help with equipment management and repair planning.

Figure 1 is a diagram explaining the overall configuration of a wayside facility inspection system according to one embodiment of the present invention. The purpose of this embodiment is to provide a wayside facility inspection system that uses photographed images of the object to detect abnormalities occurring in the object and is capable of distinguishing between abnormalities and causes of false detection such as shadows.

[System configuration]
The structure and operation of a wayside facility inspection system according to an embodiment of the present invention will be described below. The wayside facility inspection system according to the present embodiment is comprised of a camera 1, a target image extraction unit 2, and a determination unit 3, as shown in FIG.
The operation of the wayside facility inspection system will be described with reference to Fig. 1. The camera 1 is mounted on a moving railcar and captures images of the travel space, mainly in front of the railcar. The target image extraction unit 2 extracts a target image of a small area including the target from the captured image in order to focus on the target for inspection. The determination unit 3 performs image processing on the target image and outputs a state determination result indicating the presence or absence of an abnormality in the target and a false detection factor determination result indicating the presence or absence of a false detection factor.

The target objects of the wayside facility inspection system are, for example, signs, traffic signals, ground coils, etc. The abnormalities of the detection target are, for example, dirt, defects, rust, peeling paint, obscuration by vegetation, etc.
The mounting position of the camera 1 may be, for example, inside or outside the vehicle near the driver's cab of the railroad vehicle, or even on the roof, so that the entire front of the railroad vehicle can be photographed, and a wide range of facilities can be taken as the target object. Alternatively, mounting the camera on the side of the vehicle makes it possible to take detailed photographs of the target object from a position closer to the target object. Alternatively, mounting a camera that photographs not only the front of the railroad vehicle but also the rear can make it possible to target facilities along the opposing track.

The camera 1 can inspect objects at a long distance by using a telephoto lens or a high-resolution image sensor. In particular, when the system is operated while a railway vehicle is in operation and it is desired to detect abnormalities that may interfere with operation at an early stage, it is desirable to be able to inspect objects from a long distance.

For example, if the object has specific characters such as a speed sign, the target image extraction unit 2 may extract the target image by recognizing the characters using Optical Character Recognition (OCR). Alternatively, the target image may be extracted by using image processing techniques such as Deep Neural Network (DNN) to output information on the position and size of the object from the captured image, or by detecting pixels in which the object exists using semantic segmentation.

Causes of false detection include, for example, if the image is taken during the day, shadows caused by sunlight and surrounding structures, sunlight reflection, rain, snow, etc., and if the image is taken at night, shadows caused by headlights of cars traveling along the line and surrounding structures, reflections of the headlights, etc. In the following, shadows caused by sunlight and surrounding structures are described as causes of false detection, but the present invention can be applied to other causes of false detection using a similar configuration and method.

Figure 2A is an example of a captured image of the front of a railway vehicle. A sign F202 is captured in the captured image F201 acquired by the camera 1. A stain F203 has appeared on the sign F202. The target image extraction unit 2 detects the sign F202 from the captured image F201, and outputs the target image F204 shown in Figure 2B. The determination unit 3 detects the stain F203 from the target image F204, and outputs a state determination result indicating that an abnormality exists.

Figure 2C is an example of a captured image of the front of a railway vehicle. A shadow F205 is cast on sign F202 by nearby power lines. The presence of the shadow F205 is not abnormal for the object, but it can be a cause of false detection of an abnormality. Therefore, even if the shadow F205 can be judged to be abnormal from the object image F204 shown in Figure 2D, the judgment unit 3 recognizes the presence of the shadow through separate processing, and makes an integrated judgment from these results to output the status judgment result as normal.

3 is a diagram showing an example of the internal configuration of the determination unit 3. The determination unit 3 is composed of an abnormality score calculation unit 301, a shadow score calculation unit 302, and an integrated determination unit 303.
The abnormality score calculation unit 301 calculates an abnormality score indicating the degree of deviation from the normal state of the object by image processing. The shadow score calculation unit 302 calculates a shadow score indicating the degree of a shadow generated on the object by image processing. The integrated judgment unit 303 uses the abnormality score and the shadow score, and outputs a state judgment result and a shadow judgment result after taking into account an increase in the abnormality score due to the shadow.

If the anomaly score calculation unit 301 is not provided, for example, if the reliability of the extraction of the object is low, it may be assumed that an abnormality has occurred, and the inverse of the reliability of the DNN in the target image extraction unit 2 may be used as the anomaly score.
When the anomaly score calculation unit 301 is provided, since it is generally difficult to collect a large amount of abnormal images, the image processing used by the anomaly score calculation unit 301 may use only normal images for learning and use a DNN that can output the deviation from the normal state as an anomaly score. For example, an Autoencoder (AE) or Generative Adversarial Network (GAN) that has learned to reconstruct normal images may be used to output an anomaly score based on a reconstruction error. However, in these methods, there is a possibility that a shadow that occurs when conditions such as the installation location of the object, surrounding structures, and shooting time match may be erroneously detected as an anomaly.
The shadow score calculation unit 302 may use, for example, a DNN that performs binary discrimination of whether an input image has a shadow or not, and may use the confidence of the likelihood of a shadow as the shadow score.

Figure 4 is a table showing an example of the internal operation of the integrated judgment unit 303. The abnormality score is classified into two categories based on whether it is larger or smaller than a predetermined abnormality score threshold. Similarly, the shadow score is classified into two categories based on whether it is larger or smaller than a predetermined shadow score threshold. Therefore, the output of the abnormality score and the shadow score are classified into 2 x 2 = 4 patterns. Of these, in a pattern in which both the abnormality score and the shadow score are large, it is determined that the increase in the abnormality score is caused by the shadow, and the state of the object is determined to be normal. On the other hand, in a pattern in which the abnormality score is large but the shadow score is small, it is determined that the increase in the abnormality score is caused by the true abnormality factor, not the shadow, and the state of the object is determined to be abnormal. In a pattern in which the abnormality score is small, the state of the object is determined to be normal. The integrated judgment unit 303 outputs these state patterns as state judgment results. In addition, it outputs the classification result of the shadow score as a shadow judgment result.

The internal operation of the integrated judgment unit 303 shown in FIG. 4 may be adjusted as appropriate depending on whether the system is used for driving assistance or maintenance assistance. For example, in driving assistance, in order to reduce disruption to normal operation, some non-detection is tolerated while reducing false positives is prioritized. On the other hand, in maintenance assistance, since it is undesirable to overlook anomalies that need to be repaired, some false positives are tolerated while reducing non-detection is prioritized. In these cases, the anomaly score threshold is set lower in maintenance assistance than in driving assistance. Furthermore, it is possible to adjust the above-mentioned tendency by setting the shadow score threshold higher or disabling it.

Depending on the conditions of the terrain and surrounding structures, such as urban areas, plains, and mountainous areas, the way shadows are generated may differ. Specifically, it is possible that shadows are generated almost all the time, and conversely, that shadows are hardly generated at all. The shadow score threshold may be adjusted appropriately to suit such conditions.
In one example of the internal operation of the integrated determination unit 303 shown in FIG. 4, the outputs of the anomaly scores and shadow scores are classified into 2×2=4 patterns, but the number of patterns is not limited to this, and classification into, for example, 3×3=9 patterns may be used.
4, an example in which the state determination result and the shadow determination result are both output in the form of a binary value by the integrated determination unit 303 has been described, but these may be output in the form of a continuous value. For example, the abnormality score may be corrected by multiplying the reciprocal of the shadow score so that the abnormality score is higher as the deviation from the normal state increases and the degree of the shadow is smaller, and this may be output as the state determination result. Also, a continuous value of the shadow score may be output as the shadow determination result.
For factors other than shadows (such as sunlight reflection, rain, and snow), a DNN may be provided for each factor. In this case, if any factor is applicable, the system will be determined to be normal even if the abnormality score is high. A DNN that integrates multiple factors may also be provided.

[Operation sequence]
An example of the process flow of this embodiment is shown in Figure 5. After startup, the process moves to a processing loop, and in a system operation continuation confirmation step S1, the process loop monitors whether or not there is an instruction to end operation of the lineside facility inspection system of this embodiment. If operation is to be continued, in a video frame acquisition step S2, a video frame captured by the camera is acquired. Then, in a target image extraction step S3, a target image of a small area including the target is extracted from the video frame in order to focus on the target for inspection. Then, in a determination step S4, image processing is performed on the target image, and a status determination result indicating the presence or absence of an abnormality in the target and a false detection factor determination result indicating the presence or absence of a false detection factor are output.

Figure 6 is an example of a process flow showing the internal processing of the judgment step S4. In the abnormality score calculation step S401, an abnormality score indicating the degree of deviation from the normal state of the object is calculated by image processing. Then, in the shadow score calculation step S402, a shadow score indicating the degree of a shadow generated on the object is calculated by image processing. Then, in the shadow score comparison step S403, the shadow score is compared with a predetermined shadow score threshold, and if the shadow score exceeds the shadow score threshold, the value of the shadow judgment result is set to "shadow is present" in the shadow judgment result setting step S404. On the other hand, if the shadow score is equal to or less than the shadow score threshold, the value of the shadow judgment result is set to "shadow is absent" in the shadow judgment result setting step S405. Then, in the abnormality score comparison step S406, the abnormality score is compared with a predetermined abnormality score threshold, and if the abnormality score exceeds the abnormality score threshold, the shadow judgment result is confirmed in the shadow judgment result confirmation step S407. If the shadow determination result is "no shadow," it is determined that the increase in the abnormality score is due to a true abnormality factor and not a shadow, and the value of the status determination result is set to abnormal in status determination result (abnormal) setting step S408. If the abnormality score is equal to or less than the abnormality score threshold in abnormality score comparison step S406, the state of the object is deemed normal, and the value of the status determination result is set to normal in status determination result (normal) setting step S409. Alternatively, if the shadow determination result is "there is a shadow" in shadow determination result confirmation step S407, it is determined that the increase in the abnormality score is due to a shadow, the state of the object is deemed normal, and similarly, the value of the status determination result is set to normal in status determination result (normal) setting step S409.

As described above, the present invention makes it possible to realize a railway line equipment inspection system that uses photographed images of an object to detect abnormalities occurring in the object and distinguish between abnormalities and causes of false detection such as shadows.

Figure 7 is a diagram explaining the overall configuration of a wayside facility inspection system according to one embodiment of the present invention. This embodiment aims to provide a wayside facility inspection system that can distinguish between anomalies and causes of false detection, such as shadows, by accumulating anomaly scores calculated from images of an object taken from the same viewpoint and analyzing the accumulated anomaly score history.

[System configuration]
The configuration and operation of a wayside facility inspection system according to an embodiment of the present invention will be described. As shown in Fig. 7, the wayside facility inspection system of this embodiment is composed of a location information acquisition unit 4 and an abnormality score recording unit 5 in addition to the configuration shown in Fig. 1. The location information acquisition unit 4 is mounted on each mobile object. The abnormality score recording unit 5 is preferably installed in a server or the like capable of communicating with a plurality of mobile objects.

The operation of the lineside facility inspection system will be described with reference to FIG. 7. The description of the operation overlapping with the first embodiment will be omitted. The position information acquisition unit 4 acquires the current position information of the moving object using the Global Positioning System (GPS) or the like. When the moving object approaches the vicinity of the object and the camera 1, the object image extraction unit 2, and the judgment unit 3 operate to calculate the abnormality score of the object, the abnormality score recording unit 5 links the abnormality score and the position information and records them, and outputs the abnormality score history for a predetermined first period in the past at the same position going back from the current time. In other words, the abnormality score history is a history of abnormality scores obtained from images taken from the same viewpoint of the object. The judgment unit 3 analyzes the abnormality score history and outputs a state judgment result and a false detection cause judgment result.

Figure 8 is a diagram showing an example of the internal configuration of the determination unit 3. The determination unit 3 is composed of an anomaly score calculation unit 301 and an anomaly score history analysis unit 304. Unlike the determination unit 3 in Figure 3, the determination unit 3 in Figure 8 does not have a shadow score calculation unit 302 and an integrated determination unit 303.

The anomaly score history analysis unit 304 analyzes the anomaly score history output by the anomaly score recording unit 5. Specifically, if the anomaly score history has an offset component over a predetermined second period, it outputs a state determination result indicating an abnormality, and if the difference obtained by subtracting the offset component from the current anomaly score exceeds a predetermined threshold, it outputs an erroneous detection factor determination result indicating the presence of an erroneous detection factor.

Figure 9A is an example of an anomaly score history when the state of the object is normal. For example, if the anomaly score increases due to a shadow cast on the object, periodic fluctuations F901 occur in the anomaly score history due to the periodicity of solar motion. The anomaly score during time periods when no shadows are cast takes on small values. Figure 9B is an example of an anomaly score history when the state of the object is abnormal. In addition to periodic fluctuations F901 caused by the shadow, an offset component F902 caused by the true anomaly occurs. Therefore, it is possible to determine the state of the object based on the presence or absence of offset component F902.

In other words, although the camera 1 used in this embodiment is a moving camera, it is possible to distinguish between false positives such as shadows and true abnormalities and determine the state of the object, as if the object were being observed at a fixed point by a fixed camera.

In the configuration of this embodiment, since the state is judged using the anomaly score history for a specified period, it is difficult to detect an abnormality immediately when it occurs. On the other hand, for example, if the anomaly score can be predicted from the periodicity of solar motion, etc., it is possible to detect an abnormality immediately when it occurs by predicting the anomaly score caused by a false detection factor such as a shadow that may occur at the current time and calculating the difference between the anomaly score at the current time and the predicted anomaly score. The prediction can be made with higher accuracy by correcting the predicted value according to the weather, season, etc.

In the configuration of this embodiment, since the condition is judged using the anomaly score history for a specified period, it is desirable to accumulate anomaly scores evenly and frequently on the time axis. To achieve this, it is useful to configure the anomaly score recording unit 5 to be shared between multiple mobile objects, and to integrate and use the anomaly scores obtained from multiple mobile objects.

As described above, the present invention makes it possible to realize a line equipment inspection system that can distinguish between abnormalities and causes of false detection, such as shadows, by accumulating anomaly scores determined from images of an object taken from the same viewpoint and analyzing the accumulated anomaly score history.
Here, the case where a shadow is identified as a cause of false positive detection is exemplified, but the present invention can also be applied to the case where rain or snow is identified as a cause of false positive detection. An increase in the anomaly score caused by rain or snow will disappear when the rain or snow stops over time. Therefore, even if a temporary increase in the anomaly score disappears over time, if there is an offset, it is possible to determine that this offset indicates an anomaly.

Figure 10 is a diagram illustrating the overall configuration of a wayside facility inspection system according to one embodiment of the present invention. This embodiment aims to provide a wayside facility inspection system capable of automatically generating high-quality learning data for image processing used by the shadow score calculation unit in embodiment 1.

[System configuration]
The structure and operation of a wayside facility inspection system according to an embodiment of the present invention will be described below. As shown in Fig. 10, the wayside facility inspection system according to the present embodiment is composed of a learning data generating unit 6 and a learning data recording unit 7 in addition to the structure shown in Fig. 1.

The operation of the lineside facility inspection system will be described with reference to FIG. 10. Descriptions of operations that overlap with those in Examples 1 and 2 will be omitted. The learning data generation unit 6 automatically performs teacher assignment for the target image output by the target image extraction unit 2 based on the erroneous detection cause judgment result output by the judgment unit 3, and the shadow judgment result if the erroneous detection cause is a shadow. Specifically, when the shadow score calculation unit in Example 1 uses a DNN that performs binary discrimination of whether an input image has a shadow or not, for example, teacher data of "1" is assigned to a target image with a shadow and "0" to a target image without a shadow. The learning data assigned by the learning data generation unit 6 is recorded in the learning data recording unit 7.

When training a DNN that performs a binary discrimination between whether a shadow is present or not, it is desirable to make the conditions other than shadows as consistent as possible in both groups of images with and without shadows in order to minimize the impact of factors other than shadows on the discrimination results. With the configuration of this embodiment, it is possible to collect a large number of images of the target object taken from the same viewpoint at different times, i.e., images that differ only in the presence or absence of factors that cause false detection, such as shadows. Therefore, with the configuration of this embodiment, it is possible to automatically generate high-quality training data in DNN training.

As described above, the present invention makes it possible to realize a lineside facility inspection system that can automatically generate high-quality learning data for the image processing used by the shadow score calculation unit in Example 1.

Figure 11 is a diagram illustrating the overall configuration of a wayside facility inspection system according to one embodiment of the present invention. The purpose of this embodiment is to provide a wayside facility inspection system that can determine the installation state of an object.

[System configuration]
The structure and operation of a wayside facility inspection system according to an embodiment of the present invention will be described below. As shown in Fig. 11, the wayside facility inspection system according to the present embodiment is composed of a location information acquisition unit 4 and an installation state determination unit 8 in addition to the structure shown in Fig. 1.

The operation of the lineside facility inspection system will be described with reference to FIG. 11. The description of the operation overlapping with the first to third embodiments will be omitted. When the target image extraction unit 2 extracts a target image of a small area including an object from the photographed image, it outputs extraction information together with the target image. The extraction information includes information on whether the target image was extracted successfully and, if extracted, coordinate information of the extracted image in the photographed image. The installation state determination unit 8 has information on the installation state of the object, specifically, information on the installation position and height of the object in the real world, stored as a database. The installation state determination unit 8 can also refer to data indicating the dimensions of the moving body and the mounting position of the camera 1. The installation state determination unit 8 refers to the database based on the position information output by the position information acquisition unit 4, and calculates the appearance area of the object in the photographed image. When the moving body is approaching the position where the object is supposed to appear, if the object is extracted from the appearance area of the photographed image, the installation state determination result is considered to be normal. Conversely, if the moving body is approaching a position where the target object is supposed to appear but the target image cannot be extracted from the captured image, or if the target image is extracted but its coordinate information is far from the appearance area, the installation status determination result is output as an abnormality.

When comparing the extracted coordinate information with the appearance area, for example, the center coordinates of each may be calculated and the Euclidean distance between the two may be used. Alternatively, the degree of overlap between the two may be evaluated using Intersection over Union (IoU).

Examples of the installation state determination result being abnormal include obscuration by vegetation, large deformation, tilt, and loss.
12A is an example of a captured image of the front of a railway vehicle. In a captured image F201 acquired by the camera 1, a tree F1201 near a sign F202 is overgrown and covers the sign F202, making it impossible to detect the object within the object appearance area F1202. In this case, the installation state determination unit 8 outputs the installation state determination result of the object as an abnormality.

Figure 12B is an example of a captured image of the front of a railway vehicle. In the captured image F201 acquired by camera 1, the sign F202 is tilted, and the object cannot be detected within the object appearance area F1202. In this case, too, the installation state determination unit 8 outputs the object installation state determination result as an abnormality.

As described above, the present invention makes it possible to realize a railway line facility inspection system capable of determining the installation state of an object.
Specifically, the disclosed condition determination device includes a camera 1 mounted on a moving body and configured to capture an image of the driving space, a target image extraction unit 2 configured to extract a target image of a small area including an object for condition determination from the captured image, and a determination unit 3 configured to perform image processing on the target image and output a condition determination result indicating the presence or absence of an abnormality in the object and a false detection factor determination result indicating the presence or absence of a false detection factor.
With this configuration, the state determination device can use captured images of the object to detect abnormalities occurring in the object and distinguish between abnormalities and causes of erroneous detection, such as shadows.

In addition, the judgment unit 3 judges one or more factors of erroneous detection, and if a result of the erroneous detection factor judgment indicates that any of the factors of erroneous detection is present, it suppresses the output of a status judgment result indicating that an abnormality exists in the target object.
This operation prevents a result indicating an abnormality from being output based on the cause of a false detection, thereby preventing unnecessary maintenance work from occurring and avoiding a decrease in reliability of alerts.

Furthermore, when the cause of erroneous detection is a shadow, the determination unit 3 may be configured to include an abnormality score calculation unit 301 which receives the target image and calculates, by image processing, an abnormality score indicating the degree of deviation from the normal state of the target, a shadow score calculation unit 302 which receives the target image and calculates, by image processing, a shadow score indicating the degree of shadow cast on the target, and an integrated determination unit 303 which outputs a shadow determination result indicating the presence of a shadow when the shadow score is greater than a predetermined shadow score threshold, and outputs a state determination result indicating an abnormality when the abnormality score is greater than the predetermined abnormality score threshold and the shadow determination result does not indicate the presence of a shadow.
This configuration makes it possible to achieve highly accurate state determination in real time for a single moving object.

In addition, the device may further include a location information acquisition unit 4 that acquires current location information of the moving object when the cause of the erroneous detection is a shadow, and the determination unit 3 associates the location information with the abnormality score output by the abnormality score calculation unit and records them in the abnormality score recording unit 5, and acquires an abnormality score history for the object for a predetermined first period in the past from the abnormality score recording unit 5 based on the location information, and the determination unit 3 may further include an abnormality score history analysis unit 304 that outputs a state determination result that is abnormal when the abnormality score history has an offset component over a predetermined second period, and outputs a shadow determination result that a shadow is present when the difference obtained by subtracting the offset component from the current abnormality score exceeds a predetermined threshold value.
In this configuration, the abnormality scores of a large number of moving objects are statistically utilized to easily determine the state.

Furthermore, for the purpose of learning image processing for calculating a shadow score, the configuration may include a learning data generation unit 6 that uses the shadow determination result to add teacher information to the target image, and a learning data recording unit 7 that records the learning data.
In this configuration, learning is performed using the judgment results, and the accuracy of state judgment can be improved.

The device further includes a position information acquisition unit 4 that acquires current position information of the moving body, and an installation state determination unit 8 that determines the installation state of the object. When the target image extraction unit 2 extracts a target image of a small area including the object from the captured image, it outputs extraction information including information on whether the target image was extracted successfully and, if extracted, the extracted coordinate information in the captured image together with the target image. The installation state determination unit 8 has a database of information regarding the installation state including the installation position and/or height of the object, calculates the appearance area of the object in the captured image based on the position information output by the position information acquisition unit 4 and the database, compares the appearance area with the extraction information, and outputs an installation state determination result indicating normal when the object is extracted from within the appearance area of the captured image, and abnormal when the target image cannot be extracted from the captured image even though the object should appear, or the extracted coordinate information is away from the appearance area.
This configuration also allows for the determination of positional anomalies of the object, which can be done independently of the determination of the object's appearance, such as stains or shadows.

An example of the moving body is a railway vehicle, and an example of the object of state determination is a sign or a traffic light.
If the camera 1 is installed on a railway vehicle, traveling on the rails contributes to identifying the position of the camera 1. Therefore, in the second to fourth embodiments, images from the same viewpoint can be easily collected. Also, in the fourth embodiment, the "position where the object is supposed to appear" can be easily and highly accurately obtained. For objects such as signs and traffic lights, where an abnormality in appearance directly impairs their function, it is particularly important to determine their condition.

In the above-described embodiment, the present invention has been mainly applied to a wayside facility inspection system for automatically determining the state (normal/abnormal) of wayside facilities to realize railway operation assistance and unmanned operation, but it may also be applied to other purposes. For example, it may be applied to a driving assistance system or a maintenance assistance system for automobiles traveling on highways, aircraft traveling on runways, drones flying along power lines, etc.

The present invention is not limited to the above-described embodiments, but includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

Furthermore, each of the above configurations may be configured in part or in whole as hardware, or may be configured to be realized by executing a program on a processor. Furthermore, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all of the control lines and information lines in the product. In reality, it can be considered that almost all of the configurations are interconnected.

1. Camera 2. Target image extraction unit 3. Determination unit 4. Location information acquisition unit 5. Abnormality score recording unit 6. Learning data generation unit 7. Learning data recording unit 8. Installation state determination unit 301. Abnormality score calculation unit 302. Shadow score calculation unit 303. Integrated determination unit 304. Abnormality score history analysis unit F201. Captured image F202. Sign F203. Stains F204. Target image F205. Shadow F901. Periodic fluctuation F902. Offset component F1201. Tree F12 S402: Object appearance area S1: System operation continuation confirmation step S2: Video frame acquisition step S3: Target image extraction step S4: Judgment step S401: Abnormality score calculation step S402: Shadow score calculation step S403: Shadow score comparison step S404: Shadow judgment result (shadow present) setting step S405: Shadow judgment result (no shadow) setting step S406: Abnormality score comparison step S407: Shadow judgment result confirmation step S408: Status judgment result (abnormal) setting step S409: Status judgment result (normal) setting step

Claims (9)

A camera mounted on the vehicle for capturing images of the vehicle's travel space;
a target image extraction unit that extracts a target image of a small region including a target object for state determination from the captured image;
a determination unit that performs image processing on the target image and outputs a state determination result indicating the presence or absence of an abnormality in the target object and an erroneous detection factor determination result indicating the presence or absence of an erroneous detection factor;
A state determination device comprising: The state determination device according to claim 1 ,
The judgment unit judges one or more factors of false detection, and when a false detection factor judgment result indicating the presence of any of the factors of false detection is obtained, suppresses output of a status judgment result indicating the presence of an abnormality in the object. The state determination device according to claim 1 ,
The cause of the false detection is a shadow,
The determination unit is
an anomaly score calculation unit to which the target image is input and which calculates an anomaly score indicating a deviation degree of the target object from a normal state by image processing;
a shadow score calculation unit that receives the target image and calculates a shadow score indicating the degree of a shadow generated on the target object by image processing;
an integrated judgment unit that outputs a shadow judgment result indicating the presence of a shadow when the shadow score is greater than a predetermined shadow score threshold, and that outputs a status judgment result indicating an abnormality when the abnormality score is greater than a predetermined abnormality score threshold and the shadow judgment result indicates that a shadow is not present. The state determination device according to claim 1 ,
The cause of the false detection is a shadow,
A location information acquisition unit that acquires current location information of the moving object is further provided.
The determination unit is
causing an abnormality score recording unit to record the position information and the abnormality score output by the abnormality score calculation unit in association with each other, and acquiring an abnormality score history for the object during a predetermined first period in the past from the abnormality score recording unit based on the position information;
The determination unit is
The status determination device further comprises an abnormality score history analysis unit that outputs a status determination result that is abnormal when the abnormality score history has an offset component over a predetermined second period, and outputs a shadow determination result that is an influence when the difference obtained by subtracting the offset component from the current abnormality score exceeds a predetermined threshold. The state determination device according to claim 3 or 4,
a learning data generation unit that uses the shadow determination result to add teacher information to the target image for the purpose of learning image processing for calculating the shadow score;
and a learning data recording unit that records the learning data. The state determination device according to claim 1 ,
a location information acquisition unit that acquires current location information of the moving object;
an installation state determination unit that determines an installation state of an object;
Further comprising:
the target image extraction unit, when extracting a target image of a small region including an object from the photographed image, outputs extraction information including information on whether the target image was extracted successfully and, if extracted, coordinate information of the extracted target image in the photographed image together with the target image;
The installation state determination unit has a database of information regarding the installation state including the installation position and/or height of the object, calculates the appearance area of the object in the captured image based on the position information output by the position information acquisition unit and the database, compares the appearance area with the extraction information, and outputs an installation state determination result indicating that the installation state is normal when the object is extracted from within the appearance area of the captured image, and that the installation state is abnormal when the object image cannot be extracted from the captured image even though the object should appear or the extracted coordinate information is away from the appearance area. The state determination device according to claim 1, characterized in that the moving body is a railway vehicle. The state determination device according to claim 1, characterized in that the object of state determination is a sign or a traffic light. The state determination device is
An image frame acquisition step of acquiring an image frame of a traveling space of a moving object;
a target image extraction step of extracting a target image of a small region including a target object for state determination from the video frame;
a determination step of performing image processing on the target image and outputting a state determination result indicating the presence or absence of an abnormality in the target object and an erroneous detection factor determination result indicating the presence or absence of an erroneous detection factor;
A state determination method comprising:

Images