JP7092615B2 - Shadow detector, shadow detection method, shadow detection program, learning device, learning method, and learning program - Google Patents

Description

The present invention relates to a shadow detection technique for detecting a shadow region in which a shadow is captured in a captured image, and a learning technique for obtaining a trained model that outputs a degree of shadow from an image feature for a local region in the captured image.

When an object such as a person is detected from a captured image in which shade and sun are mixed, the detection accuracy may decrease due to insufficient contrast in the shaded portion of the captured image. Therefore, in order to prevent a decrease in the detection accuracy, measures such as detecting a shadow area in the captured image and correcting the contrast of the shadow area are taken.

For example, the object detection device described in Patent Document 1 analyzes the luminance distribution of a human candidate region in a captured image, and if it is monomodal and similar to the luminance distribution of a background image, the entire human candidate region is used. It is determined that it is a shadow area, or the area having the brightness value of the mountain on the low brightness side in the bimodal brightness distribution is determined as the shadow area, and the contrast correction is applied to the shadow area to prevent the detection accuracy from deteriorating. Was there.

Japanese Unexamined Patent Publication No. 2011-028348

However, in the conventional technique, there is a problem that it may be difficult to determine a shadow region for a photographed image in which a space having a complicated background is photographed.

For example, in a background composed of two types of materials, the luminance distribution may be bimodal due to the difference in the materials. That is, since the luminance distribution is bimodal even if no shadow is generated, the lower luminance region of the two types of materials is erroneously determined as the shadow region.

The present invention has been made to solve the above problems, and even if a photographed image is captured in a space having a complicated background such as the existence of a plurality of background components having different reflection characteristics, the captured image is highly accurate. It is an object of the present invention to provide a shadow detection technique capable of detecting a shadow region.

Further, the present invention learns a learning model that can output with high accuracy the high possibility that a shadow is captured in the image of a local region in the captured image even if the captured image is captured in a space having a complicated background. Another purpose is to provide learning techniques to make them.

(1) The shadow detection device according to the present invention is a device that detects a shadow region in which a shadow is captured in a captured image captured in a predetermined space, and exhibits image features in a local region set in the captured image. It is a trained model that is input and outputs the degree of shadow indicating the high possibility that a shadow is shot in the local area, and is an image feature in the local area set in the learning image taken in the space. A trained model storage means that stores a trained model that has been trained using the shadow degree for learning, which is the shadow degree estimated for the local region, and an image feature of the captured image in the local region. Is provided in the trained model, the shadow degree is compared with a predetermined reference, and a shadow determination means for determining a local region in which the shadow degree exceeds the reference is defined as the shadow region.

(2) In the shadow detection device according to (1) above, the trained model storage means has been trained for each characteristic-similar region in which the reflection characteristics of the background components that can be captured in the captured image are similar. , The trained model can be stored for each characteristic-similar region.

(3) The shadow detection device according to (2) above further includes a background information storage means for storing the characteristic-similar region, and the shadow determination means transfers an image feature of the local region to the local region. The shadow degree can be obtained by inputting to the trained model of the corresponding characteristic-similar region.

(4) In the shadow detection device according to (1) above, the learning is further provided with a background information storage means for storing a characteristic-similar region having similar reflection characteristics of a background component that can be captured in the captured image. In addition to the image features and the degree of shadow for learning about the local region set in the captured image for learning, the completed model storage means also uses attribution information indicating the characteristic-like region to which the local region belongs. The trained model is stored, and the shadow determination means inputs the image features of the local region and the attribution information about the local region into the trained model to obtain the shadow degree. Can be.

(5) In the learning device according to the present invention, an image feature in a local region set in a captured image captured in a predetermined space is input, and a shadow indicating a high possibility that a shadow is captured in the local region. A device for training a learning model that outputs a degree, and is a learning model storage means for storing the learning model, and at least an image feature in a local region set in a captured image for learning in which the space is captured, and the local region. The learning data storage means for storing the learning shadow degree, which is the estimated shadow degree, and the learning model obtained by inputting at least the image features of the local region in the learning photographed image. A learning means for updating the learning model based on an error of the shadow degree with respect to the learning shadow degree of the local region is provided.

(6) In the learning device according to (5) above, the learning model storage means stores the learning model for each characteristic-similar region in which the reflection characteristics of the background components that can be captured in the captured image are similar, and the learning model is stored. The learning means may be configured to perform the learning for the learning model for each characteristic-similar region.

(7) The shadow detection method according to the present invention is a method of detecting a shadow region in which a shadow is captured in a captured image captured in a predetermined space, and exhibits image features in a local region set in the captured image. It is a trained model that is input and outputs the degree of shadow indicating the high possibility that a shadow is shot in the local area, and is an image feature in the local area set in the learning image taken in the space. And input to input the image feature in the local area of the captured image into the trained model using the trained model in which the learning using the learning shadow degree which is the shadow degree estimated for the local region is used. The step includes a shadow determination step of comparing the shadow degree output from the trained model with a predetermined reference and determining a local region in which the shadow degree exceeds the reference as the shadow region.

(8) The shadow detection program according to the present invention is a program for causing a computer to perform a process of detecting a shadow region in which a shadow is captured in a captured image captured in a predetermined space, and the computer is used as described above. It is a trained model that inputs the image features in the local area set in the captured image and outputs the degree of shadow indicating the high possibility that a shadow is captured in the local area. A trained model storage means that stores a trained model that has been trained using the image features in the local region set in the captured image and the training shadow degree, which is the estimated shadow degree for the local region. And, the shadow degree obtained by inputting the image feature in the local region of the captured image into the trained model is compared with a predetermined reference, and the local region where the shadow degree exceeds the reference is referred to as the shadow region. It functions as a shadow determination means for determination.

(9) In the learning method according to the present invention, an image feature in a local region set in a captured image captured in a predetermined space is input, and a shadow representing a high possibility that a shadow is captured in the local region. It is a method of training a learning model that outputs a degree, and is a learning model storage means for storing the learning model, and at least an image feature in a local region set in a captured image for learning in which the space is captured, and the local region. An input step of inputting at least an image feature of the local region in the captured image for learning into the learning model by using a learning data storage means for storing the learning shadow degree which is the estimated shadow degree. And a learning step of performing learning to update the learning model based on the error of the image degree output from the learning model with respect to the learning shadow degree of the local region.

(10) In the learning program according to the present invention, an image feature in a local region set in a captured image captured in a predetermined space is input, and a shadow representing a high possibility that a shadow is captured in the local region. It is a program for causing a computer to perform a process of training a learning model that outputs a degree, and the computer is set as a learning model storage means for storing the learning model, at least, a photographed image for learning in which the space is photographed. A learning data storage means that stores an image feature in a local region and a learning shadow degree that is an estimated shadow degree for the local region, and the local area in at least the learning captured image in the learning model. The image feature of the region is input, and the image feature is made to function as a learning means for learning to update the learning model based on the error of the obtained shadow degree with respect to the learning shadow degree of the local region.

According to the shadow detection technique of the present invention, whether or not each local region in the captured image is a shadow region is determined based on learning using the captured image, so that the captured image in which the background is complicated is captured. Even if there is, the shadow area can be detected with high accuracy.

Further, the learning technique of the present invention learns the degree of shadow indicating the high possibility that a shadow is captured in the image of a local region in the captured image by using the captured image, so that a space having a complicated background is photographed. Even if it is a captured image, it is possible to train a learning model that can output the degree of shadow of the local region with high accuracy.

It is a block diagram which shows the schematic structure of the image monitoring apparatus which concerns on embodiment of this invention. It is a schematic functional block diagram when the image monitoring apparatus which concerns on embodiment of this invention functions as a learning apparatus. It is a schematic diagram which shows the example of the reflection characteristic map. It is a schematic diagram which shows an example of each of the photographed image for learning and the rendered image of a surveillance space. It is a schematic diagram explaining the learning data. It is a schematic functional block diagram when the image monitoring apparatus which concerns on embodiment of this invention functions as a shadow detection apparatus. It is a schematic flow diagram explaining the operation of the image monitoring apparatus which concerns on embodiment of this invention. It is a schematic flow chart of a shadow judgment model learning process. It is a schematic flow chart of a shadow determination process.

Hereinafter, the image monitoring device 1 according to the embodiment of the present invention (hereinafter referred to as the embodiment) will be described with reference to the drawings. The image monitoring device 1 analyzes the presence or absence of a monitoring target such as a person or a suspicious object in a predetermined space (surveillance space) from an image (photographed image) taken. In particular, the image monitoring device 1 is configured to include a shadow detecting device according to the present invention for detecting a shadow region in a captured image, and uses information on the shadow region in image processing for detecting a monitoring target. Further, the image monitoring device 1 is provided with the learning device according to the present invention, and can generate a trained model used in the shadow detection device.

[Configuration of image monitoring device]
FIG. 1 is a block diagram showing a schematic configuration of the image monitoring device 1. The image monitoring device 1 includes a camera 2, a communication unit 3, a storage unit 4, an image processing unit 5, and a notification unit 6.

The camera 2 is a surveillance camera, is connected to the image processing unit 5 via the communication unit 3, captures the surveillance space at predetermined time intervals to generate captured images, and sequentially inputs the captured images to the image processing unit 5. It is a means of photography. For example, the camera 2 is installed on a pole installed in a corner of an event venue, which is a monitoring space, with a predetermined fixed field of view overlooking the monitoring space, and the monitoring space is photographed with a frame period of 1 second to obtain a color image. To generate. The camera 2 may generate a monochrome image instead of the color image.

The communication unit 3 is a communication circuit, one end of which is connected to the image processing unit 5 and the other end of which is connected to the camera 2 and the notification unit 6. The communication unit 3 acquires a captured image from the camera 2 and inputs it to the image processing unit 5, and outputs the analysis result input from the image processing unit 5 to the notification unit 6.

For example, when the camera 2 and the notification unit 6 are installed in the monitoring center in the event venue, and the communication unit 3, the storage unit 4, and the image processing unit 5 are installed in the image analysis center in a remote location, the communication unit 3 and the camera 2 are installed. , And the communication unit 3 and the notification unit 6 can be connected by an internet line, respectively, and the communication unit 3 and the image processing unit 5 can be connected by a bus. In addition, for example, when each unit is installed in the same building, the communication unit 3 and the camera 2 are connected by a coaxial cable or a LAN (Local Area Network), the communication unit 3 and the notification unit 6 are a display cable, and the communication unit 3 and an image. The processing unit 5 is appropriately connected in a form according to the installation location of each unit, such as being connected by a bus.

The storage unit 4 is a memory device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and stores various programs and various data. The storage unit 4 is connected to the image processing unit 5 and inputs / outputs these information to and from the image processing unit 5.

The image processing unit 5 is composed of arithmetic units such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and an MCU (Micro Control Unit). The image processing unit 5 operates as various processing means / control means by reading and executing a program from the storage unit 4, reading various data from the storage unit 4 as necessary, and storing the generated data in the storage unit 4. Let me. Further, the image processing unit 5 generates an analysis result regarding the presence / absence and position of a monitoring target in the monitoring space from the captured image acquired from the camera 2 via the communication unit 3, and outputs the analysis result to the notification unit 6 via the communication unit 3. .. Further, the image processing unit 5 may output a captured image or a corrected image with contrast correction to the notification unit 6.

The notification unit 6 is a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display, and is a monitor by displaying information such as the presence / absence and position of a monitoring target included in the analysis result input from the communication unit 3. Notify to. The notification unit 6 may further include a buzzer, a lamp, or the like to emphasize the alert. The observer visually recognizes the displayed analysis results and images to determine the necessity of countermeasures, and takes measures such as rushing the responders as necessary.

In this embodiment, the image monitoring device 1 in which one camera 2 is provided for the set of the communication unit 3 and the image processing unit 5 is exemplified, but in another embodiment, the communication unit 3 and the image processing unit 3 are illustrated. It is also possible to configure two or more cameras 2 to be connected to the set of units 5. In that case, the communication unit 3 receives the captured images from each camera 2 in time division, and the image processing unit 5 processes the captured images from each camera 2 in time division processing or parallel processing.

[Functions of image monitoring device during learning]
FIG. 2 is a schematic functional block diagram when the image monitoring device 1 functions as the learning device according to the present invention. FIG. 2 shows the functions of the communication unit 3, the storage unit 4, and the image processing unit 5 exclusively. Specifically, the communication unit 3 functions as a captured image acquisition means 30 and the like, and the storage unit 4 is an environment. It functions as a model storage means 40, a camera information storage means 41, a background information storage means 42, a learning data storage means 43, a trained model storage means 44, etc., and the image processing unit 5 functions as a background information generation means 50, a learning data generation means. It functions as a means 51, a learning means 52, and the like.

The captured image acquisition means 30 sequentially acquires captured images from the camera 2, and sequentially outputs the acquired captured images to the background information generation means 50 and the learning data generation means 51. For convenience of explanation, the captured image used for learning is referred to as a learning captured image, and the captured image used for determination is referred to as a determination captured image, if necessary.

The environment model storage means 40 stores a three-dimensional model of a plurality of components (background components) constituting the background of the monitoring space as a three-dimensional background.

The background structure is, for example, a building such as a sidewalk, a road, a building, or a sign, or a non-moving natural object such as a tree if it is outdoors. Preferably, the asphalt portion and the white line portion of the road, and the ground color portion and the character / mark portion of the sign, which have significantly different reflection characteristics from each other, are stored as different background components.

The three-dimensional model of the background component includes data on the three-dimensional coordinate values represented by the position, attitude, and three-dimensional shape of each background component in the XYZ coordinate system imitating the monitoring space, and the reflection characteristic data of each background component. .. Reflective properties are generally composed of factors such as the color, texture, and reflectance of the surface of the composition. The reflectance is expressed by, for example, a bicolor reflection model having the reflectance of the mirror surface reflection component, the reflectance of the diffuse reflection component, and their ratios as parameters.

The 3D model of the background structure can be acquired from the building information of the IFC (Industry Foundation Classes) standard created at the time of building design, 3D CAD data, etc. or the actual measurement data in advance.

Further, the environmental model storage means 40 also stores the lighting model of the monitoring space in advance. The illumination model includes, for one or more light sources that illuminate the surveillance space, lighting characteristics represented by the position of the light source in the XYZ coordinate system imitating the surveillance space, the light distribution of the light source, the color temperature, and the like. The light source is artificial lighting, the sun, or the like.

The camera information storage means 41 stores in advance the camera parameters of the camera 2 in the XYZ coordinate system that imitates the surveillance space. Camera parameters consist of external parameters and internal parameters. The external parameter is the position and orientation of the camera 2 in the XYZ coordinate system. The internal parameters are the focal length of the camera 2, the center coordinates, the distortion coefficient, and the like. The camera parameters are measured by pre-calibration and stored in the camera information storage means 41. By applying this camera parameter to the pinhole camera model, the coordinates of the XYZ coordinate system can be converted into the xy coordinate system representing the shooting surface of the camera 2.

The background information storage means 42 stores a characteristic-similar region, which is a collection of local regions having similar reflection characteristics of the background component in a captured image (background image) in which the background of the surveillance space is captured. Here, the local area is preset. In the present embodiment, each pixel constituting the captured image is set as a local region. Incidentally, the local region may be a region composed of a plurality of pixels, such as a region of 2 × 2 pixels and an region of 3 × 3 pixels. By referring to the characteristic-similar region, it is possible to specify the reflection characteristics of the background component that can be photographed as a background in the local region for any local region in the learning captured image and any local region in the determination captured image. ..

Further, the background information storage means 42 stores a region (estimated shadow region) in which a shadow is estimated to be captured in the learning image. The information in the estimated shadow region is converted into a learning shadow degree indicating the high possibility that a shadow is captured in each local region of the captured image for learning, and is used as teacher data in learning.

Further, the background information storage means 42 stores a background image representing an image of a background component that can be captured in the captured image. The background image is compared with the learning image and the determination image, and is used to extract a region (foreground region) in which an image other than the background component (foreground object) is captured in each captured image.

The background information generation means 50 calculates a characteristic-similar region, an estimated shadow region, and a background image, and stores them in the background information storage means 42. For example, the characteristic-similar region can be calculated by rendering the environment model stored in the environment model storage means 40 using the camera parameters stored in the camera information storage means 41.

Specifically, the background information generation means 50 renders an environment model on the shooting surface of the camera 2 using the camera parameters of the camera 2, so that the background configuration is projected on each pixel of the image formed on the shooting surface. Identify things. In this rendering, any lighting condition may be set regardless of the lighting condition of the light source.

On the other hand, the background information generation means 50 assigns a reflection characteristic ID as an identifier for each reflection characteristic of the background component included in the environment model. At that time, a common reflection characteristic ID may be assigned to the reflection characteristics having completely matching values, or a common reflection characteristic ID may be assigned to the reflection characteristics having similar values to the extent that they can be regarded as the same. The similarity of the reflection characteristics is determined based on the above-mentioned elements and parameters constituting the reflection characteristics. Specifically, if the difference between each element and parameter is equal to or less than a predetermined threshold value, it is determined that the reflection characteristics are similar. If the environment model is originally given an ID for each reflection characteristic, the ID may be used.

Next, the background information generation means 50 creates a reflection characteristic map having pixels corresponding to each pixel of the captured image, and the background configuration projected on the pixel to the pixel value of each pixel of the reflection characteristic map. Set the reflection characteristic ID of the object. In this reflection characteristic map, each region consisting of pixels having the same pixel value is a characteristic-similar region.

Further, the estimated shadow area and the background image can be calculated by rendering the environment model stored in the environment model storage means 40 with the lighting conditions of the captured image and the camera parameters stored in the camera information storage means 41. ..

Specifically, the background information generation means 50 first estimates the lighting conditions of the light source at the time when the captured image is captured and renders the environment model under the lighting conditions. That is, rendering is performed by setting a plurality of lighting conditions, the similarity between the captured image and the rendered image obtained as a result of rendering is calculated, and the rendered image having the maximum similarity is selected as the background image. Next, the background information generation means 50 sets a region in the background image in which the shadow of the background component is formed (the region where the direct light is blocked by the background component) as the estimated shadow region. The background information generation means 50 stores the background image and the estimated shadow region calculated in this way in the background information storage means 42 as described above.

The background information generation means 50 can also use a captured image in a state where the foreground object does not exist in the surveillance space as the background image. Further, the background information generation means 50 may use a region where the luminance value of the background image is less than a predetermined threshold value as an estimated shadow region.

FIG. 3 is a schematic diagram showing an example of a reflection characteristic map. In FIG. 3, the reflection characteristic map 100 is an example corresponding to a photographed image in which a corner where a building exists across a sidewalk on the right side of the roadway is shown. As shown in the figure, the reflection characteristic map 100 can be image data having pixels corresponding to each pixel of the photographed image, and can be represented by the same xy coordinate system as the imaged surface of the camera 2.

Specifically, the reflection characteristic map 100 shows a sidewalk made of cobblestones, an asphalt-paved road, a white line drawn as a road marking on the road, and a building as background components having different reflection characteristics from the corresponding captured image. This is an example when the wall is reflected. Here, for example, the reflection characteristic ID is defined as "1" for the reflection characteristic of the stone pavement of the sidewalk, and similarly, the reflection characteristic ID is defined for the reflection characteristic of the asphalt road surface, the white road marking, and the wall of the building. Are defined as "2", "3", and "4", respectively.

In the reflection characteristic map 100, the reflection characteristic ID is set for each region of the background component having different reflection characteristics in the captured image. In the image 101, the sidewalk region 111 in the reflection characteristic map 100 is shown by diagonal lines, and the value “1” is set as the reflection characteristic ID in the pixels of the shaded portion. Similarly, in the images 102, 103, and 104, the asphalt region 112, the white line region 113, and the wall region 114 of the reflection characteristic map 100 are shown by diagonal lines, and the pixel of the shaded portion has a value of “2” as the reflection characteristic ID, respectively. , "3", "4" are set.

The learning data storage means 43 stores data (learning data) used for learning the shadow determination model. The learning data is information on reflection characteristics, image features, and a degree of shadow for learning for each local region for one or a plurality of captured images for learning. The shadow determination model can be, for example, a model of a tree structure called a random forest.

The information on the reflection characteristic of the local region can be represented by, for example, the reflection characteristic ID assigned to the characteristic-like region to which the local region belongs. Further, the information on the reflection characteristics may be a part or all of the parameters describing the reflection characteristics.

The image features of the local region are, for example, the feature amount of the captured image in the local region, the feature amount of the captured image in the vicinity region of the local region, and the position of the local region, and are represented by a vector having these values as elements. can do.

The feature amount of the captured image in the local region can be, for example, a pixel value in the local region. If the captured image is a color image, it is an RGB value, and if it is a monochrome image, it is a luminance value. When the captured image is a color image, it can be a value obtained by converting the RGB value to another color system, a luminance value obtained by converting the RGB value to a gray scale, or a RGB value or a value converted to another color system. , It is also possible to set it to 2 or more of the brightness values. It is preferable that the image features in the local region include at least the feature amount of the captured image in the local region.

The feature amount of the captured image in the vicinity region of the local region can be, for example, a representative value of the pixel value of the captured image in the neighborhood region such as the average brightness value and the minimum brightness value of the captured image in the neighborhood region. When the captured image is a color image, the average RGB value, the average value of the RGB values converted into another color system, and the like can be used. Alternatively, the pixel value itself of the captured image in the vicinity region may be two or more of the above-mentioned values. By adding the feature amount of the captured image in the vicinity region to the image feature, learning / judgment including the spatial continuity of the shadow becomes possible.

The position of the local region can be the xy coordinate of the local region in the captured image. When the local region consists of two or more pixels, it may be the xy coordinate of a predetermined representative pixel (for example, the upper left pixel). By adding the position of the local region to the image features, it is possible to learn and judge small secular changes such as dark spots on asphalt.

The degree of shadow for learning in a local region is a value indicating the high possibility that a shadow is captured in the local region in the photographed image for learning. Here, in the present embodiment in which the shadow determination model is modeled in a random forest, for example, the likelihood output by the shadow determination model can be a continuous value having a range of [0,1]. The closer the likelihood is to 1, the more likely it is to be a shadow region, and the closer it is to 0, the more likely it is to be a non-shadow region. Correspondingly, the learning shadow degree of the pixels in the estimated shadow area can be set to 1.0, and the learning shadow degree of the pixels outside the estimated shadow area can be set to 0.0.

The learning data generation means 51 generates learning data from the learning photographed image, the reflection characteristic information stored in the background information storage means 42, the estimated shadow region, and the background image, and the generated learning data is used for learning. It is stored in the data storage means 43.

Each time the learning photographed image is input, the learning data generation means 51 generates learning data for each local region of the learning photographed image and stores it in the learning data storage means 43. Specifically, for each local region, the image features of the local region are extracted from the captured image for learning, and if the local region belongs to the estimated shadow region, the degree of shadow for learning is assigned to 1.0. If not, it is set to 0.0, and the reflection characteristic ID, the image feature, and the degree of shadow for learning are associated with each other and added to the learning data storage means 43.

At this time, it is preferable to generate the learning data by excluding the local region where the foreground object is photographed in the learning image. Therefore, the learning data generation means 51 performs background subtraction processing or background correlation processing between the learning photographed image and the background image, and performs learning on a change region that exceeds a predetermined threshold and is different from the background image. Learning data is generated from the learning captured image excluding the change area from the captured image.

The trained model storage means 44 stores a shadow determination model that inputs an image feature of a local region set in the captured image and outputs a shadow degree indicating the high possibility that a shadow is captured in the local region. do. The shadow determination model stored in the trained model storage means 44 has been trained using the image features of the local region in the captured image for training and the learning shadow degree which is the estimated shadow degree for the local region. It is a model. In the present embodiment, each of the trained models for each characteristic-similar region is stored in association with the reflection characteristic ID of the characteristic-similar region.

By the way, the learning is performed iteratively, and the shadow judgment model is updated accordingly. In particular, as will be described later, the image monitoring device 1 performs shadow determination model learning processing every time a captured image is acquired and updates the shadow determination model even in the monitoring operation. That is, the determination model handled by the learning device has an aspect as a trained model and an aspect as a model in the middle of learning to be further learned. Based on this, if the determination model in the learning device is simply expressed as a learning model, the learned model storage means 44 corresponds to the learning model storage means in the learning device of the present invention.

The learning means 52 uses the learning data stored in the learning data storage means 43 to machine-learn the shadow determination model stored in the trained model storage means 44, and trains the trained model generated thereby. It is stored in the model storage means 44. That is, the learning means 52 updates the shadow determination model so that the degree of shadow obtained by inputting the image feature of the local region in the captured image for learning into the shadow determination model is close to the degree of shadow for learning in the local region. By doing so, a trained model is generated.

In the present embodiment, the learning means 52 generates a trained model for each characteristic-similar region. That is, for each characteristic-similar region, the degree of shadow obtained by inputting the image feature of the local region belonging to the characteristic-similar region in the image for learning into the shadow determination model is updated to approach the learning shadow degree of the local region. Is performed on the shadow judgment model to generate a trained model.

The shadow determination model can be applied to two-class problems such as Support Vector Machine (SVM), AdaBoost type discriminator, or discriminant type CNN (Convolutional Neural Network) instead of random forest. Various known models can be used.

An example of learning will be described with reference to FIGS. 4 and 5. FIG. 4 is a schematic diagram of an image of the monitoring space, and shows an example of each of the learning photographed image 200 and the rendered image 210. In the captured image 200 for learning, the background composition is common to the reflection characteristic map shown in FIG. 3, and four regions corresponding to the values “1” to “4” of the reflection characteristic ID as characteristic-similar regions, specifically, There are sidewalks, asphalt surfaces, white road markings, and building walls. Further, in the learning image 200, a person 201 photographed in the sidewalk area is also photographed in addition to the background structure. Further, the shaded area represents the shadow area 202.

The rendered image 210 is calculated by rendering from an environment model or the like by the background information generating means 50 described above. The rendered image 210 is generated corresponding to the captured image 200 for learning, and the background image and the estimated shadow region 212 (hatched portion) are shown. In the rendered image 210, the boundary between the shadow region 202 and the Hinata region in the learning photographed image 200 is shown by a alternate long and short dash line. This illustrates that the estimated shadow region 212 has an estimation error with respect to the true shadow region 202. Regarding the estimation of the estimated shadow region, it is sufficient that the number of pixels correctly estimated as the shadow region and the non-shadow region sufficiently exceeds the number of pixels estimated incorrectly, and an error is allowed.

FIG. 5 is a schematic diagram illustrating learning data. The shaded areas of the images 310 to 345 are areas used for learning data in the learning photographed image 200. Specifically, assuming that the value of the reflection characteristic ID is r, the shaded portion of the image 315 corresponds to the portion of the estimated shadow region 212 (shaded portion) of the rendered image 210 in the learning captured image 200, where r = 1. The shaded portion of the image 310 corresponds to the portion of the learning captured image 200 other than the estimated shadow region 212, that is, the portion where r = 1 in the region estimated to be non-shadow (estimated non-shadow region). Further, the shaded areas of the images 320 and 325 correspond to the estimated non-shadow area and the estimated shadow area 212 of r = 2 in the image 200 for learning, and similarly, the images 330 and 335 have r = 3. Images 340 and 345 show the portion corresponding to the estimated non-shadow region and the estimated shadow region 212 at r = 4.

The data shown in tabular form on the right side of FIG. 5 schematically represent the learning data stored in the learning data storage means 43, and each row of the table is for one local region of one learning photographed image. It is data, and the data includes a reflection characteristic ID, an image feature, and a degree of shadow for learning. The image feature is represented by the symbols F _{* and r} (m), and in the notation, the subscript * of F is a label that distinguishes whether the local region is located in the estimated shadow region or the estimated non-shadow region. If the character of * is S, it means shadow, and if it is N, it means non-shadow. Further, the subscript r of F is a reflection characteristic ID, and m is a number that distinguishes a plurality of local regions that can be set for each combination of the shadow / non-shadow label (*) and the reflection characteristic ID (r).

For example, the first local region among the local regions belonging to the region (shaded portion of the image 310) excluding the foreground region from the region estimated to be the non-shadow region on the sidewalk was extracted from the learning photographed image 200. The image feature F _{N, 1} (1) is stored in association with 1 which is the reflection characteristic ID of the sidewalk and the shadow degree 0.0 for learning. Here, the degree of shadow for learning is set to 0.0 for the estimated non-shadow region, that is, the local region with the label N, and 1.0 for the estimated shadow region, that is, the local region with the label S. Set. By the way, for the local region located in the estimation error of the shadow region sandwiched between the one-point chain line and the estimated shadow region 212 in the rendered image 210, the label is N because it is an estimated non-shadow region, and the degree of shadow for learning is 0. However, since the image features F _{N and r} (m) are extracted from the captured image 200 for learning, they are pixel values and the like corresponding to shadows.

As described above, the learning means 52 generates a trained model for each characteristic-similar region. Therefore, in the example of FIG. 5, for example, for the sidewalk, the learning data associated with the value “1” of the reflection characteristic ID, specifically, the image features F _{N, 1} (1), F _{N, 1} (2). ), ... And _{FS, 1} (1), FS _{, 1} (2), ..., And the learning shadow degree associated with them is used to train the shadow determination model. Similarly, the shadow determination model for the asphalt surface, the white road sign, and the building wall is also learned using the learning data of the corresponding reflection characteristic IDs.

[Function of image monitoring device at the time of judgment]
FIG. 6 shows a schematic function when the image monitoring device 1 functions as a shadow detection device according to the present invention, performs shadow determination on a captured image, and analyzes the presence or absence of a monitoring target in the monitoring space using the result. It is a block diagram. FIG. 6 exclusively shows the functions of the communication unit 3, the storage unit 4, and the image processing unit 5. Specifically, the communication unit 3 functions as a captured image acquisition unit 30, an analysis result output unit 31, and the like. The storage unit 4 functions as an environment model storage means 40, a camera information storage means 41, a background information storage means 42, a trained model storage means 44, and the like, and the image processing unit 5 has a background information generation means 50, a shadow determination means 53, and the like. It functions as a foreground extraction means 54, a foreground information analysis means 55, and the like.

The functions of the captured image acquisition means 30, the environment model storage means 40, the camera information storage means 41, the background information storage means 42, the trained model storage means 44, and the background information generation means 50 in the shadow detection device are the contents described above for the learning device. Since it is the same as the above, the description thereof is omitted here.

The shadow determination means 53 predetermines a shadow degree (determination shadow degree) obtained by inputting an image feature in a local region of a determination photographed image into a trained model stored in the trained model storage means 44. A local region in which the degree of shadow for determination exceeds the reference is determined as a shadow region as compared with the reference, and the determination result is output to the foreground extraction means 54.

In the present embodiment, the characteristic-similar region stored in the background information storage means 42 is used, and the shadow determination means 53 inputs the image feature of the local region into the trained model of the characteristic-similar region corresponding to the local region. And get the degree of shadow.

Therefore, the shadow determination means 53 first extracts an image feature from each local region of the image for determination. The image features to be extracted are feature vectors of the same format as the image features of the training data. Further, the shadow determination means 53 refers to the reflection characteristic map stored in the background information storage means 42 to specify the reflection characteristic ID of the characteristic-similar region to which each local region belongs. The shadow determination means 53 selects a trained model for each characteristic-similar region stored in the trained model storage means 44 that corresponds to the reflection characteristic ID, and sets the image features of each local region as the trained model. To obtain the degree of shadow for judgment.

Here, the criterion for judgment is one threshold value, and the local region in which the judgment shadow degree is equal to or higher than the threshold value is the shadow region, while the local region in which the judgment shadow degree is less than the threshold value. Can be determined to be a non-shadow region, but in the present embodiment, in order to suppress erroneous determination in the foreground region, the determination criteria are set to two threshold values _TS and _TN ( _TS > _TN ). Configure. That is, it is determined that the local region having the judgment shadow degree of _TS or more is a shadow region and a shadow label is attached, and the local region having the judgment shadow degree of less than _TN is determined to be a non-shadow region. A non-shadow label is attached, and label interpolation processing is performed for a local region where the degree of shadow for determination is _TN or more and less than _TS .

In the label interpolation process, for example, a label image in which a label attached to the local area is set as a pixel value of a pixel corresponding to each local area is generated, and a shadow label is set in the label image and a non-shadow image. It can be an expansion process that expands the area where the label is set toward the area where the label is indefinite.

Alternatively, the label interpolation process can be a fill-in-the-blank process in which a region in which the label is indefinite in the label image is regarded as a hole and the label image is repeatedly filtered by the fill-in-the-blank filter.

Alternatively, the label interpolation process can be a process of replacing with the information of the estimated shadow region estimated by the background information generation means 50. That is, if the local area with an indefinite label is an estimated shadow area, a shadow label is given to the local area, and if the local area with an indefinite label is not an estimated shadow area, a non-shadow label is given to the local area.

The foreground extraction means 54 inputs a captured image from the captured image acquisition means 30, and also inputs a determination result of whether or not it is a shadow region for each local region set in the captured image from the shadow determination means 53, and the foreground is input from the captured image. The area where the object appears is detected, and the foreground information which is the detection result is output to the foreground information analysis means 55.

For example, the foreground extraction means 54 sets a region of the input captured image to which a non-shadow label is attached as a non-shadow portion image in which the sun or the like is photographed, and assigns a shadow label to the label image. The shaded area is taken as a shadow partial image, and the foreground region is detected in each of these partial images, and the foreground information in the captured image is obtained by synthesizing them.

For example, for a non-shadow partial image, the foreground extraction means 54 performs a difference process with the background image. Then, the degree of difference in the pixel values is obtained for each pixel, and a region in which the degree of difference is larger than a predetermined threshold value is detected as a foreground region.

On the other hand, for the shadow partial image, the foreground extraction means 54 performs difference processing with the background image to obtain the degree of difference in pixel values for each pixel, and the degree of difference is larger than a predetermined first threshold value. Is a strong change region, and the region below the first threshold value is a non-strong change region. Then, the foreground extraction means 54 first extracts the strong change region as the foreground region for the shadow portion image.

Further, the foreground extraction means 54 generates a luminance histogram in the non-strong change region of the shadow partial image for each characteristic-similar region. Then, when there are a plurality of peaks in the histogram, assuming that a foreground object exists in a non-strong change region in the characteristic-like region, pixels constituting the peaks other than those generated by the background among the plurality of peaks. The area consisting of the group is extracted as the foreground area. Here, for example, it can be estimated that the highest of the plurality of mountainous areas is caused by the background. It is also possible to estimate the mountainous area due to the background from the rendering result using the environment model.

The foreground extraction means 54 synthesizes the foreground region extracted from the non-shadow partial image, the foreground region extracted separately into the strong change region and the non-strong change region in the shadow partial image, and the region in which the foreground object is captured in the captured image. Ask for. Then, foreground information including one or more of the presence / absence of the foreground object, the position or region where the foreground object exists, the image of the foreground object (foreground image), and the like is generated and output to the foreground information analysis means 55.

The foreground information analysis means 55 analyzes the image, position, and movement of the foreground object output by the foreground extraction means 54, and outputs the analysis result to the analysis result output means 31. The foreground information analysis means 55, for example, detects a monitored object from a foreground object, estimates the posture of the monitored object, tracks the monitored object, and the like.

The analysis result output means 31 outputs the analysis result input from the foreground information analysis means 55 to the notification unit 6.

[Operation of image monitoring device]
FIG. 7 is a schematic flow diagram illustrating the operation of the image monitoring device 1.

The image processing unit 5 first functions as the background information generation means 50, and calculates a characteristic-similar region (step S1). For example, in the example of the reflection characteristic map 100 of FIG. 3, the shaded areas of the images 101 to 104 are obtained as characteristic-similar regions. The background information generation means 50 stores the calculated characteristic-similar region in the background information storage means 42.

The communication unit 3 operates as the captured image acquisition means 30 in a state where the characteristic-similar region is stored in the background information storage means 42, and sequentially acquires the captured images from the camera 2 (step S2).

The image processing unit 5 operates as the background information generation means 50 every time the captured image is acquired from the captured image acquisition means 30, and calculates an estimated shadow region (step S3). Specifically, as described above, the shadow region is estimated by rendering using the environment model storage means 40, the environment model stored in the camera information storage means 41, and the camera parameters. Further, the background information generation means 50 calculates the background image by rendering every time the captured image is acquired. Each time a captured image is acquired, rendering is performed in consideration of changes in a light source such as the sun, so that an estimated shadow area and a background image corresponding to the captured image can be obtained. The background information generation means 50 stores the calculated estimated shadow area and background image in the background information storage means 42.

Next, the image processing unit 5 functions as the learning data generation means 51, generates learning data based on the estimated shadow region and the background image updated in step S3, and stores the learning data in the learning data storage means 43. Accumulate (step S4).

The image processing unit 5 determines whether or not the learning data of the local region belonging to the estimated shadow region and the learning data of the local region not belonging to the estimated shadow region are accumulated in the learning data storage means 43 for all the characteristic-similar regions (step S5). .. Then, when the learning data of the local region belonging to the estimated shadow region is not accumulated for any characteristic similar region, or the local region not belonging to the estimated shadow region for any characteristic similar region If the learning data is not accumulated, it is determined that the accumulation is not sufficient (in the case of "NO" in step S5), and the processes of steps S2 to S4 are repeated until sufficient accumulation is achieved. By the way, if the latest image for determination satisfies the conditions, it is determined that even one image is sufficiently accumulated.

Further, whether or not the accumulation is sufficient may be determined by the accumulation time or the number of accumulated data. Since the acquisition rate of captured images and the number of acquired data in each image are basically constant, the determination based on the accumulation time and the determination based on the number of accumulated data are equivalent. Further, the condition of presence / absence of data to be attributed / not to be attributed and the condition of accumulation time or the number of accumulated data may be used in combination, and it may be determined that sufficient accumulation is satisfied if either of them is satisfied.

When the learning data is sufficiently accumulated (when "YES" in step S5), the image processing unit 5 starts the process of detecting the monitoring target in the captured image.

In this monitoring target detection process, first, the shadow determination model learning process S6 according to the learning device of the present invention is performed, and then the shadow determination process S7 according to the shadow detection device of the present invention is performed.

FIG. 8 is a schematic flow chart of the shadow determination model learning process S6. The image processing unit 5 functions as a learning means 52 to perform the processing. The learning means 52 sequentially sets each characteristic-similar region to the region of interest (step S60), and performs the processes of steps S61 to S63 for all the characteristic-similar regions by loop processing (step S64). In the loop, the learning means 52 reads the learning data of the attention region from the learning data storage means 43 (step S61), generates and updates the shadow determination model of the attention region (step S62), and uses it as a trained model. It is stored in the trained model storage means 44 (step S63).

When the learning means 52 has not completed the processing of steps S61 to S63 for all the characteristic-similar regions (when “NO” in step S64), the processing is returned to step S60 and the unprocessed characteristic-similar regions are focused on the region of interest. Set to and repeat the loop process.

On the other hand, when the loop processing is completed for all the characteristic-similar regions (when “YES” in step S64), the image processing unit 5 advances the processing to step S7.

FIG. 9 is a schematic flow chart of the shadow determination process S7. The image processing unit 5 functions as a shadow determination means 53 to perform the processing. Incidentally, in step S3, the shadow area is estimated by rendering, but in the shadow determination process S7, the shadow area is re-estimated using the shadow determination model. The shadow determination means 53 sequentially sets each local region in the captured image as a local region of interest (step S70), and performs estimation processing for whether or not it is a shadow region in steps S71 to S76 for all local regions by loop processing. Repeat (step S77).

First, the shadow determination means 53 reads a trained model of a characteristic-similar region to which the local region of interest belongs from the trained model storage means 44, and calculates the degree of shadow of the local region of interest using the trained model (step S71). ).

When the shadow degree of the attention local region is equal to or higher than the threshold value _TS (when “YES” in step S72), the shadow determination means 53 assigns a shadow label to the attention local region (step S73), while the shadow determination means 53 applies the shadow label. When the degree of shadow is less than the threshold value _TN (when "NO" in step S72 and "YES" in step S74), a non-shadow label is attached (step S75).

After these label assignment processes, the shadow determination means 53 performs the label interpolation process already described (step S76). The label interpolation process determines a label for a local region of interest whose label is undecided. Specifically, the label interpolation process is performed when the degree of shadow is _TN or more and less than _TS (when "NO" in step S72 and "NO" in step S74).

When the shadow determination means 53 has not completed the processing of steps S71 to S76 for all the local regions (when “NO” in step S77), the processing is returned to step S70 and the unprocessed local region is the attention local region. Set to and repeat the loop process.

On the other hand, when the loop processing is completed for all the local regions (when “YES” in step S77), the image processing unit 5 advances the processing to step S8 in FIG. 7.

The image processing unit 5 functions as the foreground extraction means 54 and the foreground information analysis means 55, and the foreground extraction means 54 extracts the foreground object appearing in the captured image by using the information of the shadow area input from the shadow determination means 53. The foreground information is output to the foreground information analysis means 55 (step S8). Further, the foreground information analysis means 55 analyzes the foreground information input from the foreground extraction means 54 and outputs the analysis result to the analysis result output means 31 (step S9). Then, the analysis result output means 31 outputs the analysis result input from the foreground information analysis means 55 to the notification unit 6 (step S10).

When the above processing is completed for the captured image acquired in step S2, the processing is returned to step S2 again, and the above-mentioned training data generation and shadow determination model for the newly acquired captured image are described. The processing of steps S3 to S10 including the learning of the above and the shadow determination using the same is repeated.

[Modification example]
(1) In the above embodiment, the background information generation means 50 renders the environment model to calculate the characteristic similar region, but the background information generation means 50 applies a process called semantic segmentation to the background image. By doing so, it is also possible to calculate a characteristic-similar region.

For more information on semantic segmentation, see, for example, "Pyramid Scene Parsing Network" Hengshuang Zhao, et al. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 and "DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution," and Fully Connected CRFs "LC Chen, et al. IEEE transactions on pattern analysis and machine intelligence 40 (4), 834-848.

In that case, the storage unit 4 stores a trained model learned in advance for each of the images of the background and foreground components including the image of the background component and the image of the object that will appear in the monitoring space. back. Then, the background information generation means 50 divides the entire photographed image into areas for each background / foreground component by searching for the photographed image using the trained model, and each of the areas of the background component in the divided areas. By assigning different reflection characteristic IDs to each other, a characteristic-similar region is calculated.

(2) In the above embodiment and each modification thereof, an example of the camera 2 in which the field of view is fixed and the camera parameter is a constant value has been described. However, in the above embodiment and the modification thereof, panning, tilting, and zooming are possible. It is also possible to use a camera 2 whose camera parameters change, such as a PTZ camera, an in-vehicle camera, an aerial camera, or the like. In that case, the image processing unit 5 updates the characteristic-similar region when it detects a change in the camera parameter.

For example, the camera 2 calculates the camera parameters at the time of shooting each time and outputs them together with the shot image. In step S2 of the processing flow shown in FIG. 7, the captured image acquisition means 30 outputs the input camera parameters to the background information generation means 50, and the background information generation means 50 outputs the input camera parameters to the camera information storage means 41. It is determined whether or not they match by comparing with the camera parameters stored in, and if they do not match, the input camera parameters are overwritten and stored in the camera information storage means 41, and the characteristics are the same as in step S1 of FIG. A similar area is calculated, and the calculated characteristic similar area is overwritten and stored in the background information storage means 42.

(3) In the above embodiment and each modification thereof, the training data is generated from the local region of each learning captured image, and the shadow determination means 53 corresponds to the determination data one by one. Although an example of extracting from the local area of the image for determination is shown, in the above embodiment and each modification, the training data is generated from the time series of the image for learning in the local area, and the judgment data is the local area. It may be extracted from the time series of the image taken for determination in.

For example, the image feature in the training data can be a vector in which five feature vectors extracted from the same local region of five learning images taken every 10 minutes are arranged in descending order of time and concatenated. Correspondingly, each image feature in the determination data is, for example, a vector in which five feature vectors extracted from the same local region of five determination captured images every 10 minutes are connected.

In that case, the degree of shadow for learning and the degree of shadow for determination are also in chronological order. For example, in the above example using the time series of five captured images, the shadow degree for learning and the shadow degree for determination are five-dimensional vectors in which the five shadow degrees are arranged in descending order of time, respectively, with respect to the latest captured image for determination. The degree of shadow at the time of determination is the first value of the five-dimensional vector.

By using the image features and the degree of shadow as time-series data in this way, it is possible to learn and judge including the time variation from shadow to non-shadow and from non-shadow to shadow, so shadow learning can be performed with higher accuracy. However, it becomes possible to determine the shadow area with higher accuracy. This configuration is useful when the accumulation of learning data over a long period of time is acceptable.

(4) In the above-described embodiment and each modification thereof, an example in which the latest captured image is included in the captured image for learning is shown, but if long-term accumulation is performed in the same visual field, the update of the shadow determination model is stopped. It is also possible to configure the configuration so that the latest captured image is not included. In that case, for example, if the threshold value used for the accumulation time in step S5 is T ₁ , a threshold value T ₂ that is sufficiently larger than T ₁ is used, and the learning data generation means 51 has an accumulation time of T ₂ or more. If so, the generation of learning data is stopped, and if the accumulation time is T ₂ or more, the learning means 52 stops updating the shadow determination model.

(5) In the above-described embodiment and each modification thereof, the shadow determination means 53 uniquely selects a trained model for each characteristic-similar region and which has a characteristic-similar region to which the local region belongs. An example of inputting the image features of is shown. Regarding this point, in the above-described embodiment and each modification, the shadow determination means 53 does not selectively select the trained model, but determines using the trained models of a plurality of characteristic-similar regions with respect to the local region. May be good.

For example, the shadow determination means 53 inputs the image features of the local region in the captured image for determination into all the trained models, obtains a plurality of determination shadow degrees for the local region, and sets the maximum value of the determination shadow degree as a shadow. Compare with the judgment threshold. By doing so, it is possible to perform highly accurate shadow determination even if there is a setting error in the characteristic-similar region.

Further, instead of inputting to all the trained models, it may be input to the trained model for the characteristic-similar region to which the local region belongs and the characteristic-similar region adjacent to the characteristic-similar region. Even in this way, highly accurate shadow determination is possible even if there is a setting error in the characteristic-similar region.

(6) In the above-described embodiment and each modification thereof, an example in which a shadow determination model is learned for each characteristic-similar region and shadow determination is performed using the trained model for each characteristic-similar region is shown. In each modification, one shadow determination model common to all characteristic similar regions can be learned, and the shadow determination can be performed using the one learned model.

In that case, the trained model is trained in which the image feature of the local region and the information of the reflection characteristic (for example, the reflection characteristic ID) for the characteristic similar region to which the local region belongs are input and the degree of shadow for the local region is output. It is preferable to use it as a model. Correspondingly, the learning means 52 for learning the degree of shadow obtained by inputting the image characteristics of the local region in the captured image for learning and the information of the reflection characteristics for the characteristic similar region to which the local region belongs into the shadow determination model. A trained model is generated by updating the shadow determination model to bring it closer to the degree of shadow. On the other hand, the shadow determination means 53 inputs the image characteristics of the local region in the image for determination and the reflection characteristic information of the characteristic similar region to which the local region belongs into the trained model, and the local region is based on the degree of shadow obtained. It is determined whether or not the area is a shadow area.

In the image monitoring device 1 described above, the shadow determination means 53 takes a picture using a trained model that outputs the degree of shadow of the local area based on learning when the image feature of the local area of the captured image is input. Performs shadow judgment on the image. Therefore, it is possible to determine with high accuracy the shadow area in the captured image obtained by capturing the surveillance space having a complicated background.

In particular, in the image monitoring device 1, the shadow determination means 53 determines the shadow of the captured image using the trained model learned for each characteristic-similar region. As a result, even if the captured image is a photographed image of a surveillance space having a complicated background (a photographed image of a background composed of a plurality of background components having different reflection characteristics from each other), a substantially single reflection characteristic is obtained in a characteristic-similar region. Since it becomes the background of, highly accurate shadow judgment is possible.

Further, in the image monitoring device 1, since the above model can be learned by the learning means 52, a trained model capable of accurately determining the shadow region in the captured image obtained by photographing the surveillance space having a complicated background is generated. be able to.

In particular, in the image monitoring device 1, the learning means 52 learns for each characteristic-similar region. As a result, even if the background is a photographed image of a complicated surveillance space, the background has substantially a single reflection characteristic in the characteristic-similar region, so that a trained model capable of highly accurate shadow determination can be generated. ..

1 image monitoring device, 2 cameras, 3 communication units, 4 storage units, 5 image processing units, 6 notification units, 30 captured image acquisition means, 31 analysis result output means, 40 environment model storage means, 41 camera information storage means, 42. Background information storage means, 43 Learning data storage means 43, 44 Learned model storage means, 50 Background information generation means, 51 Learning data generation means, 52 Learning means, 53 Shadow determination means, 54 Foreground extraction means, 55 Foreground information Analytical means.

Claims (10)

A shadow detection device that detects a shadow area in which a shadow is captured in a captured image captured in a predetermined space.
This is a trained model in which the image features in the local area set in the captured image are input and the degree of shadow indicating the high possibility that a shadow is captured in the local area is output, and the space is photographed. A trained model memory that stores a trained model that has been trained using the image features in the local region set in the captured image for training and the training shadow degree that is the estimated shadow degree for the local region. Means and
A shadow obtained by inputting an image feature in the local region of the captured image into the trained model is compared with a predetermined reference, and a local region in which the shadow degree exceeds the reference is determined to be the shadow region. Judgment means and
A shadow detection device characterized by being equipped with. The trained model storage means stores the trained model for each characteristic-similar region in which the training is performed for each characteristic-similar region in which the reflection characteristics of the background component that can be captured in the captured image are similar. The shadow detection device according to claim 1, wherein the shadow detection device is characterized by. Further provided with a background information storage means for storing the characteristic-similar region,
The shadow determination means inputs the image feature of the local region into the trained model of the characteristic-like region corresponding to the local region to obtain the shadow degree.
The shadow detection device according to claim 2. Further, a background information storage means for storing a characteristic-similar region having similar reflection characteristics of a background component that can be captured in the captured image is provided.
In addition to the image features and the degree of shadow for learning about the local region set in the captured image for learning, the trained model storage means also includes attribution information indicating the characteristic-similar region to which the local region belongs. The trained model in which the training was performed is stored using the trained model.
The shadow determination means inputs the image feature of the local region and the attribution information about the local region into the trained model to obtain the shadow degree.
The shadow detection device according to claim 1. A learning device that trains a learning model that inputs image features in a local area set in a captured image captured in a predetermined space and outputs a degree of shadow that indicates the high possibility that a shadow is captured in the local area. And,
A learning model storage means for storing the learning model and
At least, a learning data storage means that stores an image feature in a local region set in a learning image obtained by capturing the space and a learning shadow degree that is the estimated shadow degree for the local region.
At least the image features of the local region in the captured image for learning are input to the learning model, and learning to update the learning model based on the error of the obtained shadow degree with respect to the learning shadow degree of the local region is performed. The learning method to be performed and
A learning device characterized by being equipped with. The learning model storage means stores the learning model for each characteristic-similar region in which the reflection characteristics of the background components that can be captured in the captured image are similar.
The learning means performs the learning about the learning model for each characteristic-similar region.
The learning device according to claim 5. It is a shadow detection method that detects a shadow area where a shadow is captured in a captured image captured in a predetermined space.
This is a trained model in which the image features in the local area set in the captured image are input and the degree of shadow indicating the high possibility that a shadow is captured in the local area is output, and the space is photographed. Using a trained model that has been trained using the image features in the local region set in the captured image for training and the shadow degree for learning, which is the estimated shadow degree for the local region, is used.
An input step of inputting an image feature in the local region of the captured image into the trained model, and
A shadow determination step in which the shadow degree output from the trained model is compared with a predetermined reference, and a local region in which the shadow degree exceeds the reference is determined to be the shadow region.
A shadow detection method comprising. It is a program for causing a computer to perform a process of detecting a shadow area in which a shadow is photographed in a photographed image in which a predetermined space is photographed.
This is a trained model in which the image features in the local area set in the captured image are input and the degree of shadow indicating the high possibility that a shadow is captured in the local area is output, and the space is photographed. A trained model memory that stores a trained model that has been trained using the image features in the local region set in the captured image for training and the training shadow degree that is the estimated shadow degree for the local region. Means and
A shadow obtained by inputting an image feature in the local region of the captured image into the trained model is compared with a predetermined reference, and a local region in which the shadow degree exceeds the reference is determined to be the shadow region. Judgment means,
A shadow detection program characterized by functioning as. A learning method that trains a learning model that inputs image features in a local area set in a captured image captured in a predetermined space and outputs a degree of shadow that indicates the high possibility that a shadow is captured in the local area. And,
A learning model storage means for storing the learning model and
At least, a learning data storage means for storing an image feature in a local region set in a learning image obtained by photographing the space and a learning shadow degree which is the shadow degree estimated for the local region. Use,
An input step of inputting at least the image features of the local region in the captured image for learning into the learning model, and
A learning step of learning to update the learning model based on an error of the shadow degree output from the learning model with respect to the learning shadow degree of the local region, and a learning step.
A learning method characterized by including. A process of training a learning model that inputs an image feature in a local area set in a captured image taken in a predetermined space and outputs a degree of shadow indicating the high possibility that a shadow is captured in the local area. It is a program to make a computer do, and the computer is
A learning model storage means for storing the learning model,
At least, a learning data storage means that stores an image feature in a local region set in a learning image obtained by capturing the space and a learning shadow degree that is the estimated shadow degree for the local region, and a learning data storage means.
At least the image features of the local region in the captured image for learning are input to the learning model, and learning to update the learning model based on the error of the obtained shadow degree with respect to the learning shadow degree of the local region is performed. Learning means to do,
A learning program characterized by functioning as.

Images