JP2021056885A - Detector, detection method, and program - Google Patents

Abstract

To reduce processing costs of a detector to improve detection accuracy.SOLUTION: One or more objects are detected from a photographed image. In accordance with positions of one or more objects detected from photographed image at first time, a detection target region for one or more objects is set for photographed image at second time, following the first time, which is referenced by detection means.SELECTED DRAWING: Figure 6

Description

The present invention relates to a detection device, a detection method, and a program.

There is a technique for estimating the position of a subject using a fixed camera. Most of these techniques estimate the trajectory of a subject by detecting the subject in a plurality of images that are continuous in time and determining their identity. For example, Patent Document 1 discloses a method of controlling and tracking a pan-tilt zoom based on prediction and updating of the movement of a tracking target object using a state space model.

Further, in recent years, many techniques have been proposed to execute object detection of a plurality of categories at high speed by using a convolutional neural network (hereinafter referred to as CNN). For example, in the technique disclosed in Non-Patent Document 1, by inputting an input image having a size of 352 × 352 into a neural network, 20 categories of object detection problems can be executed at 81 frames per second.

On the other hand, the resolution of the image captured by a general surveillance camera is larger, for example, 1920 × 1080 size. If an image of such a size is resized to a small size and input to the CNN, the detection accuracy of the subject is lowered. Non-Patent Document 2 discloses a method of selecting a partial region to be selectively zoomed in for detecting a subject from an image obtained by resizing the original image to reduce the resolution.

Japanese Patent No. 5018321 Joseph Redmon, Ali, Farhadi, "YOLO9000: Better, Faster, Stronger", CVPR2017 Mingfei Gao, Ruichi Yu, Ang Li, Vlad I. Morariu, Larry S.M. Davis, "Dynamic Zoom-in network for fast object detection in range images", arXiv: 1711.05187v1

However, in the method described in Non-Patent Document 2, in order to determine a partial region, processing on the original image using a CNN-based detector, which has a high processing cost, is always performed every hour, which becomes a bottleneck of processing. ing.

An object of the present invention is to reduce the processing cost of subject detection processing.

In order to achieve the object of the present invention, for example, the detection device according to one embodiment has the following configuration. That is, the first detection means referred to by the detection means according to the position of the detection means for detecting one or more subjects from the captured image and the position of one or more subjects detected from the captured image at the first time by the detection means. It is characterized by including a setting means for setting a detection target area of one or more subjects in the captured image at a second time following the time of.

The processing cost of the subject detection process can be reduced.

The figure which shows an example of the captured image in the detection apparatus which concerns on Embodiment 1. FIG. The figure which shows an example of the functional structure of the detection apparatus which concerns on Embodiments 1-3. The flowchart of the processing example in the detection method which concerns on Embodiments 1-3. The figure which shows the creation example of the candidate area in the detection method which concerns on Embodiment 1. FIG. The figure which shows the structural example of the candidate area in the detection method which concerns on Embodiment 1. FIG. The figure which shows the setting example of the detection target area in the detection method which concerns on Embodiment 1. FIG. The figure which shows an example of the candidate area list in the detection method which concerns on Embodiment 1. FIG. The figure which shows the Example of the detection apparatus which concerns on Embodiment 3. FIG. The flowchart which shows the setting example in the detection method which concerns on Embodiment 3. The figure which shows the setting example of the detection target area in the detection method which concerns on Embodiment 3. FIG. The figure which shows an example of the list of the candidate area in the detection method which concerns on Embodiment 3. The figure which shows an example of the detection target area list in the detection method which concerns on Embodiment 3. The figure which shows an example of the functional structure of the detection apparatus which concerns on Embodiment 4. FIG. The flowchart of the processing example in the detection method which concerns on Embodiment 4. The flowchart which shows the setting example in the detection method which concerns on Embodiment 4. The figure which shows an example of the structure of the imaging system which concerns on Embodiment 1. FIG. The figure which shows an example of the internal structure of the detection apparatus which concerns on Embodiment 1. FIG. The figure which shows an example of the internal structure of the client apparatus which concerns on Embodiment 1. FIG.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

[Embodiment 1]
FIG. 16 is a block diagram showing an example of the configuration of the imaging system 1600 according to the present embodiment. The imaging system 1600 shown in FIG. 16 includes a detection device 1605, a client device 1602 connected to each other via a network 1601 in a communicable state, an input device 1603, and a display device 1604. The detection device 1605 may be, for example, a surveillance camera or a network camera that captures and processes a moving image.

FIG. 17 is a block diagram showing an example of the internal configuration of the detection device 1605 according to the present embodiment. The optical unit 1701 is composed of a focus lens, a blur correction lens, an aperture, and a shutter, and collects light information of a subject. The image sensor unit 1702 is an element that converts the light information collected by the optical unit 1701 into a current value, and acquires color information by combining with a color filter or the like. In addition, an image sensor capable of setting an arbitrary exposure time for all pixels is used. The CPU 1703 is involved in all the processes of each configuration, sequentially reads and interprets the instructions stored in the ROM (Read Only Memory) 1704 and the RAM (Random Access Memory) 1705, and executes the processes according to the result. The CPU 1703 executes each process according to the present embodiment by reading and executing various programs stored in the ROM 1704 or the like into the RAM 1705, and controls transmission / reception of various information to / from the client device 1602.

Further, the image pickup system control unit 1706 performs control instructed by the CPU 1703, such as focusing, opening the shutter, and adjusting the aperture, on the optical unit 1701. The control unit 1707 performs control such as controlling the imaging range of the detection device 1605 in response to an instruction from the client device 1602. The A / D conversion unit 1708 converts the amount of light of the subject detected by the optical unit 1701 into a digital signal value. The image processing unit 1709 performs image processing on the image data of the above digital signal. The encoder unit 1710 performs a process of converting the image data processed by the image processing unit 1709 into a file format such as Motion Jpeg, H.264, or H.265. The still image or moving image data generated by the conversion process in the encoder unit 1710 is provided to the client device 1602 as a "delivered image" via the network 1601. The network I / F 1711 is an interface used for communication with an external device such as the client device 1602 via the network 1601.

The network 1601 is a network that connects the detection device 1605 and the client device 1602. The network 1601 is composed of a plurality of routers, switches, cables and the like that satisfy communication standards such as Ethernet (registered trademark). In the present embodiment, the network 1601 may be any network 1601 as long as it can communicate between the detection device 1605 and the client device 1602, regardless of its communication standard, scale, and configuration. For example, the network 1601 may be configured by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), or the like.

FIG. 18 is a block diagram showing an example of the internal configuration of the client device 1602 corresponding to the present embodiment. The client device 1602 includes a CPU 1801, a main storage device 1802, an auxiliary storage device 1803, an input I / F 1804, an output I / F 1805, and a network I / F 1806. The elements are communicatively connected to each other via the system bus. The client device 1602 can operate as a setting device for making various settings of the detection device 1605.

The CPU 1801 controls the operation of the client device 1602. The main storage device 1802 is a storage device such as a RAM that functions as a temporary storage place for data of the CPU 1801. The auxiliary storage device 1803 is a storage device such as an HDD, a ROM, or an SSD that stores various programs, various setting data, and the like. The input I / F 1804 is an interface used when receiving an input from the input device 1603 or the like. The output I / F 1805 is an interface used for outputting information to a display device 1604 or the like. The network I / F1806 is an interface used for communication with an external device such as the detection device 1605 via the network 1601. The client device 1602 can acquire and store a captured image or video from the detection device 1605 via the network I / F 1806. The client device 1602 may function as a server that stores and provides such images. Further, it is not limited to the auxiliary storage device 1803 that the client device 1602 stores various programs, various setting data, and the like. For example, the client device 1602 may store such data or the like in an external storage unit (not shown) such as a server or a storage device via the network I / F 1806.

The CPU 1801 reads and executes various programs stored in the auxiliary storage device 1803 in the main storage device 1802 to execute each process according to the present embodiment, and transmits and receives various information to and from the detection device 1605. Control. Further, the input from the input device 1603 is received via the input I / F 1804, and the display control of the image and various information in the display device 1604 is performed via the output I / F 1805. Further, the client device 1602 may use the auxiliary storage device 1803 and an external storage unit (not shown).

The input device 1603 is an input device composed of a mouse, a keyboard, a touch panel, buttons, and the like. The display device 1604 is a display device such as a display monitor that displays an image output by the client device 1602. In the present embodiment, the client device 1602, the input device 1603, and the display device 1604 can be independent devices. In this case, for example, the client device 1602 can be configured as a personal computer (PC), the input device 1603 can be a mouse or keyboard connected to the PC, and the display device 1604 can be a display connected to the PC. In addition to the configuration, the client device 1602 and the display device 1604 may be integrated, or the input device 1603 and the display device 1604 may be integrated like a touch panel. Further, the client device 1602, the input device 1603, and the display device 1604 may be integrated like a smartphone or a tablet terminal. Further, the display device 1604 may function as a monitoring unit 1300 described later.

The detection device according to the present embodiment detects one or more subjects from the captured image at the first time, and according to the position of the detected subject, the subject at the second time following the first time. The detection target area is set in the captured image. For such processing, the detection device 1000 according to the embodiment shown in FIG. 2A has an imaging unit 1100 and a processing unit 1200. Here, the detection device 1000 may be the detection device 1605 shown in FIG. In this case, the processing of the processing unit 1200 can be realized by the control unit 1707 of the detection device 1605. Further, the detection device according to the embodiment of the present invention may be composed of a plurality of devices connected via a network. For example, the function of the detection device 1000 shown in FIG. 16 may be realized by the detection device 1605 and the client device 1602 shown in FIG. For example, the detection device 1605 may be used as the imaging unit 1100, and the client device 1602 may be used as the processing unit 1200. In this case, the processing of the processing unit 1200 can be realized by the CPU 1801 of the client device 1602.

FIG. 2 is a block diagram showing an example of the functional configuration of the detection device according to each embodiment, and FIG. 2A shows an example of the detection device according to the first embodiment. The imaging unit 1100 has a moving image acquisition unit 1001. The moving image acquisition unit 1001 acquires an image captured by the imaging device. In this embodiment, the moving image acquisition unit 1001 can acquire, for example, a captured image of a predetermined area including a subject. The resolution of the captured image by the moving image acquisition unit 1001 is not particularly limited, but in the present embodiment, for the sake of explanation, the moving image acquisition unit 1001 shall acquire the captured image having a resolution of FHD (1920 × 1080 pixels). The moving image acquisition unit 1001 can acquire captured images at predetermined time intervals. For example, the moving image acquisition unit 1001 may perform imaging at a speed of 30 frames per second, may perform imaging at intervals of about several tens of milliseconds, or may perform imaging at wider intervals. In addition, the moving image acquisition unit 1001 can output the acquired captured image to the processing unit 1200. Further, the imaging unit 1100 is connected to the processing unit 1200. The connecting means between the imaging unit 1100 and the processing unit 1200 is not particularly limited. The imaging unit 1100 and the processing unit 1200 may be connected via a communication path such as a local area network, or may be connected by wire via a USB cable or the like. Further, for example, the imaging unit 1100 may store the output captured image in a storage device (not shown), and the processing unit 1200 may acquire a predetermined frame from the storage device.

In the example of FIG. 2A, the processing unit 1200 has an initial value setting unit 1002, a detection unit 1003, a correspondence unit 1004, an area setting unit 1005, and a visualization unit 1006. When the image pickup unit 1100 performs the subject tracking process, each unit of the processing unit 1200 can repeat the process. The initial value setting unit 1002 is used when the detection unit 1003 first detects the subject, and initially sets the detection target area set in the captured image. The detection unit 1003 detects one or more subjects from the detection target area in the captured image. The association unit 1004 associates the image of the subject detected in the previous repetition with the image of the subject detected this time, or allocates the identification information to the subject in the case of the first time. The area setting unit 1005 sets the detection target area used when the detection unit 1003 detects the subject in the captured image in the next repetitive process. The visualization unit 1006 visualizes the trajectory of the subject. Details of these functions will be described later together with the flowchart of FIG. 3 (a).

The monitoring unit 1300 can display the result of processing by the processing unit 1200. For example, the monitoring unit 1300 may superimpose and display the locus of the subject visualized by the visualization unit 1006 as a locus or a point on the captured image in the monitor. Further, the monitoring unit 1300 may be connected to the processing unit 1200. The connection method between the monitoring unit 1300 and the processing unit 1200 is not particularly limited. For example, the monitoring unit 1300 and the processing unit 1200 may be connected by wire or may be connected via wireless communication.

FIG. 1 is a diagram for explaining an example of acquiring a captured image by the detection device 1000 according to the present embodiment. The arrangement example 104 of FIG. 1A is a bird's-eye view showing an arrangement example of a group of people existing in the space and a camera 101 which is an imaging unit 1100 installed in the space. In this example, the camera 101 is capturing images of people 1, 2, 3 and 4. An example of an image captured by such a camera 101 is shown in image example 110 of FIG. 1 (b). Persons 1, 2, 3 and 4 in FIGS. 1 (a) and 1 (b) correspond to each other, respectively. The detection target area 111 shown in FIG. 1B is an example of the detection target area set by the detection device 1000. Further, 112 and 113 are bounding boxes corresponding to the detection results by the detection device 1000, which correspond to the persons 1 and 2, respectively. The bounding box may be a rectangle represented by a total of four-dimensional numerical values of position and width in the vertical direction (u-axis direction) and the horizontal direction (v-axis direction) of the image, respectively. In this example, the detection device 1000 is learned to output as many bounding boxes surrounding the heads of the human body as the number of heads detected on the image. However, the detection result output by the detection device 1000 is not particularly limited, and identification information such as an ID associated with the subject or a name corresponding to the ID may be displayed. In addition to such a bounding box, the detection device 1000 can output a score indicating the reliability of the detection result.

The score representing the reliability of the detection result may be, for example, a model showing how accurately the detection device 1000 detects such a subject with respect to the subject included in the detection range. In the example of the first embodiment, the detection device 1000 may detect the subject by the same method as in Non-Patent Document 1. For example, the detection unit 1003, which will be described later, can divide each of the detection target areas into a grid of S × S (S is a predetermined number given in advance). Further, the detection unit 1003 may estimate a predetermined number of bounding boxes and a reliability score in each bounding box from each grid in which the subject exists. Next, the detection device 1000 can estimate a bounding box having a score exceeding an arbitrary threshold value as a bounding box surrounding the subject from among a plurality of bounding boxes set in the detection target area. In the example of Non-Patent Document 1, the bounding box and the score are estimated using a neural network. In this example, a neural trained to give the product of the probability that a subject exists and IoU (the ratio of the correct subject area to the area obtained by adding the correct subject area and the falsely detected area as the subject) as a score. A network is used. As described above, the score is at least one of the correctness of the position of the estimated subject area, the correctness of the estimated size of the subject area, and the probability that the subject exists in the estimated subject area. It may be a value indicating. In addition, the detection device 1000 can detect a plurality of subjects. Further, in this example, the head of a person is detected, but the detection target of the detection device 1000 is not limited to this. The detection device 1000 may detect an animal such as a dog or a horse, or may detect a soccer ball.

Hereinafter, the flow of the detection method performed by the detection device 1000 according to the present embodiment will be described with reference to FIG. 3A. FIG. 3A is a flowchart showing an example of a processing procedure when a subject is recognized in the present embodiment. In the present embodiment, the detection device 1000 detects a subject from each of the captured images captured at each of the plurality of times 1 to t, and detects the subject with respect to the captured image captured at the next time. Set the detection target area. In the loop L4001, the detection device 1000 can repeat the following operations of steps S4002 to S4005 in order for each of the captured images captured from time 1 to t, and proceed to the captured image at the next time. .. In the following, this time refers to the current loop that processes the captured image at a certain time, the previous time refers to the loop that processes the captured image at the previous time, and the next time refers to the loop that processes the captured image at a later time. It shall point to.

In step S4001, the initial value setting unit 1002 sets one or more detection target areas for first detecting the subject in the captured image acquired by the moving image acquisition unit 1001. As the detection target region, for example, a region having a size of 640 × 360 may be used with respect to a captured image of FHD (1920 × 1080 size). In such a case, the initial value setting unit 1002 can first set the detection target area in the upper left corner, for example. Next, the initial value setting unit 1002 slides the detection target area by 640 pixels in the horizontal direction and 360 pixels in the vertical direction at a maximum of two times in each direction, thereby causing a total of nine detection targets. The area may be set. For example, from the viewpoint of enabling the subject to be detected anywhere in the image, the initial value setting unit 1002 sets the detection target area so that the set of the detection target areas covers the entire area of the captured image without gaps. You may. However, the method of setting the detection target area is not particularly limited as such. For example, the initial value setting unit 1002 may set the detection target area so as to cover such a range without a gap when a range of positions where the subject can exist is given in advance. Further, the initial value setting unit 1002 may set the detection target area so that the adjacent detection target areas have overlapping areas in consideration of the possibility that the subject exists on the boundary line between the detection target areas. ..

In step S4002, the detection unit 1003 detects the subject from the detection target area set in step S4001 or set in step S4004 (described later) in the previous loop in the captured image. Further, when t = 1, that is, when the first detection is performed, the detection unit 1003 may detect the subject by using the detection target area. The detection unit 1003 can output a bounding box indicating the subject and a score indicating the reliability of the detection result for the detected subject. Further, when the detection unit 1003 detects the same subject in a plurality of detection target regions, the results may be integrated. In such a case, the method of integration is not particularly limited. For example, the detection unit 1003 may integrate the detection results by calculating the center coordinates (u, v) of the bounding box of the same subject in each detection target area and averaging them. Further, for example, when weights based on the size of each detection target area are set, the detection unit 1003 determines the subject (u, v) based on the weights of the detection target areas having the same subject. The results may be integrated by taking a weighted average of the values of).

In step S4003, the associating unit 1004 associates the identification information associated with the subject detected in the previous loop with the subject detected this time. That is, the images of the same subject in the previous time and this time are associated with each other. When a newly detected subject exists, the association unit 1004 allocates new identification information to the subject. Further, when t = 1, the association unit 1004 allocates identification information to each of the detected subjects. The association unit 1004 is, for example, the previous one and the current one of the three-dimensional values (u, v, q) based on the center coordinates (u, v) of the bounding box of each subject and the reliability score (q). The Euclidean distance to and can be calculated for all combinations. In such a case, the association unit 1004 may associate the image of the subject as, for example, as an allocation problem of the linear programming method. That is, for example, the association unit 1004 may associate the previous image with the current image using the above-mentioned Euclidean distance by using a known technique such as the Hungarian method. As the identification information, an ID is used in this embodiment, but the information is not particularly limited as long as it can identify each subject.

In step S4004, the area setting unit 1005 sets the detection target area to be used for detection in the next step S4002. In this embodiment, the area setting unit 1005 first acquires the coordinates of the candidate area that is a candidate for the detection target area in the captured image. The candidate area will be described later. The area setting unit 1005 can select a candidate area including one or more subjects from a plurality of candidate areas as a detection target area. Further, for example, the area setting unit 1005 may select a detection target area satisfying a desired condition from among the candidate areas including one or more subjects by using a score indicating the reliability of the detection result of the subject. Good. The process performed by the area setting unit 1005 will be described in detail after step S4005. In step S4005, the visualization unit 1006 can visualize and display the subject detected from the processed captured image. The method of visualizing the subject by the visualization unit 1006 is not particularly limited. The visualization unit 1006 may display the subject on the monitoring unit 1300 as a bounding box, for example. Further, the visualization unit 1006 may display identification information such as a subject ID or a name corresponding to the subject together with the subject or the trajectory of the subject. Further, for example, the visualization unit 1006 may display on the monitoring unit 1300 a line indicating the transition of the center point of the bounding box over a plurality of times, which is detected from the captured images at different times as the subject.

Hereinafter, the processing performed by the area setting unit 1005 will be described in detail. FIG. 4 is a diagram for explaining the acquisition of the above-mentioned candidate region. The area setting unit 1005 can select at least one or more of the candidate areas having different sizes, that is, sizes, as the detection target area. That is, the selected detection target areas may have different sizes from each other. In the example of FIG. 4, the area setting unit 1005 acquires the coordinates of the candidate areas of three types of sizes 1 to 3, and creates a partial image from the captured image based on the candidate areas of each size. .. The candidate area is a candidate area when selecting the detection target area in step S4004 described later. The position and shape of the candidate region can be predetermined, for example, as shown in FIG. The shape of the candidate region is not particularly limited and may be, for example, a triangle or a circle, but in the following, for the sake of explanation, the candidate region is assumed to be a rectangular region. The area setting unit 1005 may acquire the coordinates of the four corners of the candidate area which is a rectangle, for example.

Reference numeral 400 denotes a diagram in which the area setting unit 1005 creates a candidate area of size 1 within the imaging range, and Nu1 × Nv1 candidate areas are created. In this 400, the candidate area 401 indicates the first candidate area (C _1,1,1 ), and the candidate area 402 indicates the Nu1 × Nv1th candidate area (C1 _{, Nu1, Nv1} ). That is, for example, in the 400, the area setting unit 1005 creates first a _{(C 1, 1, 1), and,} Nu1 pieces laterally from _{(C 1, 1, 1)} (in the order _{(C 1,} Candidate areas can be created up to _Nu1,1). Next, the area setting unit 1005 can create one Nv candidate area in the vertical direction (for example, up to (C _{1, 1, Nv1} ) in order) from each of the one Nu 1 candidate area in the horizontal direction. Similarly, in 410 and 420, the area setting unit _{1005, (C 2,1,1)} from _{(C 2, Nu2, Nv2)} Nu2 × Nv2 amino up, and _{(C 3,1,1)} from (C It is possible to create each of Nu3 × Nv3 candidate regions up to _{3, Nu3, Nv3).}

The candidate regions may have overlapping ranges, may be in contact with each other, or may be separated from each other within a range in which a desired detection result can be obtained. In this example, the intervals between adjacent candidate regions of the same size are set at equal intervals in the vertical direction and the horizontal direction, but this is not particularly limited. For example, in the captured image, there may be a range in which the candidate regions are arranged at appropriate narrow intervals (that is, for example, the candidate regions widely overlap each other). According to such a configuration, since the detection process is performed a plurality of times in the range where the candidates overlap each other, the robustness of the detection can be improved. As one pattern of the candidate area to be created, an area similar to the detection target area set in S4001 may be created.

The area setting unit 1005 can confirm in which area of the above-mentioned candidate areas the subject detected in step S4002 is included. For example, in each of the candidate regions, the region setting unit 1005 may associate the identification information of the subject with the score of the subject for the subject covered by the candidate region. An example of a table listing such associated candidate regions is shown in FIG. The table shown in FIG. 7 displays the ID, which is the identification information of the subject covered by the candidate area, and the score of the candidate area for each of the candidate areas. In the example of FIG. 7, when one candidate area covers a plurality of subjects, the score of the candidate area having the highest value among the scores of the covered subjects is displayed. There is. According to such a setting, in the selection of the detection target area described later, a candidate area having a high score, that is, a candidate area covering a subject that is easy to detect is preferentially selected.

FIG. 5 shows an example of a candidate region for explaining a state in which the candidate region covers the subject. In this example, the candidate region 500 has a coverage determination region 501 and a buffer width 502. When the candidate region 500 has a subject in the covering determination region 501, for example, the candidate region 500 may cover the subject. The buffer width is set outside the covering determination area with a margin above the detection target area so that the subject does not easily go out of the detection target area when the candidate area 500 is selected as the next detection target area. It may be the width of the buffer area to be created. The value of the buffer width 502 is not particularly limited. The value of the buffer width 502 may be set as a value equal to the movement distance that the subject can move by the next time, considering that the candidate area 500 is used for detecting the next subject, for example. .. Such a moving distance of the subject may be given in advance or calculated during detection, but such an example will be described in detail in the second embodiment. Further, the buffer width may have different values at the right end and the left end in the horizontal direction and the upper end and the lower end in the vertical direction in the image. That is, for example, when the traveling direction of the subject is fixed, the value of the buffer width may be set so that the area due to the buffer width becomes larger in the same direction as the traveling direction.

FIG. 6 is a diagram for explaining a bounding box of the detected subject, a candidate area, and a detection target area obtained in step S4004. Example 600 showing the region has bounding boxes 601, 602, 603 and 604, candidate regions 605, 606, and 608, and detection target regions 607 and 609. The detection target area set here is used by the detection unit 1003 in the detection in step S4002 in the next loop.

As described above, the area setting unit 1005 selects a detection target area satisfying a desired condition from among the candidate areas including one or more subjects by using a score indicating the reliability of the detection result of the subject. You may. An example of selecting such a detection target region will be described with reference to the table of FIG. 7. First, for example, the area setting unit 1005 removes an area that does not cover the subject, such as _{(C 1, 1, 1) in FIG. 7, from the candidate areas.} Next, the area setting unit 1005 selects one candidate area, that is, a detection target area from a plurality of candidate areas having the same set of subjects to be covered and the same score of the candidate areas from the remaining candidate areas. can do. The conditions for selection here are not particularly limited. For example, the area setting unit 1005 may preferentially select an area having a smaller area size among the areas covering the same subject and having the same score of the candidate areas. Further, for example, the area setting unit 1005 covers the same subject and selects a candidate area whose center position is closest to the average position of the covered subject from the candidate areas having the same score and size. You may. In the candidate area remaining by such processing, the combination of the set of covered subjects and the score is different from each other.

Further, in consideration of tracking all the subjects in the captured image, the area setting unit 1005 selects one or more of the above-mentioned detection target areas from a plurality of candidate areas so as to cover all the subjects at the present time. can do. In such a case, for example, the area setting unit 1005 covers all the detected subjects at least once from the candidate areas having combinations of different subject sets and scores remaining by the above processing. The above detection target area may be selected. Further, from the viewpoint of improving the accuracy of detection, the area setting unit 1005 can select the detection target area so that the total value of the scores of the candidate areas selected for the detection target area becomes large. For that purpose, the optimization method of the set cover problem may be applied, that is, the following conditional optimization may be solved.

In this equation, i is an index relating to the subject and j is an index relating to the candidate region. j is an index attached to a candidate area to be selected, for example, a candidate area having a combination of different sets of subjects and scores remaining by the above processing, and n is a candidate to be selected. Represents the number of candidate areas. s _j indicates the score of the candidate area j. Further, x _{j is x j} = 1 when the candidate area j is selected, and _{x j} = 0 otherwise. Further, a _{ij is a ij} = 1 when the candidate region j covers the subject i, _{and a ij} = 0 when the candidate region j does not cover the subject i.

Such an optimization problem is not particularly limited to the above equation. That is, the area setting unit 1005 may use different formulas as appropriate according to desired conditions. For example, the area setting unit 1005 covers the entire subject at least twice or more by _{setting Σa ij} ≥ 1 in _{the above equation (1) to Σa ij ≥ 2 in consideration of improving the robustness of detection.} A detection target area may be created.

Further, from the viewpoint of reducing the processing cost, the area setting unit 1005 can select the detection target area so that the total number of the detection target areas to be selected is reduced. That is, the above optimization problem can be solved in this way. The method of solving the optimization problem so that the total number of detection target areas is small is not particularly limited. For example, the region setting unit 1005 may select a detection target region by applying a known optimization method such as a greedy method or a Lagrange relaxation method to this problem. Further, for example, the area setting unit 1005 may select the detection target area so that the total number of detection target areas is equal to or less than a predetermined number.

Next, the process proceeds to the processing of the captured image at the next time, and in step S4002, the detection unit 1003 detects the subject from the selected detection target area.

According to such a configuration, a detection device that detects one or more objects from the captured image and sets a detection target area that can be used in detecting the subject at a subsequent time according to the position of the objects. Obtainable. Therefore, it is possible to achieve both calculation cost and detection accuracy for a subject passing through the field of view of a single fixed camera, and to track the subject with higher accuracy at a low calculation cost.

[Embodiment 2]
The detection device according to the second embodiment can predict the position of the subject at the next time and set the detection target area based on the prediction. In particular, the detection device according to the second embodiment can set a detection target area having a buffer width corresponding to the predicted position of the subject and the width of the amount of deviation that can occur from the prediction. Therefore, the detection device according to the present embodiment can perform the detection process at a low processing cost without taking an extra buffer width for the detection target area even when the subject is stopped, for example. For such processing, the detection device 2000 according to the present embodiment has a prediction unit 2001. Further, the detection device 2000 is the same as that of the first embodiment except that the detection device 2000 has the prediction unit 2001, and the duplicate description will be omitted.

FIG. 2B is a block diagram showing an example of the functional configuration of the detection device 2000 according to the second embodiment. The prediction unit 2001 predicts the position of each subject when the next detection is performed. The area setting unit 1005 sets the detection target area in consideration of the position of the subject predicted by the prediction unit 2001.

Hereinafter, the flow of the detection method performed by the detection device 2000 according to the present embodiment will be described with reference to FIG. 3 (b). FIG. 3B is a flowchart showing an example of a processing procedure for performing the detection according to the present embodiment. The processing procedure of the detection device 2000 according to the present embodiment can be performed in the same manner as in the first embodiment except for step S5001 and step S5002.

In step S5001, the prediction unit 2001 predicts the position where each subject is detected at the next time. The method for the prediction unit 2001 to predict the position of the next subject is not particularly limited. In the loop L5001, the subject is associated with the image of the subject having the same identification information detected at the previous time in step S4003. That is, the prediction unit 2001 can acquire the coordinates of each time up to the present time for a specific subject. For example, the prediction unit 2001 calculates the movement distance and the movement direction from the previous detection of the subject to the current detection by taking the difference between the previous position of the subject and the current position, and based on them, the next time. The position of the subject at the time of detection may be predicted. Further, for example, the prediction unit 2001 moves from the previous detection of the subject to the current detection by appropriately using the position of the subject at an arbitrary time before the previous time in addition to the previous position and the current position of the subject. Information about distance and direction of travel can be calculated. According to such processing, the prediction unit 2001 is smoother than the case where the movement distance and the movement direction of the subject from the previous time to the present time are calculated using only the previous position and the current position of the subject. Information can be calculated. Even in such a case, the prediction unit 2001 can predict the position of the next subject from the calculated subject information.

In step S5002, the area setting unit 1005 sets the detection target area to be used for detection in the next step S4002. In this example, the area setting unit 1005 performs the same processing as in step S4004 of the first embodiment except for the movement processing of the detection target area and the buffer width setting method, and thus the duplicate description will be omitted. The area setting unit 1005 can move the position of the detection target area covering the subject based on the position of the next subject predicted in step S5001. In such a case, the method of moving the detection target area is not particularly limited. For example, when the detection target area covers one subject, the area setting unit 1005 may move the detection target area in the same manner as the movement of the subject to the predicted position. Further, when the detection target area covers a plurality of subjects, the area setting unit 1005 may move the detection target area according to the movement of the subject having the highest score among them. The detection target area may be moved according to the average of the predicted movements of the subject. Further, the area setting unit 1005 may appropriately calculate the width of the noise component (the amount of deviation from the true value) generated when the position of the subject is predicted, and set the buffer width of the value of such noise component. .. In such a case, the area setting unit 1005 uses, for example, the noise component to determine the deviation between the predicted position and the detected position of the subject in the same manner as in the method in S5001. It may be calculated as an average of the amounts. That is, the area setting unit 1005 may calculate the value of the noise as the average of the amount of deviation between the predicted position of the subject and the detected position in the loop up to the present time.

According to such a configuration, it is possible to set a detection target area suitable for detection based on the predicted position of the subject. In addition, the buffer area can be set based on the predicted position of the subject. Therefore, it is possible to track a subject passing through the field of view of a single fixed camera at a lower calculation cost.

[Embodiment 3]
The detection device according to the third embodiment detects a subject from each of the captured images obtained by a plurality of cameras, and tracks the subject using the result. At that time, the detection device determines the current state of the subject based on the predicted values of the state (that is, position, posture, and velocity) of the subject in the three-dimensional space estimated from the observed values of the subject in the previous loop. Can be predicted. Further, based on the predicted current state of the subject, the predicted values of the coordinates and the score on the captured image of the subject are further acquired, the detection target area is set based on the coordinates and the score of the subject, and the subject is set. Can be detected. Further, it is possible to set a detection target area that maximizes the detection score of the subject predicted at the next time. In the following, a case where a plurality of fixed cameras are used to image a small-scale indoor pitch of soccer called futsal will be described, but the present invention is not limited to this application. That is, it is assumed that the subjects in the present embodiment are the head of a person and a soccer ball (hereinafter referred to as a ball).

FIG. 8 is a diagram for explaining an embodiment of the detection device 3000 assumed in the present embodiment. The camera arrangement example 800 is a bird's-eye view of the camera arrangement and pitch according to the present embodiment, and is an X-axis, Y-axis, and Z-axis of three-dimensional coordinates with the origins 807 and 807 of the cameras 801 to 806 and the three-dimensional space as the origins. It has 808, 809 and 810, and a pitch 811. In the present embodiment, each camera is fixed to a space wall surface at a certain height from the ground, and may be installed so as to image a subject existing on the pitch. Further, each camera included in the detection device 3000 is given an internal parameter and an external parameter by camera calibration, respectively. Therefore, in the following, it is assumed that the detection device 3000 can obtain the pixel coordinates of the subject from the three-dimensional coordinates of the subject. Since camera calibration is a known technique, detailed description thereof will be omitted. Further, in this example, the plane formed by the X-axis 808 and the Z-axis 810 is the ground, and the Y-axis 809 is the direction representing the height.

The person arrangement example 820 is an example of the arrangement of the person and the ball existing in the same space at a certain time. The pitch 811 of the person arrangement example 820 is the same pitch as the pitch 811 of the camera arrangement example 800. 821 is a halfway line of the same pitch, and 822 is a center mark. A0, A1, A2, A3 and A4 are players (persons) of team A, and B0, B1, B2, B3 and B4 are players (persons) of team B. Further, S0 is a ball.

Image examples 830, 840, 850, 860, 870, and 880 are image examples when the person and ball arrangement of the person arrangement example 820 are imaged by the cameras 801, 802, 803, 804, 805, and 806, respectively. Further, A0, A1, A2, A3, A4, B0, B1, B2, B3, and B4, and S0 in each image example are a person and a ball, and A0, A1, A2, A3, and A4 of the person arrangement example 820. , B0, B1, B2, B3, and B4 and S0, respectively.

FIG. 2C is a block diagram showing an example of the functional configuration of the detection device 3000 according to the third embodiment. The detection device 3000 has an imaging unit 3100 and a processing unit 3200. The imaging unit 3100 has a total of K moving image acquisition units, including a first moving image acquisition unit 3001, a Kth moving image acquisition unit 3002, and a moving image acquisition unit omitted in the drawing. For example, in the example of FIG. 8, since the number of cameras is 6, K is 6. Each of these moving image acquisition units according to the present embodiment has the same configuration as the moving image acquisition unit 1001 in the first embodiment. In the example of FIG. 2C, the processing unit 3200 includes an initial value setting unit 3003, a prediction unit 3004, an area setting unit 3005, a detection unit 3006, an association unit 3007, a weight calculation unit 3008, an update unit 3009, and a visualization unit. It has a part 3010.

The initial value setting unit 3003 sets the values of the position, posture, and speed of the subject at the initial time of the detection process. The prediction unit 3004 predicts the position, posture, and speed of each subject in the three-dimensional space, and predicts the observed value of the subject for each camera. Although detailed description will be described later, the observed value is the position of the subject on the pixel coordinates and the detection score. The area setting unit 3005 sets the detection target area at the present time. The detection unit 3006 detects a subject in the detection target area in the image from the detection target area set by the area setting unit 3005 and the image acquired by each camera, and acquires the position and score of the subject. The association unit 3007 associates the image of the previous subject with the image of the current subject. The weight calculation unit 3008 calculates the weight of each observed value of each camera. The update unit 3009 updates the state, that is, the position, the posture, and the speed of the subject by using the observed value in the previous loop and the weight of the observed value for each subject. The visualization unit 3010 visualizes the locus of the position of each subject at the time of detection. Details of the processing performed by these functional units of the processing unit 3200 will be described later together with the flowchart of FIG. 3C. The processing unit 3200 may be connected to the monitoring unit 1300 in the same manner as the processing unit 1200 of the first embodiment.

The detection device according to the present embodiment can estimate the state of the subject from the observed value of the subject. In explaining the framework for tracking a subject in a three-dimensional space, the observed value of the subject detected by the detection device 3000 and the state variable of the subject estimated from the observed value will be described. The observed value of the subject by the detection device 3000 is the position (u, v) and the score (q) of the subject on the pixel coordinates of the captured image, and is represented by a total of three dimensions (u, v, q). You may. The position (u, v) of the subject is the position of the center of the bounding box surrounding the subject, and can be calculated by the detection device 3000 from the coordinate information of the bounding box and the coordinate information of the detection target area including the bounding box. .. Further, the value of the score (q) for the same subject may differ depending on the size of the detection target area including the subject.

The state variable of the subject is a variable representing the state of the subject in the three-dimensional space, that is, the position, posture, and speed. That is, by estimating this state variable, the detection device can estimate the position of the subject in the three-dimensional space and track the subject. The detection device 3000 according to the present embodiment can estimate the state variable of the subject from the observed value of the subject.

The prediction unit 3004 can acquire the current state variable and the predicted distribution of the observed value from the state variable of the subject in the previous loop. In this embodiment, since the subject is the head or the ball, the state variables for each are considered. The state variables of the head are a total of 9-dimensional variables such as the position (x, y, z), posture (φ, θ, ψ), and velocity (x', y', z') of the subject in the three-dimensional space. Given as. Further, the state variables of the ball are the position (x, y, z) and velocity (x, y, z) and velocity (x, y, z) of the subject in the three-dimensional space because the ball is spherical and the shape seen from the camera does not change even if the posture of the ball changes. It is given as a total of 6-dimensional variables of x', y', z'). That is, the observed value y, the state variable x ^{head of the head} , and the state variable x ^ball of the ball can be described by the following equations.

In the above equation, the subscript t represents the time. Further, k _sj represents an observed value of the j-th detection target region in the detection target region of size s in the image captured by the camera k. T is transpose. Further, the subscript n represents a person, and in the present embodiment, it may be a value of identification information such as an ID of the person.

Further, by the process of step S6006 described later, the _{association between n and k sj} is performed. As a result, y _{t, ksj} becomes y _{t, k, s, n} = [ _{ut, k, s, n} , v _{t, k, s, n} ,, q _{t, k, s, n,} ] ^T. Associated. Here, y _{t, k, s, and n} represent the observed values of the subject n at time t in the detection target region of size s in the image captured by the camera k. In the present embodiment, by using the state space model having the above-mentioned observed values and state variables, the head and the ball are detected and tracked by using the extended Kalman filter that estimates the state from the observed values. Since the extended Kalman filter is known, detailed description thereof will be omitted.

FIG. 3C is a flowchart showing an example of a processing procedure for performing the detection according to the present embodiment. In the loop L6001, the detection device 3000 can repeat the following operations of steps S6002 to S6009 in order from time 1 to t, and proceed to the next time. In step S6001, the initial value setting unit 3003 acquires the initial state of the subject at the start time (t = 1). At the start time, the velocity and posture in the state variable of the subject can be set to 0. Further, the position in the state variable of the subject may be a value close to the value of the correct position on the three-dimensional coordinates of the subject in consideration of associating the observed value of a plurality of subjects in the detection target area with the subject. Good.

Hereinafter, a method of acquiring a value close to the value (x, y, z) of the correct position on the three-dimensional coordinates of the subject will be described. In step S6001, the initial value setting unit 3003 detects the subject from the captured image of each camera and acquires the position (u, v) of the subject on the pixel coordinates of each camera. Next, the initial value setting unit 3003 assumes a value y in the height direction of the subject according to the type of the subject. Since an accurate value of the height y of the subject is not necessary for associating the observed value with the subject, the initial value setting unit 3003 may assume the height of the subject as a rough value. For example, the initial value setting unit 3003 may assume that the height of the head is 1.5 m and the height of the ball is 0.1 m. In the following, for the sake of explanation, the height of the head is assumed to be 1.5 m and the height of the ball is 0.1 m, but the height of the subject is not limited to that. Absent. Next, the initial value setting unit 3003 uses a perspective projection matrix to obtain a position (x, 1.5, z) or (x, 0.) of the subject in the three-dimensional space from such (u, v). 1, z) is acquired.

Further, the initial value setting unit 3003 associates the images of the same subject with the positions on the three-dimensional coordinates of each subject acquired by all the cameras. The initial value setting unit 3003 clusters the acquired positions of each subject on the three-dimensional coordinates by a method such as a known k-means method, and sets the subjects included in each clustered cluster. It may be the same subject. In such a case, the initial value setting unit 3003 may acquire the initial position (x, y, z) of each subject by averaging the values of the positions included in each cluster.

By such a process, the initial value setting unit 3003 _{acquires the initial value x ball 0, n} = (x _{0, n} , 0.1, z _{0, n} , 0, 0, 0) of ^{the state variable of the ball.} be able to. Further, the initial value setting unit 3003 uses the initial value of the state variable of the ^head _{x head 0, n} = (x _{0, n} , 1.5, z _{0, n, 0, 0, 0, 0, 0, 0} ). Can also be obtained. These initial values can be the first-order moment (mean) x _{0 | 0, n} of the initial filter distribution (posterior distribution) of the state variable. In such a case, the quadratic moment (variance-covariance matrix) of the initial filter distribution of the state variable may be a semi-normal definite matrix of an appropriate size.

In step S6002, the prediction unit 3004 acquires the state variables of each subject and the prediction distribution of the observed values. The prediction unit 3004 can acquire the prediction distribution for the subject, which is the head, by using, for example, the following system equation (3). In this equation, Δt represents the time width (seconds) from the previous time to the present time in L6001. Further, _st is white Gaussian noise called process noise (that is, noise generated during a prediction process, for example). Q _t is the variance of the inverse covariance matrix of _{S t.} In the present embodiment, the prediction unit will be described as acquiring the prediction distribution of the state variables of the subject using the equation (3), but the method is not particularly limited. In this system equation, the change in the position of the subject (x, y, z) is treated as a trend component model of the position and velocity of the subject modeled in the second-order Markov process. Further, the posture (φ, θ, ψ) is modeled as a first-order Markov process of the posture of the subject.

Further, as the system equation when the subject is a ball, the following system equation (4) is used, which ignores the dimensions related to the posture (φ, θ, ψ) from the equation (3). In the following, for the sake of simplicity, x _{head tun} and x ^ball _tn will be referred to as _{x tn} ^{unless it is necessary to clearly distinguish the head from the ball.}

The prediction unit 3004 can acquire the predicted distribution of the observed values of the subject by using the observation equations (6) and (7), which will be described later, and (8) or (8'). The observation equations (6) and (7) are derived based on the following equation (5). Equation (5) is an equation that projects a point in three-dimensional space onto the pixel coordinates of the camera. As described above, since the internal parameters and external parameters of the camera included in the detection device 3000 are acquired in advance, the detection device 3000 can project a point in the three-dimensional space on the pixel coordinates. Such a projection can be described by the following equation (5). Here, _{pxx and k} are each element of the perspective projection matrix in the camera k. γ is a parameter of the simultaneous coordinate system.

The prediction unit 3004 can derive the following observation equations (6) and (7) as described above based on the equation (5). From these observation equations, the prediction unit 3004 can calculate the position (u, v) which is the observed value of the subject from the position (x, y, z) in the three-dimensional space of the subject. w _t is a white Gaussian noise, called the observation noise.

The equations (6) and (7) are equations that model the process in which the position (x, y, z) of the subject in the three-dimensional space is observed as the pixel coordinates (u, v). In the present embodiment, the position of the subject acquired by each of the cameras is deviated due to the asynchronousness of the plurality of cameras, and the position of the subject is deviated due to frame dropping of some cameras. In addition, the deviation of the position of the subject that occurs in the processing process of the detection device 3000 and the deviation of the position of the subject due to the error of the camera calibration also occur. w _t is considered to be observed in the detector 3000 by these factors, it is obtained by modeling the deviation of position of the object in a three-dimensional space.

Further, the following observation equations (8) and (8') are observation equations for the score of the subject, and correspond to the case where the subject is the head and the case where the subject is a ball, respectively.

Here, C _k represents the position of the camera k in the three-dimensional space. Further, || x−C || ₂ represents the Euclidean distance between the subject and the camera. α _s ⁽⁰⁾ , α _s ⁽¹⁾ , α _s ⁽²⁾ , α _s ⁽³⁾ , and α _s ⁽⁴⁾ are model parameters. θ _x , θ _y , and θ _z are the matrices when the rotation matrix of the external parameters of the camera is R and the rotation matrix obtained from the posture of the head (φ, θ, ψ) is Ro (the following equation). It can be expressed using the elements of (9)). For example, in this case, θ _x can be expressed as asin (r ₃₂ ), θ _y can be expressed as atan (−r ₃₁ / r ₃₃ ), and θ _z can be expressed as atan (r ₂₁ / r _11).

Equations (8) and (8') are multiple regression models that model the process by which the detection device 3000 observes the score of the subject. A detection device that detects a predetermined subject can generally change a score to be output based on the size and posture of the captured subject, and can detect the subject according to such a score. In general, the size of the subject in the captured image often correlates with the distance between the camera and the subject. Therefore, the detection device 3000 may change the score of the subject according to the distance between the camera and the subject. In addition, the detection device that does not bias the learning data for detection robustly acquires and detects image features such as textures for the subject when the subject to be detected is projected large in the captured image. The score of is also high. In addition, especially when the subject to be detected is a person, when the person is facing the front of the camera, the visibility of important parts related to identification such as eyes, nose, and mouth is stable, so the detection score. Tends to be high. Conversely, when the person is facing in the opposite direction to the camera, such parts, which are clues to identification, are less visible and tend to have lower detection scores. On the other hand, when the ball is the subject, the shape of the ball does not change due to the change in posture, so that the detection score may change only according to the distance between the subject and the camera.

The first term of equation (8) is a constant term. The second term of the equation (8) is a term expressing the relationship between the distance from the camera to the subject and the score of the subject. The third, fourth, and fifth terms of the equation (8) are terms in which the relationship between the posture of the head seen from the camera and the score of the subject is modeled by the cosine function. Further, the sixth term is a noise term. Change in the score of the object that these elements and factors, the detection since it is considered that the target area varies according to the size, model parameters alpha _{s ⁽⁰⁾} of formula (8) and (8 ') ~ _.alpha.s ^{( 4)} may take different values based on the size of the detection target area. Further, since the change in posture is not involved in the detection of the ball for the above-mentioned reason, the equation (3') may be modeled as the observation equation of the score when the ball is detected.

The prediction unit 3004 can estimate the model parameters α _s ⁽⁰⁾ , α _s ⁽¹⁾ , α _s ⁽²⁾ , α _s ⁽³⁾ , and α _s ⁽⁴⁾ . The method of this estimation is not particularly limited. For example, the prediction unit 3004 can give a correct answer value of the orientation in the three-dimensional space to each of a plurality of heads in the captured image, and can acquire a score for each of the heads. Next, the prediction unit 3004 may estimate the parameters by performing the least squares method using a plurality of head samples having such orientation information and scores. The least squares method is a method of finding a plausible function representing the relationship between x and y when a plurality of data sets (x, y) are given, but since it is a known technique, detailed description thereof will be omitted. .. For example, the prediction unit 3004 may calculate the likelihood of the model with respect to the observed value of the subject by using the likelihood function of the equation (16) described later. In such a case, the prediction unit 3004 calculates the log-likelihood from a large number of observed values using the equation (16), and uses a known parameter search method such as a grid search or a Bayesian optimization method. , Model parameters that maximize log-likelihood can be estimated. For example, by estimating the model parameters using the above method using the likelihood function, the prediction unit 3004 can efficiently estimate the model parameters without the need to give the correct answer value by the user input. .. Further, the method of estimating the model parameters is not limited to these, and is performed by, for example, a recursive search method using the EM method, or a method of making a self-organizing model in which the model parameters are also incorporated into the state space. May be good. Since the above method is a known technique, detailed description thereof will be omitted.

In the following, the above equations (6), (7), (8) and (8') will be summarized and expressed as the following equation (10). Here, the variance-covariance matrix of the observation noise w _t is assumed to be R. Further, the likelihood function P ( _{yt, kj, s} | _{xt, n} ) is obtained from this equation (10).
y _{t, k, j, s} = _{ht, k, s} (x _{t, n} ) + W _t equation (10)

According to the following equations (11) to (14) using the above system equations and observation equations, the prediction unit 3004 starts from the state one hour before the subject (head) n (time t-1) to the present state of the subject. The state and observed value at (time t) can be predicted. Here, x _{t | t-1, n} and V _{t | t-1, n} represent the first moment and the second moment of the predicted distribution of the state variables, respectively. Further, y _{t | t-1, k, s, n} and U _{t | t-1, k, s, n} represent the first moment and the second moment of the predicted distribution of the observed values, respectively. Further, Q _t represents the variance-covariance matrix of the process noise, and R _t represents the variance-covariance matrix of the observed noise w _t. H _{t, k, s} is a Jacobian matrix of ht _{, k, s} (x _{t, n).}

In the following, for the sake of simplicity, the predicted distributions of state variables and observed values according to the Gaussian distribution having the first and second moments shown in the above equations (11) to (14) are set to P ( _{xt, n} | Y). _{It is expressed} as t-1) and P (y _{t, k, s, n} | Y _t-1 ). Here, Y _t-1 is a set of observed values of the subject up to the time t-1. Further, y _{t, k, s, and n} are observed values of the subject n in the detection target region of size s in the image captured by the camera k at time t. When the time t = 1, the observed value and the state variable of the subject are assumed to be initial values.

In step S6003, the area setting unit 3005 sets the detection target area used for detecting the subject in step S6005 described later. FIG. 9 is a flowchart showing an example of a processing procedure for setting the detection target area in step S6003.

In step S7001, the area setting unit 3005 acquires the pixel coordinates of the second candidate area created in the previous loop, which has the subject. The second candidate area used in step S7001 is a candidate for selecting a detection target area in step S7004 described later, and one different second candidate area is assigned to each subject in step S7003. Is created in. A subject to which the second candidate region is assigned is referred to as a representative covering element in the second candidate region, and another subject possessed by the second candidate region is referred to as a non-representative covering element. Further, the area setting unit 3005 can move the second candidate area assigned to the subject based on the current predicted position of the subject which is the representative covering element. The current pixel coordinates of each subject are predicted in step S6002 (that is, the first moment of the predicted distribution of the observed values (Equation (6))). For example, the area setting unit 3005 may move the second candidate region in the same manner as the movement of the representative covering element from the previous position to the current position, or the center coordinate of the second candidate region is the representative covering element. The second candidate region may be moved so as to match the predicted position. Further, when the representative covering element is not assigned to the second candidate region, the region setting unit 3005 does not have to move such a second candidate region. Further, when the representative covering element comes out of the field of view of all the cameras, the area setting unit 3005 may delete the corresponding second candidate area. When the time t = 1, since the second candidate region does not exist, the process moves to step S7002.

In B7001, the area setting unit 3005 checks the covering of all subjects. For example, the area setting unit 3005 determines whether or not all the subjects are covered with any of the second candidate areas based on the second candidate area created in the previous loop and the position of the subject predicted in S6002. Can be determined. When all the subjects are not covered by the second candidate area, the second candidate area can be assigned. In addition, the area setting unit 3005 can determine whether or not there is a subject to which the second candidate area is not assigned. The area setting unit 3005 may determine whether or not there is a subject newly moved within the field of view of any of the cameras since the previous detection. When there is a subject to which the second candidate area is not assigned, the second candidate area can be assigned. Also, when the time t = 1, the second candidate area can be assigned to the subject. When allocating the second candidate area, the process proceeds to step S7002. If not, the process proceeds to step S7004.

In step S7002, the area setting unit 3005 is set to cover all the subjects existing in each captured image from the set of candidate areas (created in the same manner as in step S4004 of the first embodiment) for each camera. Select the first candidate area of. The area setting unit 3005 may remove the area that does not cover the subject from the candidate area, as in step S4004, for example. Next, in step S7003, the area setting unit 3005 assigns at least one different area to all the subjects from the set of the first candidate areas selected for all the cameras. To select. For that purpose, for example, the area setting unit 3005 can select the second candidate area by solving the following integer programming problem (Equation (15)). Here, i is the index of the subject, and m is the total number of subjects. Further, j is the index of the candidate area, and s _j is the score of the candidate area. x _{j is x j} = 1 if the candidate region is selected, and 0 otherwise. Further, a _ij is 1 when the candidate region j covers the subject i, and 0 when the candidate region j does not cover the subject i. At this time, the area setting unit 3005 determines the predicted value of the highest score among the scores of the subjects possessed by the area from the predicted value of the detection score of each subject, as in the example of FIG. 7 of the first embodiment. It can be used as a score for that area. The area setting unit 3005 can make the above allocation for the equation (15) by using the greedy method, the Hungarian method, or the like. In this way, the second candidate area can be assigned to the subject, and the subject to which a certain second candidate area is assigned is treated as a representative covering element for the second candidate area.

In step S7004, the area setting unit 3005 selects a detection target area to be used in step S6005, which will be described later, from the second candidate area. The area setting unit 3005 may obtain the detection target area by solving the equation (1) in the first embodiment, for example, using such a second candidate area as a candidate area.

FIG. 10 is an example similar to that of FIG. 8, and is a diagram for explaining a second candidate region and a detection target region using an image of six viewpoints captured by the detection device 3000 assumed in the present embodiment. .. The image of each viewpoint is the same as that of the viewpoint in FIG. 8 with the same reference number. Each image shows a subject (A1-4, B1-4 and C0) similar to each image in FIG. Image examples 1400, 1410, 1420, 1430, 1440, 1450, and 1460 in FIG. 10 are examples of images captured by cameras (viewpoints) 801, 802, 803, 804, 805, and 806, respectively. In FIG. 10, the region 1401 is the second candidate region C1 and is finally selected as the detection target region. Further, the regions 1411 and 1412 are the second candidate regions C2 and C3, respectively, and finally, C2 is not selected as the detection target region, but C3 is selected as the detection target region. The region 1421 is the second candidate region C4, and is finally selected as the detection target region. Regions 1431, 1432, 1433 and 1434 are the second candidate regions C5, C6, C7 and C8, respectively, and finally C5 and C6 are not selected as detection target regions, but C7 and C8 are detection target regions. Be selected. Regions 1441, 1442 and 1443 are second candidate regions C9, C10 and C11, respectively, and finally C9, C10 and C11 are selected as detection target regions. The second candidate region does not exist in the image example 1450.

An example of a table listing the second candidate regions shown in FIG. 10 is shown in FIG. In FIG. 11, as described above, the second candidate area is assigned to all the subjects (11 in this example) one by one. The number of the second candidate regions is not particularly limited as long as there are different second candidate regions allocated to each subject. For example, there may be two different second candidate regions for each subject, that is, a total of 22 second candidate regions may exist in this example.

FIG. 12 shows an example of a table listing the detection target regions selected based on the equation (1) from the second candidate region shown in FIG. In step S7004, the area setting unit 3005 executes the conditional optimization of the equation (1) based on the score of the second candidate area, so that the total score is maximized, as shown in FIG. The detection target area can be selected. In this example, eight detection target areas are selected, and the calculation cost is reduced as compared with the case where one detection target area is set for each subject.

According to such a process, the second candidate area and the detection target area for performing the detection in the current loop can be set. Since the second candidate area selected in step S7003 is also used in step S7004 in the next loop, the area setting unit 3005 may store the second candidate area in a storage device (not shown). Further, when the second candidate area is not allocated in B7001, the second candidate area moved in step S7001 may be stored in the storage device. The storage device here may exist inside the detection device 3000, or may exist outside. Further, the detection device 3000 may store in the storage device via a USB cable, may store via an SD card or the like, or may store via wireless communication.

In step S6004, the K moving image acquisition units of the imaging unit 3100 acquire captured images at a certain time. The imaging of the cameras included in these K moving image acquisition units may be controlled in any way. For example, the shutters of K cameras may be imaged in a cycle synchronized by an electrical signal such as a trigger pulse or a synchronization signal, or by a clock of a microcontroller inside the camera, respectively, by an autonomous cycle. May be done. Further, the number of K cameras that capture images at the same time is not particularly limited. For example, half of the K cameras may simultaneously take an image, and then the other half of the cameras may simultaneously take an image. Further, the means for connecting the imaging unit 3100 and the processing unit 3200 is not particularly limited. The imaging unit 3100 and the processing unit 3200 may be connected via a communication path such as a local area network, or may be connected by wire via a USB cable or the like. For example, the imaging unit 3100 may store the output captured image in a storage device (not shown), and the processing unit 3200 may acquire a predetermined frame from the storage device.

In the present embodiment, for the sake of explanation, it is assumed that the imaging unit and the processing unit are connected via a communication path. According to such a configuration, the imaging frame acquired and transmitted by the imaging unit 3100 and received by the processing unit 3200 is dropped due to the performance of a relay unit such as a switching hub existing in the network path or the limitation of the band. Can occur. From such a viewpoint, the processing apparatus according to the present embodiment may buffer the frames acquired by the imaging unit 3100 at all times. In such a case, the frame acquired at that time when a frame drop occurs may be the same as the frame acquired at the previous time.

In step S6005, the detection unit 3006 detects the subject from the detection target area set in step S6003 among the captured images acquired in step S6004. In this embodiment, a detection device having the same configuration as that used in the first embodiment is used. Further, in this example, since the subject is the head of a person or a soccer ball, a detection device learned to detect the head and the ball may be used.

In step S6006, the association unit 3007 associates the observed value of the subject obtained from the captured image at time t with the subject in the three-dimensional space in each camera. _{At time t, the association unit 3007 has J observation values {y t, ks 1} , y _{t, ks 2} ... y _{t, ksJ including} false detections in the detection target area of the size s in the image captured by the camera k. } Can be obtained. At this time, from the first and second moments of the predicted distribution of the observed values acquired by the prediction unit 3004 by the equations (13) and (14), the following Gaussian distribution (formula) is applied to an arbitrary j-th observed value. (16)) is described. _{By giving the observed values y t and ksj} as factors to this function, the association unit 3007 can calculate _{the likelihood l ksj and n} as the observed values of the subject n. For example, the associating unit 3007 _{applies the equation (16) to each of the plurality of observed values {y t, ks1} , y _{t, ks 2} ... y _{t, ksJ}, and sets the} observed value having a high likelihood as the subject n. By associating the observed values with each other, the observed values can be associated with the subject. When the time t is 1, that is, when it is the first loop, identification information is assigned to each of the detected subjects.
l _{ksj, n} = N (y _{t, ksj} ; y _{t | t-1, k, s, n} , U _{t | t-1, k, s, n} ) Equation (16)

The method of associating in step S6006 is not particularly limited. For example, the association unit 3007 can assign the observation value having the maximum likelihood among the plurality of observation values of the subject as the observation value of the subject based on the greedy algorithm. Further, for example, the association unit 3007 may associate the subject with the observed value so that the sum of the likelihoods of the observed values of each subject is maximized by the linear programming method. In such a case, for example, using the Mahalanobis distance calculated based on the observed value and the first and second moments of the predicted distribution, the association that minimizes the sum of the Mahalanobis distances should be calculated by the Hungarian method. Then, the correspondence that maximizes the sum of the likelihoods can be obtained.

In step S6007, the weight calculation unit 3008 calculates the weight of each observed value at time t. For example, when the subject is concealed by another subject, the weight calculation unit 3008 can calculate the weight of the concealed subject to be low. In the present embodiment, the association unit 3007 may predict and quantify the probability that such concealment will occur, that is, the predicted concealment rate. Further, the association unit 3007 quantifies such a predicted concealment rate based on the similarity of the predicted distribution of the observed values between the subject and the other subject and the context of the subject and the other subject with respect to the camera. be able to.

Hereinafter, a lightweight quantification method of the predicted concealment rate using only the first moment of the predicted distribution of the observed value of the subject according to the present embodiment will be described. This calculation process is not particularly limited, but in this example, the similarity of the predicted distribution of the observed values is expressed by using the cosine similarity. That is, the weight calculation unit 3008 determines the degree of similarity between the subject n and the subject m as cos ^β (y _{t | t-1, k, s, m} , y _{t | t-1, k, s, n} ). Can be expressed as. Here, β is a predetermined exponent given in advance.

Further, the weight calculation unit 3008 can calculate the anteroposterior relationship between the subject n and the subject m with respect to the camera by the following equation (17). Here, C _k is the position of the camera k in the three-dimensional space. That is, the function of the equation (17) is a function that returns 1 when the subject m is closer to the subject n when viewed from the camera k, and returns 0 otherwise. By using this equation, the weight calculation unit 3008 can calculate the predicted concealment rate ^pocct _{, k, s, n} from the following equation (18).
min (max (|| x _{t, n −} C _k || ₂ − || x _{t, m} −C _k || ₂ , 0), 1) Equation (17)

Here, N _{t, k, and s} are the number of subjects detected in the detection target region of size s in the image captured by the camera k at time t. In the equation (18), when another subject exists in front of the subject with respect to the camera k and the line of sight connecting the subjects from the camera k is similar, the subject is viewed from the camera k. It is based on the idea that it is hidden by the subject. The value obtained by multiplying the equation (17) by the cosine similarity is when the subject m is closer to the camera k than the subject n and the subjects are closer to each other in pixel coordinates. In addition, the value is close to 1. Equation (10) is such a calculation calculated and normalized by one subject to all other subjects. That is, when the ^OCC _{t, k, s, n} is 1, the subject n is completely concealed by another subject, and when the ^OCC _{t, k, s, n} is 0, the subject n Indicates that is not concealed at all.

Over equations (17) and (18), the weight calculation unit 3008 quantified the predicted concealment rate by using only the first-order moments of the predicted distribution of the observed values, but the method is particularly limited thereto. It's not something. For example, the weight calculation unit 3008 measures the distance between distributions by KL divergence or the like in consideration of up to the second moment of the predicted distribution of the observed value as the similarity of the line of sight from the camera to each subject, and measures the value. By using it, the predicted concealment rate may be quantified. Further, in the present embodiment, the weight calculation unit 3008 quantifies the predicted concealment rate between subjects, but the conditions are not particularly limited. For example, the weight calculation unit 3008 may quantify the predicted concealment rate of the subject and a shield other than the subject, for example, a non-moving shield such as a signboard. KL divergence is a measure of how similar the two probability distributions are, and is defined as the following equation (25).

In step S6008, the update unit 3009 updates the predicted distribution of the state variable of the subject by using the observed value at time t, and acquires the filter distribution of the state variable of the subject. Further, at this time, in the state space model according to the present embodiment, the number of observed values for a specific subject may change due to changes in the number of cameras capable of observing the subject as the subject moves. In view of such a situation, the update unit 3009 updates the predicted distribution of the state variable of the subject as the integrated value by integrating a plurality of observation values output by each camera for the specific subject. May be good. In the following, the filter distribution refers to the filter distribution of the state variable of the subject and is referred to as such.

In the present embodiment, the update unit 3009 can integrate the observed values of the subject in consideration of the predicted concealment rates of ^OCC _{t, k, s, n, for example.} That is, the update unit 3009 attaches a weight corresponding to the predicted concealment rate to the observed value predicted to be concealed, that is, lowers the reflection rate of the state variable in the update, and sets it as another observed value. Can be integrated. Further, for example, the update unit 3009 _{uses the scores of the observed values q t, k, s, and n} to obtain the observed values of the subject that are likely to have suitable conditions for detection such as the distance or orientation with respect to the camera. The rate of reflection in updates can be increased and integrated with other observations. Further, the method in which the update unit 3009 integrates the observed values is not particularly limited. Two policies will be described below for such an integration method.

[Integration method 1]
The update unit 3009 adds (1- ^pocc _{t, k, s, n} ) to the observed noise variance-covariance matrix R _t of the likelihood function P (y _{t, ksj} | x _{t, n) of each camera, for example.} You may multiply by the reciprocal of q _{t, k, s, n.} Subsequently, the update unit 3009 calculates an integrated likelihood function that integrates the observation values of each camera as a joint distribution on the premise that each camera independently acquires the observation values, for example, the following equation (19). ) Can be modeled. Here, Y _{t, KSn, and n} are a set of observed values of the subject observed in a plurality of detection target regions of a plurality of cameras at time t. The variance-covariance matrix of P (y _{t, k, s, n} | x _{t, n} , q _{t, k, s, n} , ^OCc _{t, k, s, n ) is (q t, k,} It can be assumed that _{s, n} · (1- ^pocc _{t, k, s, n} )) ^-1 · R _t. That is, this equation can be modeled so that the smaller the detection score and the higher the predicted concealment rate, the larger the observation noise of the subject.

[Integration method 1-1]
The update unit 3009 can apply the update of a normal extended Kalman filter by calculating the distribution of the product of the likelihood functions using, for example, the following equation (20). That is, the update unit 3009 can estimate the predicted distribution of the state variables from the distribution of the product of the likelihood functions. Here, _Skn is a value of the total number of detection target regions in the image captured by the camera k at a certain time in the subject n. According to this method, the update unit 3009 can estimate the predicted distribution of the state variables by being given the value of the number of cameras used. That is, the update unit 3009 can calculate the equation (20) by calculating the product of a plurality of Gaussian distributions in advance and implementing the product of the Gaussian distributions as a function for the observed values from 1 to a predetermined number. it can.

[Integration method 1-2]
Further, the update unit 3009 can calculate the product of the likelihood functions by using, for example, the following recursive equation (21). According to such a method, the predicted distribution of state variables can be estimated even when the total number of cameras used for detection is unknown, for example.

[Integration method 2]

The update unit 3009 uses, for example, the integrated likelihood function as the product of (1- ^pocc _{t, k, s, n} ) and q _{t, k, s, n} as a mixture ratio, and the likelihood function P (1) of each camera. It may be modeled as the following equation (22) with a mixture distribution of _{y t, ksj} | x _{t, n).} According to this policy, the update unit 3009 can integrate the likelihood distribution in each city cleaning other than the intersection of the lines of sight of a plurality of cameras (the straight line connecting the high-priced center of the camera and the subject).

For example, the update unit 3009 may integrate the observed values by calculating the sum of the weighted Kalman feedbacks using the following equation (23). According to such a method, modeling using the product of the Gaussian distribution is not performed, so that the variance of the distribution does not degenerate even when the number of cameras used for detection is large, for example. That is, the observations of all cameras are not excluded but integrated. As a result, the update unit 3009 can estimate the state variable whose temporal change is smooth.

According to any of these methods, the filter distribution of state variables is corrected for multiple observations and prediction errors by integrating the observations for each camera and updating to reflect such integration. It becomes possible to execute the acquisition of. Further, in all cameras, when the predicted concealment rate ^pocct _{, k, s, n of} all subjects is 1, or when all the observed values are missing, the update unit 3009 has a filter distribution. As a result, the predicted distribution of state variables that have not been updated may be acquired. That is, the following equation (24) may be executed.

In step S6009, the visualization unit 3010 visualizes the estimated positions in the three-dimensional space and the time series of such estimated positions for the subject. That is, the estimated position of the subject is visualized in chronological order. The visualization unit 3010 may visualize the estimated position of the subject according to the time series by drawing it in a virtual three-dimensional space, or as a locus or a point on the captured image acquired by the camera. Visualization may be performed by superimposing the display. In addition, the visualization unit can transmit the result of such visualization to the monitoring unit 1300. Next, it is reflected at the next time, and the prediction unit 3004 acquires the predicted distribution of the state variable and the observed value at the next time using the updated state variable.

According to such a configuration, it is possible to set a detection target region in which each subject is covered with a detection target region in at least one captured image from a plurality of images captured by a plurality of imaging devices. That is, the state of the subject in the three-dimensional space can be predicted based on the predicted value of the state of the subject in the previous loop and the observed value of the subject by the images acquired by the plurality of cameras. Further, based on the predicted current state of the subject, the predicted values of the coordinates and the score on the captured image of the subject are further acquired, the detection target area is set based on the coordinates and the score of the subject, and the subject is set. Can be detected. Further, it is possible to set an area for maximizing the predicted subject detection score at the next time. Therefore, the loci of a plurality of subjects existing in the three-dimensional space, especially the head and the ball in this example, are estimated according to the position and time series while reducing the processing cost and improving the detection accuracy. be able to.

[Embodiment 4]
The detection device according to the fourth embodiment controls the posture of the image pickup device according to the predicted position of the subject, and sets the detection target area based on such a posture control amount. FIG. 13 is a block diagram showing an example of the functional configuration of the detection device according to the fourth embodiment. The detection device 8000 according to the present embodiment can track a subject with reduced processing costs by using a camera capable of pan, tilt, and zoom operations (hereinafter referred to as PTZ operations). Therefore, the detection device 8000 has an imaging unit 8100 and a processing unit 8200. The K moving image acquisition units (for example, 8001 and 8002) included in the imaging unit 8100 and the imaging unit 8100 are of the K units included in the imaging unit 3100 and the imaging unit 3100 in the third embodiment, except that the PTZ operation is possible. Since it is the same as the video acquisition unit, duplicate explanations will be omitted. The PTZ operation is an operation including one or more of a panning operation which is a horizontal orientation control, a tilting operation which is a vertical orientation control, and a zoom operation which enlarges / reduces a subject. That is, the moving image acquisition unit capable of PTZ operation can control the imaging range in the horizontal direction, the vertical direction, or a direction in which these two directions are combined by the PTZ operation. The processing unit 8200 has the same configuration as the processing unit 3200 in the third embodiment except that it has a control unit 8003, and duplicate description will be omitted. The control unit 8003 controls the imaging range of the imaging unit 8100 by controlling each moving image acquisition unit. For example, the control unit 8003 can control the imaging range of the imaging unit 8100 by operating each moving image acquisition unit in PTZ.

FIG. 14 is a flowchart showing an example of a processing procedure for performing the detection according to the present embodiment. The processing procedure of the detection device 8000 according to the present embodiment can be performed in the same manner as in the third embodiment except for steps S9001, S9003, and S9004.

In step S9001, the initial value setting unit 3003 sets the values of the position, posture, and speed of the subject at the initial time of the detection process, and initializes and sets each control parameter of the camera as the initial value. Since the setting of the position, posture, and speed value of the subject is the same as that in the third embodiment, the description thereof will be omitted. In this example, the pan angle, tilt angle, and zoom amount states at time t of the camera are represented as _{P t} , T _t , and Z _t. Further, each movable range of the PTZ operation is expressed as P _min ≤ P _t ≤ P _max , T _min ≤ T _t ≤ T _max , and Z _min ≤ Z _t ≤ Z _max . Further, the control amount of the imaging range controlled by the PTZ operation (hereinafter, this is referred to as a PTZ control amount) is expressed as _{ΔP t} , ΔT _t , and ΔZ _{t, respectively.} The PTZ control at time t, the range of controllable PTZ control amount at 1 _{time, Δ min P t ≦ ΔP t} ≦ Δ max P t, Δ min T t ≦ ΔT t ≦ Δ max T t, and delta _min Z _It is expressed as t ≤ ΔZ _t ≤ Δ _max Z _t. The values related to the control of such PTZ may be the same among a plurality of cameras, or may differ depending on the position and type of the cameras. For example, at least one of the plurality of cameras may capture the entire pitch range regardless of the movement of the subject. According to such a configuration, when there is a subject that cannot be tracked temporarily due to, for example, a malfunction of the detection device, the subject is tracked based on the detection result from the image that captures the entire pitch range. Will be easier to restart.

Further, in step S9001, the initial value setting unit 3003 may set the values of P _t , T _t , and Z _t to 0, respectively. However, _{the values of P t} , T _t , and Z _t set here are not particularly limited and may be appropriately set according to the initial state of the camera.

In step S9003, the area setting unit 3005 sets the detection target area used for detecting the subject in step S6006 in consideration of the control amount of the PTZ operation of the camera. Further, the area setting unit 3005 acquires the PTZ control amount required for imaging such a detection target area. The detailed processing procedure in step S9003 will be described later together with the flowchart of FIG.

In step S9004, the control unit 8003 acquires the imaging range of each camera of the imaging unit 8100 based on the PTZ control amount acquired in step S9003. In this example, the control unit 8003 controls the imaging range of _{each camera based on the PTZ control amounts ΔP t} , ΔT _t , and ΔZ _{t estimated at time t.}

Hereinafter, the setting process performed by the area setting unit 3005 in step S9003 will be described with reference to FIG. FIG. 15 is a flowchart showing an example of a processing procedure for making the setting according to step S9003. Since the processes after steps S1502, S1503 and S1506 are the same as the processes after steps S7002, S7003 and S7004 in FIG. 9 of the third embodiment, the description thereof will be omitted.

In step S1501, the region setting unit 3005 moves the second candidate region set in the previous loop based on the position of the representative covering element at the present time, as in step S7001 in the third embodiment. At this time, the area setting unit 3005 may move the second candidate area in consideration of not only the position of the representative covering element at the present time but also the control range of the PTZ control amount of the camera, for example. That is, for each camera, the area setting unit 3005 sets _{the values of the vertical control amount of Δ max} P _{t and} the horizontal control amount of Δ _max T _{t in} the vertical and horizontal directions in addition to the range of the captured image. The range added to the top, bottom, left and right of the captured image may be calculated. Next, the area setting unit 3005 may move the second candidate area within the range calculated so as for the captured image of each camera. For example, when the second candidate region moves beyond the original imaging range due to such movement, in a later step S9004, the control unit 8003 takes an image by PTZ control according to the moved position of the second candidate region. You can move the range. That is, the area setting unit 3005 can acquire such a PTZ control amount. When a plurality of second candidate regions move beyond the original imaging range, the area setting unit 3005 may acquire the PTZ control amount so that the camera can image all the second candidate regions after such movement. Good. Further, when the camera cannot capture the entire second candidate region after movement even by the PTZ control, the region setting unit 3005 sets the priority according to the score of the second candidate region, and has a higher priority. The control amount may be acquired so that the two candidate regions are imaged. In such a case, the area setting unit 3005 may move the second candidate region having a low priority so as not to go out of the imaging range according to the movement of the visual field edge. Further, the region setting unit 3005 does not have to move the second candidate region in which the representative covering element does not exist.

Since the processing in B1501 is basically the same as the processing in B7001 of the third embodiment, only the different parts will be described. When there is a subject that has not been detected last time in the second candidate area, the area setting unit 3005 can _{return the states P t} , T _t , and Z _t of the operation amount of the PTZ at that time to the initial values. it can. In such a case, the initialization of the value of _{Z t} may be performed after returning _{P t} and T _{t to the initial values.}

In step S1504, the area setting unit 3005 estimates the zoom amount that maximizes the score of the second candidate area. The area setting unit 3005 uses, for example, a polynomial regression model that estimates the detection score using the distance between the subject and the camera as an explanatory variable, so that the zoom control amount ΔZt _max that maximizes the detection score is set by the zoom operation. It may be estimated within a controllable range. The region setting unit 3005 may learn such a polynomial regression model in the same manner as the regression models of the equations (8) and (8') in the first embodiment, for example. Further, the search method of the zoom control amount that maximizes the score is not particularly limited, and may be performed by a known method such as a grid search. Further, the area setting unit 3005 sets the zoom control amount to 0 when the increase width of the score due to the zoom control amount calculated in this way is smaller than a predetermined threshold value, that is, even if the zoom operation is not performed. Good. According to such processing, the processing cost can be reduced by omitting the zoom operation having a minute effect.

In step S1505, the area setting unit 3005 updates the existing score to the changed score when the detection score is changed by the zoom control of the amount estimated in step S1505.

According to such a configuration, it is possible to control the postures of a plurality of cameras with respect to the predicted position of the subject. Further, the detection target area can be set based on the control amount of such a posture. Therefore, it is possible to provide a detection device capable of efficient tracking of a plurality of subjects while suppressing the cost of detection.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

1001: Video acquisition unit, 1002: Initial value setting unit, 1003: Detection unit, 1004: ID mapping unit, 1005: Area setting unit, 1006: Visualization unit, 1100: Imaging unit, 1200: Processing unit, 1300: Monitoring unit

Claims (19)

A detection means that detects one or more subjects from the captured image,
According to the position of one or more subjects detected from the captured image at the first time by the detecting means, the one or more of the captured images at the second time following the first time referred to by the detecting means. Setting means for setting the detection target area of the subject and
A detection device comprising. The detection device according to claim 1, wherein the setting means selects the detection target area from a plurality of predetermined candidate areas. The detection device according to claim 2, wherein the setting means selects a plurality of candidate regions having different sizes as the detection target region. The detection means outputs a score indicating the reliability of detection of the subject, and outputs a score.
The detection device according to claim 2 or 3, wherein the setting means selects the detection target area from the plurality of candidate areas based on the score of the subject included in the candidate area. The setting means according to any one of claims 1 to 4, wherein the detection target area is selected so that at least one detection target area covers all of the one or more subjects. The detector described. The setting means is characterized in that the detection target area is set so that the detection target area covers the position of the subject at the first time or the predicted position of the subject at the second time. The detection device according to any one of claims 1 to 5. The setting means predicts the position of the subject at the second time based on the position of the subject at the first time and the position of the subject at a time before the first time. The detection device according to claim 6, which is characterized. The detection target area includes a covering determination area and a buffer area set outside the covering determination area.
The detection according to claim 6 or 7, wherein the setting means sets the detection target area so that the cover determination area of the detection target area covers the position of the subject or the predicted position. apparatus. An estimation means for predicting the state of the subject by using the detection result of the subject by the detection means is further provided.
The detection device according to any one of claims 1 to 8, wherein the setting means sets the detection target area according to the predicted state of the subject. The estimation means predicts a score representing the reliability of detection of the subject according to the predicted state of the subject.
The detection device according to claim 9, wherein the setting means further sets the detection target area of the subject according to the predicted score. Further provided with an acquisition means for acquiring a plurality of captured images by acquiring captured images from each of the plurality of imaging devices.
The setting means according to any one of claims 1 to 10, wherein the detection target area of the subject is set for each captured image so that the subject is included in the detection target area in at least one captured image. The detection device according to any one item. The setting means is
A region corresponding to each of the subjects and different from each other including the subject is set in at least one captured image at the first time.
Based on each predicted position of the subject at the second time, the area including the subject corresponding to each of the subjects is moved.
The detection device according to claim 11, wherein at least one of the regions including the subject corresponding to each of the subjects after the movement is selected as the detection target region. The setting means predicts the position of the subject in the three-dimensional space at the second time from each position of the subject detected by the plurality of captured images at the first time.
The eleventh or twelfth aspect of claim 11 or 12, wherein the position of the subject in each of the captured images at the second time is predicted from the predicted position of the subject in the three-dimensional space at the second time. Detection device. The setting means is
For each of the plurality of captured images, one or more regions covering the position of the subject at the first time or the predicted position of the subject at the second time are set.
Claims 11 to 13 are characterized in that the detection target region is selected from the one or more regions of the plurality of captured images so that the subject is included in the detection target region in at least one captured image. The detection device according to any one of the above. Further, a control means for controlling the posture of the image pickup apparatus for capturing the captured image according to the predicted position of the one or more subjects at the second time is further provided.
The detection device according to any one of claims 1 to 14, further comprising setting the detection target area based on the posture control amount of the image pickup device. The detection device according to any one of claims 1 to 15, further comprising a visualization means for visualizing and outputting the position of the subject for each time series. The detection means outputs a score indicating the reliability of detection of the subject, and outputs a score.
The visualization means has the position of the subject and the score at the first time, and the position of the subject and the score at the first time based on the position of the subject and the score at the second time. The detection device according to claim 16, wherein the position of the subject is visualized for each time series by associating the position of the subject with the position of the subject at the time of 2. It is a detection method performed by the detection device.
The process of detecting one or more subjects from the captured image and
According to the position of one or more subjects detected from the captured image at the first time in the detecting step, the captured image at the second time following the first time referred to in the detecting step is described. The process of setting the detection target area of one or more subjects, and
A detection method comprising. A program for causing a computer to function as each means of the detection device according to any one of claims 1 to 17.

Images