JP7685850B2 - Monitoring terminal and monitoring method - Google Patents

Description

The present invention relates to technology for monitoring.

Japan is said to be an aging society, with approximately 30% of the population currently aged 65 or over. As the elderly population increases, the number of elderly people living alone (so-called elderly people living alone) is also on the rise, exceeding 6 million as of 2015. Because these elderly people living alone live alone, it is often difficult for them to take appropriate action when illness, accidents, or other problems occur, and there is a risk that their injuries or illnesses will become more serious. This problem is not limited to elderly people living alone, but can also occur in facilities such as elderly care facilities that are designed for people to live in their own space, and it can also occur for elderly people who are alone for part of the day even if they live with relatives or other people.

The worsening of injuries and illnesses leads to an increase in the number of elderly people requiring nursing care. This causes great losses to society, such as an increase in the nation's social security costs and an increase in the number of people in their prime leaving work to care for elderly people. As such, one of the challenges facing society is that monitoring the socially vulnerable, particularly the elderly, will become increasingly important in the future. In response to this issue, technology has been proposed that monitors the elderly by installing cameras in their living spaces (for example, Patent Document 1).

JP 2013-206012 A

However, constant monitoring of the behavior, facial expressions, and other status of elderly people through video footage may solve the problem from the perspective of injury or illness, but it also gives rise to privacy issues. These issues have not been resolved, and sufficient measures have not been put in place. As a result, there are cases where illness or injury is discovered late, leading to the illness becoming more severe or the person dying alone. These issues are not limited to the elderly, but are common to all those being monitored (hereafter referred to as "targets of monitoring").

In view of the above circumstances, the present invention aims to provide technology that enables surveillance while protecting the privacy of the person being monitored.

One aspect of the present invention is a monitoring terminal that includes a target area estimation unit that estimates target area information indicating an area in which a monitoring target is located from an image of the monitoring target, an abnormality determination unit that determines whether or not an abnormality has occurred in the monitoring target, and a notification unit that generates a notification image that includes a target area image that is an image indicating the target area information but does not include an image of the monitoring target itself when the abnormality determination unit determines that an abnormality has occurred, and transmits the notification image to another device.

One aspect of the present invention is the above-mentioned monitoring terminal, which is installed in a monitored facility where the monitored object is active.

One aspect of the present invention is the above-mentioned monitoring terminal, further comprising a posture estimation unit that estimates the posture of the monitored object, wherein the target area estimation unit estimates the area using a trained model obtained by machine learning using a first learning dataset including teacher images in which the monitored object is photographed and teacher images in which the monitored object is not photographed, and the posture estimation unit estimates the posture using a trained model obtained by machine learning using a second learning dataset in which the proportion of teacher images in which the monitored object is not photographed is relatively smaller than that of the first learning dataset.

In one aspect of the present invention, the monitoring terminal further includes a safety confirmation unit that outputs a voice from an audio output unit prompting the monitored subject to respond as to whether or not the monitored subject is in a safe state, and then generates information on the confirmation result based on the voice input from the audio input unit, and the notification unit also transmits the confirmation result to another device.

One aspect of the present invention is a monitoring method having a target area estimation step of estimating target area information indicating an area in which a monitoring target is located from an image of the monitoring target, an abnormality determination step of determining whether or not an abnormality has occurred in the monitoring target, and a notification step of generating a notification image including a target area image that is an image indicating the target area information but not including an image of the monitoring target itself, and transmitting the notification image to another device, if it is determined in the abnormality determination step that an abnormality has occurred.

The present invention aims to provide technology that allows surveillance to be carried out while protecting the privacy of the person being monitored.

1 is a schematic block diagram showing a system configuration of a monitoring system 100 according to the present invention. 2 is a schematic block diagram showing a specific example of a functional configuration of a monitoring terminal 10. FIG. FIG. 13 is a diagram showing a specific example of a background image. 4A to 4C are diagrams showing specific examples of images captured by the imaging unit 12. 11A to 11C are diagrams illustrating a specific example of processing performed by a target region estimation unit 161. FIG. 13 is a diagram showing a first specific example of a notification image. FIG. 11 is a diagram showing a second specific example of a notification image. 2 is a schematic block diagram showing a specific example of the functional configuration of the control device 20. FIG. 4 is a sequence chart showing a specific example of the flow of operations of the monitoring system 100. 10 is a sequence chart showing another specific example of the flow of operations of the monitoring system 100. 10 is a sequence chart showing another specific example of the flow of operations of the monitoring system 100.

Hereinafter, specific configuration examples of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic block diagram showing a system configuration of a monitoring system 100 of the present invention. The monitoring system 100 is a system for monitoring a monitoring target active in a monitoring target facility 90. The monitoring target facility 90 may be, for example, a residence where the monitoring target lives, an apartment building or facility where multiple monitoring targets live (e.g., a nursing home, etc.), a facility where one or more monitoring targets act (e.g., a nursery school, kindergarten, school), or a facility where one or more monitoring targets live (e.g., a prison, etc.). The monitoring system 100 includes a monitoring terminal 10, a control device 20, and a user terminal 30. The monitoring terminal 10, the control device 20, and the user terminal 30 are communicatively connected via a network 40. The network 40 may be a network using wireless communication or a network using wired communication. The network 40 may be configured by combining multiple networks.

The monitoring terminal 10 is installed in the monitored facility 90. The monitoring terminal 10 captures an image of the monitored object in the monitored facility 90 and converts the image of the monitored object into a target area image. The target area image is an image showing the area in which the whole or part of the monitored object is located (hereinafter referred to as the "target area"). The monitoring terminal 10 generates a notification image including the target area image without including an image of the monitored object. The notification image generated by the monitoring terminal 10 is transmitted to the control device 20 and the user terminal 30. Since the notification image does not include an image of the monitored object, the privacy of the monitored object can be protected. In addition, the notification image includes the target area image. Therefore, the state of the monitored object can be monitored based on the notification image. The user terminal 30 is operated by a person who plans to respond to an abnormality when an abnormality occurs in the monitored object. For example, the user terminal 30 may be used by a relative or acquaintance of the monitored object, by a person who performs the task of monitoring (watching) the monitored object, or by another person. With such a mechanism, the monitoring system 100 makes it possible to monitor the monitored object while protecting its privacy. The details of the monitoring system 100 are described below.

FIG. 2 is a schematic block diagram showing a specific example of the functional configuration of the monitoring terminal 10. The monitoring terminal 10 is configured using information devices such as a smartphone, tablet, personal computer, portable game console, stationary game console, and dedicated device. The monitoring terminal 10 includes a communication unit 11, an imaging unit 12, an audio input unit 13, an audio output unit 14, a display unit 15, a control unit 16, and a memory unit 17.

The communication unit 11 is a communication device. The communication unit 11 may be configured as, for example, a network interface. The communication unit 11 communicates data with other devices via the network 40 in accordance with the control of the control unit 16. The communication unit 11 may be a device that performs wireless communication or a device that performs wired communication.

The imaging unit 12 is configured using a camera. The imaging unit 12 may be configured as a camera itself, or may be configured as an interface for connecting the camera to the monitoring terminal 10 as an external device. The camera captures a predetermined space in the monitored facility 90. The imaging by the camera may be realized as a process of forming an image by receiving visible light, may be realized as a process of forming an image using a radar, or may be realized by other implementation methods. It is desirable that the predetermined space imaged by the camera is a space in which the monitored object may be located. For example, if the monitored object is an elderly person, it may be a space in which the elderly person usually lives (e.g., a living room, a hallway, a bedroom, etc.). For example, if the monitored object is an infant or a small child, it may be a space within the range in which the infant or small child is active (e.g., a living room, a living room, on a baby bed, etc.). One or more cameras may be provided. For example, if the monitored object is an animal, it may be a space within the range in which the animal is active (e.g., a cage, a living room, a hut, a toilet, etc.). The imaging unit 12 outputs data of the image captured by the camera to the control unit 16.

The audio input unit 13 is configured using a microphone. The audio input unit 13 may be configured as a microphone itself, or may be configured as an interface for connecting the microphone to the monitoring terminal 10 as an external device. The microphone acquires audio from a specific space within the monitored facility 90. It is desirable that the specific space from which audio is acquired by the microphone is a space in which a monitored object may be located. For example, it is desirable that the space is the object of imaging by the above-mentioned camera. The audio input unit 13 outputs data of the audio acquired by the microphone to the control unit 16.

The audio output unit 14 is configured using a speaker. The audio output unit 14 may be configured as a speaker itself, or may be configured as an interface for connecting the speaker to the monitoring terminal 10 as an external device. The speaker outputs audio to a specified space within the monitored facility 90. It is desirable that the specified space into which audio is output by the speaker is a space in which a monitored object may be located. For example, it is desirable that the space is the object of imaging by the above-mentioned camera. The audio output unit 14 outputs audio corresponding to the audio signal output by the control unit 16.

The display unit 15 displays an image to the monitored subject in response to control by the control unit 16. The display unit 15 may display, for example, an image to prompt the monitored subject to respond. For example, the display unit 15 may display a character string such as "Are you OK?" or a character string such as "If you are OK, please press the button below" together with an operation object such as a button containing characters or an image indicating that no problem has occurred, such as "OK."

The control unit 16 is configured using a processor such as a CPU (Central Processing Unit) and a memory. The control unit 16 functions as a target area estimation unit 161, a posture estimation unit 162, an abnormality determination unit 163, a notification image generation unit 164, a notification unit 165, a safety confirmation unit 166, a data acquisition unit 167, and an information management unit 168 by the processor executing a program. All or part of the functions of the control unit 16 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., an SSD: Solid State Drive), and storage devices such as a hard disk and a semiconductor storage device built into a computer system. The above program may be transmitted via an electric communication line.

The target area estimation unit 161 estimates information (hereinafter referred to as "target area information") indicating the area (target area) in which all or part of the monitored object is located in the image obtained from the imaging unit 12. The target area may be, for example, a rectangle including the area in which all or part of the monitored object is located. In this case, the target area information may be information indicating the coordinates of the four corners of the rectangle. The target area is not limited to a rectangle and may be other geometric shapes (for example, a circle, an ellipse, a triangle, a square, a pentagon, etc.). The target area may be a shape indicated by multiple straight lines and curves that follow the shape of the monitored object.

The target area estimation unit 161 may perform a process of estimating the target area (hereinafter referred to as "target area estimation") by using a trained model obtained by performing machine learning processing in advance using a teacher image (first training data set), for example. In this case, an image of the whole of a living thing of the same kind as the monitoring target or an image of a part of the living thing of the same kind (for example, only the head, only the upper body including the head, only the upper body not including the head, only the left or right arm, only the lower body, only the legs and feet) may be used as the correct image of the teacher image. In this case, it is desirable that the correct image of the teacher image is given a label indicating that it is correct and information indicating the area where the whole or part of the living thing of the same kind as the monitoring target is located (target area information). However, the outer frame of the teacher image itself may be used as the target area information. In this case, the target area information does not necessarily need to be associated with the teacher image.

In addition, as each correct image, an image of the front, side, or back of the whole or part of a living organism of the same species as the monitoring target may be used. When the attributes of the monitoring target (e.g., age, sex, race, etc.) are determined, an image of a living organism whose attributes match those of the monitoring target may be used as the correct image. By using such correct images, the accuracy of target area estimation can be improved.

Furthermore, an image that does not include either the whole or part of a living organism of the same species as the monitoring target may be used as the incorrect teacher image. For example, an image that does not include an image of a living organism may be used as the incorrect teacher image, or an image of a whole organism of a species different from the monitoring target or an image of a part of a different species of organism may be used. In this case, a label indicating that the incorrect teacher image is incorrect may be given. A first learning dataset including a teacher image of a correct image and a teacher image of an incorrect image may be used to generate a trained model used in the target region estimation unit 161. Note that the technology used to implement the target region estimation unit 161 does not need to be limited to the above-mentioned supervised learning, and may be any technology.

The posture estimation unit 162 estimates posture information of the monitored object in the image obtained from the imaging unit 12. The posture information is information indicating the posture of the monitored object. The posture information may be information selected from a plurality of predetermined posture candidates, for example. Such posture candidates may include information indicating a normal posture and information indicating an abnormal posture. Specific examples of posture candidates include a standing state, a seated state, a floor sitting state, a lying state, a fallen state, a crouched state, a fallen state, and the like. Specific examples of posture candidates may include two states: a fallen state (FALL) and a not fallen state (NOT FALL).

The posture information may be estimated based on, for example, the monitoring target area obtained for the monitoring target. The posture estimation unit 162 may perform a process (hereinafter referred to as "posture estimation") of estimating posture information by using a trained model obtained by performing machine learning processing in advance using a teacher image (second learning data set), for example. In this case, the proportion of images that do not include either the whole or part of a living thing of the same kind as the monitoring target among the teacher images included in the data set of teacher images used in machine learning (second learning data set) may be lower than that of the first learning data set. This proportion may be zero. In other words, the data of the teacher images may be configured not to use images in which the monitoring target is not photographed. Since it is clear that the monitoring target is included in the target area estimated in the processing of the target area estimation unit 161, when posture estimation is performed as a subsequent processing, it is not necessary to use images of a living thing of a different kind from the monitoring target. Rather, by using only images of a living thing of the same kind as the monitoring target, it is possible to improve the accuracy of posture estimation.

As teacher image data, each image may be given information indicating the state of the organism captured in that image as a label. In this case, it is desirable that the content of each label is selected from predefined state candidates. Note that the technology used to implement the posture estimation unit 162 does not need to be limited to processing using supervised learning as described above, and any technology may be used.

The abnormality determination unit 163 determines whether an abnormality occurs in the monitoring target. The abnormality determination unit 163 may determine an abnormality based on, for example, target area information that is the estimation result of the target area estimation unit 161. The abnormality determination unit 163 may determine an abnormality based on, for example, posture information that is the estimation result of the posture estimation unit 162. More specifically, posture information that is likely to have an abnormality (for example, "FALL") and posture information that is likely to have no abnormality (for example, "NOT FALL") are predefined, and whether an abnormality occurs may be determined based on the estimated posture information. The abnormality determination unit 163 may determine an abnormality based on, for example, target area information that is the estimation result of the target area estimation unit 161 and posture information that is the estimation result of the posture estimation unit 162. The abnormality determination unit 163 may determine an abnormality based on, for example, abnormality determination conditions prestored in the storage unit 17. That is, the abnormality determination unit 163 may determine that an abnormality has occurred if the information used for abnormality determination (e.g., target area information, posture information, or other information) satisfies an abnormality determination condition.

As the abnormality determination condition, for example, target area information and posture information indicating that the monitoring target is highly likely to have an injury or a seizure may be registered in advance. As the abnormality determination condition, a condition indicating that the light is on for a predetermined time even during a time period indicating a predetermined late night (for example, from 0:00 to 4:00) may be defined. Whether the light is on or not may be obtained, for example, based on the brightness of the image captured by the imaging unit 12, or based on the output of a brightness sensor (not shown) connected to the monitoring terminal 10, or may be determined by acquiring information indicating the state and power consumption of electrical equipment in the monitored facility 90 from a system such as a HEMS. As the abnormality determination condition, it may be defined that the monitoring target cannot be detected for a predetermined time or more in the video captured by the imaging unit 12. For example, it may be determined that the monitoring target has been detected according to the estimation of the target area information by the target area estimation unit 161, or it may be determined that the monitoring target has been detected when the posture estimation unit 162 can determine the posture, or the monitoring target may be detected using other human body detection algorithms (for example, detection of a human face, detection of a silhouette of a whole image of a human, use of a human body detection sensor, etc.). The condition for determining an anomaly may be defined as the detection of a phenomenon that is significantly different from normal using a trained model obtained by unsupervised learning based on normal captured images.

The notification image generating unit 164 generates a notification image by superimposing an image (target area image) showing the target area information estimated by the target area estimating unit 161 on an image (hereinafter referred to as the "background image") previously captured by the imaging unit 12. FIG. 3 is a diagram showing a specific example of a background image. The background image is an image obtained by capturing an image of the imaging range of the camera used in the imaging unit 12 in advance. The background image does not include an image of the monitoring target itself or an image showing the monitoring target. The background image may be updated at a predetermined timing, for example. For example, an image captured at a predetermined time every day (a time when the monitoring target is generally likely to be inactive) may be used as the background image. For example, an image in which the monitoring target area has not been estimated may be recorded as the background image every time a predetermined time (e.g., 24 hours) has passed. The target area image may be an image showing a frame showing the target area, an image showing the outline of the monitoring target, or an image such as an illustration imitating the shape of the monitoring target.

Fig. 4 is a diagram showing a specific example of an image captured by the imaging unit 12. Fig. 5 is a diagram showing a specific example of a target area estimated by the target area estimation unit 161. In the image in Fig. 4, a person to be monitored is lying on his/her side in front of a door. The target area estimation unit 161 estimates a rectangle 51 surrounding the person to be monitored who is lying in front of the door as the target area.

Figure 6 is a diagram showing a first specific example of a notification image. In the notification image of Figure 6, a frame image (rectangle 51) indicating the target area estimated in the image of Figure 5 is superimposed on the background image of Figure 3.

Figure 7 is a diagram showing a second specific example of a notification image. In the notification image of Figure 7, an illustrated image 52 corresponding to the estimated target area, rather than the estimated target area itself in the image of Figure 5, is superimposed on the background image of Figure 3. A plurality of such illustrated images may be prepared in advance and recorded in the storage unit 17. For example, an illustrated image stored in advance in correspondence with the shape of the estimated target area or estimated posture information may be selected based on these. For example, if the target area estimation unit 161 or posture estimation unit 162 can estimate the orientation of the monitored target, the illustrated image may be selected based further on the estimation result.

As is clear from the examples of notification images in Figures 6 and 7, the notification image shows the position of the monitored subject and also shows the general state of the subject. In this case, it can be seen that the subject is lying on his/her side. At the same time, privacy is protected because the texture and detailed movements of the monitored subject itself are unknown. For example, even if the monitored subject is naked or has a runny nose or drool, such information is not included in the notification image. Also, because a background image captured in advance is used for the notification image, information indicating waste is not included in the notification image even if the monitored subject has accidentally lost control or is vomiting. In this way, information that the monitored subject would not want recorded is not included in the notification image.

When the notification condition is satisfied, the notification unit 165 transmits notification information including a notification image to another device via the communication unit 11. The notification condition may be, for example, that an abnormality is determined by the abnormality determination unit 163. The notification condition may be, for example, that an abnormality is determined by the abnormality determination unit 163 and safety is not confirmed by the safety confirmation unit 166 described below. The notification condition may be defined as other conditions. When the notification unit 165 obtains a confirmation result by the safety confirmation unit 166, it generates notification information including information about the confirmation result. The information about the confirmation result may be, for example, a character string, an image, or identification information indicating the content of the confirmation result. The character string indicating the content of the confirmation result may be, for example, a character string such as "As a result of the safety confirmation, there was a response that the person was OK," a character string such as "As a result of the safety confirmation, there was a response that the person requested help," or a character string such as "As a result of the safety confirmation, there was no response."

When the safety confirmation condition is met, the safety confirmation unit 166 confirms the safety of the monitoring subject by outputting a predetermined sound from the audio output unit 14. The safety confirmation condition may be, for example, that an abnormality is determined by the abnormality determination unit 163. The predetermined sound output by the safety confirmation unit 166 may be a sound that prompts the monitoring subject to respond as to whether or not they are safe. For example, the sound may be "Are you OK?" or "Do you need help?" After outputting the sound, the safety confirmation unit 166 outputs the confirmation result to the notification unit 165.

The safety confirmation unit 166 may generate a confirmation result by, for example, the following process. The safety confirmation unit 166 acquires data of the voice emitted from the monitoring target after outputting the voice from the voice input unit 13. If the safety confirmation unit 166 cannot acquire data of the voice emitted from the monitoring target within a predetermined time, it may output to the notification unit 165 a confirmation result indicating that no answer has been obtained. The safety confirmation unit 166 may perform voice recognition on the acquired voice, and if the content is a predetermined content indicating danger, output a confirmation result indicating danger to the notification unit 165. The safety confirmation unit 166 may perform voice recognition on the acquired voice, and if the content is a predetermined content indicating safety, output a confirmation result indicating safety to the notification unit 165. The safety confirmation unit 166 may perform voice recognition on the acquired voice, and output a character string that is a result of the voice recognition to the notification unit 165 as a confirmation result. In these cases, the notification unit 165 may generate notification information including the confirmation result output by the safety confirmation unit 166.

The data acquisition unit 167 acquires various data regarding the monitored object and the situation within the monitored facility 90. The data acquisition unit 167 may acquire information indicating the shape and installation position of furniture and the like installed in the space within the camera's shooting range, for example, by detecting objects from images captured by the imaging unit 12. The data acquisition unit 167 may acquire information indicating the route (traffic line) of movement of the monitored object, for example, based on time-series images captured by the imaging unit 12. The data acquisition unit 167 periodically transmits the acquired data to the control device 20 via the communication unit 11.

The memory unit 17 is configured using a memory device such as a magnetic hard disk device or a semiconductor memory device. The memory unit 17 stores data used by the control unit 16. The memory unit 17 may store, for example, a terminal ID assigned to the monitoring terminal 10 and attribute information (such as name, address, sex, age, identification information, etc.) of the target of monitoring targeted by the monitoring terminal 10.

Figure 8 is a schematic block diagram showing a specific example of the functional configuration of the control device 20. The control device 20 is configured using information equipment such as a personal computer or a server device. The control device 20 includes a communication unit 21, a storage unit 22, and a control unit 23.

The communication unit 21 is a communication device. The communication unit 21 may be configured as, for example, a network interface. The communication unit 21 communicates data with other devices via the network 40 in accordance with the control of the control unit 23. The communication unit 21 may be a device that performs wireless communication or a device that performs wired communication.

The storage unit 22 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 22 stores data used by the control unit 23. For example, the storage unit 22 may store a destination information table. The destination information table is a table that associates identification information assigned to the monitoring terminal 10 (hereinafter referred to as "monitoring identification information") with information on the destination of user notification information generated based on notification information transmitted from the monitoring terminal 10 (hereinafter referred to as "notification destination information").

The control unit 23 is configured using a processor such as a CPU and a memory. The control unit 23 functions as a notification control unit 231 and a data control unit 232 by the processor executing a program. All or part of the functions of the control unit 23 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs), and storage devices such as hard disks and semiconductor storage devices built into a computer system. The above program may be transmitted via a telecommunications line.

When the notification control unit 231 receives notification information from the monitoring terminal 10, it generates user notification information based on the received notification information. The user notification information includes, for example, a character string indicating that an abnormality has occurred in the monitored object (for example, "An abnormality has occurred. Please check.") and notification address information indicating the memory area in which the notification image is stored. The character string included in the user notification information may be, for example, a fixed character string that is predetermined regardless of the monitoring terminal 10 or the user terminal 30. The character string included in the user notification information may be a character string defined in association with the monitoring terminal 10 or the user terminal 30. These character strings may be stored in the memory unit 22.

The notification address information may be generated by the notification control unit 231, or may be generated by another server that publishes the notification image. For example, the notification control unit 231 may upload the notification image to a storage area accessible from other information devices. In this case, address information (e.g., a URL) indicating the storage area to which the image is uploaded may be used as the notification address information.

The notification control unit 231 determines the notification destination based on the identification information included in the received notification information. The notification control unit 231 may determine, as the notification destination, the notification destination information associated with the monitoring identification information that is the sender of the notification information, based on, for example, a destination information table stored in the storage unit 22. The notification control unit 231 transmits the user notification information to the determined notification destination.

The data control unit 232 registers the data transmitted from the monitoring terminal 10 in the storage unit 22 in association with the monitoring identification information of the monitoring terminal 10 that is the sender. As described above, the data transmitted from the monitoring terminal 10 does not include an image of the monitoring target itself. Therefore, the data registered by the data control unit 232 is unlikely to include information that violates privacy. The data control unit 232 may register the data transmitted from the monitoring terminal 10 in the storage unit 22 in association with the identification information (e.g., terminal ID) of the monitoring terminal 10 that is the sender.

Figure 9 is a sequence chart showing a specific example of the flow of the operation of the monitoring system 100. When the imaging unit 12 of the monitoring terminal 10 captures an image, the control unit 16 acquires the image data captured by the imaging unit 12 (step S101). The target area estimation unit 161 estimates the target area of the monitoring target in the captured image (step S102). The attitude estimation unit 162 estimates the attitude of the monitoring target in the captured image (step S103). The abnormality determination unit 163 performs an abnormality determination (step S104). If an abnormality is not determined, that is, if it is determined to be normal (step S105-NO), the processing of the control unit 16 returns to the processing of step S101. On the other hand, if an abnormality is determined, that is, if it is not determined to be normal (step S105-YES), the notification unit 165 generates a notification image (step S106). The notification unit 165 generates notification information including the notification image and transmits the notification information to the control device 20 (step S107).

When the notification control unit 231 of the control device 20 receives the notification information, it generates user notification information according to the notification information. The notification control unit 231 also determines a destination of the user notification information according to the received notification information. The notification control unit 231 then transmits the user notification information to the user terminal 30 (step S108). The notification control unit 231 also records the notification information in the storage unit 22 (step S109).

When the user terminal 30 receives the user notification information, it outputs an indication that the user notification information has been received. For example, an image or text indicating that the user notification information has been received may be displayed in a specified area of the screen of the user terminal 30, or audio output or activation of a vibrator may be performed according to a notification method preset in the user terminal 30. When a user instructs the output of the user notification information by operating the user terminal 30, the user terminal 30 outputs the contents of the user notification information. For example, the user terminal 30 may display on the screen a character string indicating that an abnormality has occurred in the monitored object and a character string indicating the notification address information (step S110). When a user operates the user terminal 30 to instruct the display of a notification image, the user terminal 30 accesses the notification address information and requests the transmission of the notification image (step S111).

When the control device 20 receives a request to send a notification image to the notification address information, it sends the notification image in response to the request to the user terminal 30 (step S112). When the user terminal 30 receives the notification image, it displays the received notification image on the screen (step S113).

Figure 10 is a sequence chart showing another specific example of the flow of operation of the monitoring system 100. In the sequence chart shown in Figure 10, a safety check is performed by the monitoring terminal 10. Below, only the differences from the sequence chart shown in Figure 9 will be explained. Note that the same processes are denoted by the same symbols in Figures 9 and 10.

When the notification unit 165 notifies the safety confirmation unit 166 of the monitoring terminal 10, the safety confirmation unit 166 outputs a safety confirmation voice via the audio output unit 14 (step S201). The safety confirmation unit 166 accepts the input of the voice uttered thereafter (step S202). The safety confirmation unit 166 generates confirmation notification information based on the input voice. For example, when a voice with a predetermined sound pressure or more is input, it means that the monitoring target has reacted, so confirmation information indicating that the safety has been confirmed may be generated. For example, when a voice with a predetermined sound pressure or more is not input, it means that the monitoring target has not reacted, so confirmation information indicating that the safety has not been confirmed may be generated. The safety confirmation unit 166 transmits the generated confirmation information to the control device 20 (step S203). The notification control unit 231 of the control device 20 transmits the received confirmation information to the user terminal 30 (step S204). When the user terminal 30 receives the confirmation information, it outputs the received confirmation information (step S205). The confirmation information may be output by audio output, by displaying text or an image on a screen, or in other ways. Note that the processes from step S101 to step S113 are the same as those in FIG. 9.

Figure 11 is a sequence chart showing another specific example of the flow of operation of the monitoring system 100. In the sequence chart shown in Figure 11, data generated by the monitoring terminal 10 is accumulated in the control device 20. Below, only the differences from the sequence chart shown in Figure 9 will be explained. Note that the same processes are denoted by the same symbols in Figures 9 and 11.

The data acquisition unit 167 of the monitoring terminal 10 detects a traffic line indicating the route of movement of the monitored object based on the time series data of the image captured by the imaging unit 12, and records it in the storage unit 17 (step S301). If the data acquisition unit 167 does not determine an abnormality in step S104, it discards the information of the traffic line detected before the predetermined threshold. On the other hand, if an abnormality is determined in step S104, the data acquisition unit 167 generates accumulated data including information of the traffic line for a predetermined period recorded in the storage unit 17 (step S302). The predetermined period is, for example, a period of a predetermined number of seconds (e.g., 5 seconds or 10 seconds) going back from the timing when the abnormality was detected. The data acquisition unit 167 transmits the generated accumulated data to the control device 20 (step S303). When the data control unit 232 of the control device 20 receives the accumulated data, it records the received accumulated data in the storage unit 22 (step S304).

In the surveillance system 100 configured in this manner, the notification image used when sending an external notification regarding the surveillance target does not include an image of the surveillance target itself, but includes an image showing the target area. In this way, since the notification image does not include an image of the surveillance target itself, the privacy of the surveillance target can be protected. In addition, since the notification image includes an image showing the position of the surveillance target, the status of the surveillance target can be monitored based on the notification image. With this mechanism, the surveillance system 100 is able to monitor the surveillance target while protecting its privacy.

In addition, in the process of estimating the area in which the monitored object is located (target area), an image in which only a portion of a living organism of the same species as the monitored object is captured is also used as the correct answer teacher image. Therefore, even if part of the monitored object is hidden behind furniture or the like in the captured image, it is possible to estimate the position of the monitored object with higher accuracy based on only the remaining captured portion. For the same reason, it is possible to more accurately estimate the state of the monitored object and determine whether or not there is an abnormality.

In addition, in the monitoring system 100, information indicating the arrangement of furniture and the movement of the monitored subject is stored as accumulated data in the control device 20. This data does not include images of the monitored subject itself. Therefore, it is possible to protect the privacy of the monitored subject even when using this data for analysis. As shown in the sequence chart of FIG. 11, information on the movement is accumulated for a predetermined period of time going back from the time when an abnormality occurred. Therefore, it is possible to efficiently obtain data useful for analysis of what movement is likely to cause an abnormality, what furniture arrangement has resulted in a movement that is likely to cause an abnormality, etc.

(Modification)
The target region estimation unit 161 and the posture estimation unit 162 may be implemented as a single function. That is, the target region estimation and the posture estimation may be performed simultaneously. Such processing may be realized, for example, by using data of teacher images to which labels indicating posture information are previously assigned when generating a trained model used for target region estimation.

The posture estimation unit 162 and the anomaly determination unit 163 may be implemented as a single function. That is, posture estimation and anomaly determination may be performed simultaneously. Such processing may be realized, for example, by using data of teacher images to which a label indicating whether or not an image is abnormal is assigned in advance when generating a trained model used for posture estimation.

The control unit 16 may be configured not to perform processing as the posture estimation unit 162. In this case, the abnormality determination unit 163 may be implemented by providing a label indicating whether or not an abnormality is present, rather than posture information, to the data of the teacher image when learning the above-mentioned trained model used by the posture estimation unit 162. In this case, it is possible to simply determine whether or not an abnormality is present, without estimating posture information. Details are as follows.

The abnormality determination unit 163 may perform the abnormality determination by using a trained model obtained in advance by performing machine learning processing using a teacher image, for example. In this case, only images of the same kind of organism as the monitoring target may be used as the data of the teacher image. In other words, since it is clear that the monitoring target is included in the target area estimated in the processing of the target area estimation unit 161, when abnormality determination is performed as a subsequent processing, it is not necessary to use images of organisms of a different kind from the monitoring target. Rather, by using only images of the same kind of organism as the monitoring target, it is possible to improve the accuracy of the abnormality determination. As the data of the teacher image, each image is given information indicating whether the state of the organism captured in the image is abnormal or not as a label.

The user notification information may include notification image data instead of notification address information. In this case, the user terminal 30 may display the notification image more smoothly. Furthermore, when the user terminal 30 receives the notification image data, it may display the notification image data on the screen without requiring any operation from the user. By configuring in this manner, it is possible to more quickly notify the user of an abnormality in the monitored object.

When the safety result satisfies a predetermined emergency condition, the notification unit 165 may select a predetermined emergency call destination instead of the user terminal 30 as the notification destination, and send a notification to the emergency call destination. The predetermined emergency condition may be, for example, that the character string included in the safety result contains a predetermined keyword indicating a high urgency, such as "help" or "die."

The monitoring terminal 10 may be equipped with an input device for accepting operations from the monitored subject. For example, when the monitoring terminal 10 is configured using a terminal device such as a smartphone, such an input device is already provided. The monitoring terminal 10 may be configured to temporarily suspend imaging in the imaging unit 12 in response to accepting a predetermined operation from the monitored subject. This configuration makes it possible to more firmly protect the privacy of the monitored subject.

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of the present invention.

100...monitoring system, 10...monitoring terminal, 20...control device, 30...user terminal, 40...network, 90...monitored facility, 11...communication unit, 12...imaging unit, 13...audio input unit, 14...audio output unit, 15...display unit, 16...control unit, 161...target area estimation unit, 162...posture estimation unit, 163...abnormality determination unit, 164...notification image generation unit, 165...notification unit, 166...safety confirmation unit, 167...data acquisition unit, 17...storage unit, 21...communication unit, 22...storage unit, 23...control unit, 231...notification control unit, 232...data control unit

Claims (5)

a target area estimation unit that estimates target area information indicating an area in which a monitoring target is located from an image of the monitoring target;
a posture estimation unit that estimates posture information indicating a posture of the monitoring target;
an abnormality determination unit that determines whether or not an abnormality has occurred in the monitoring target;
a notification unit that, when it is determined by the abnormality determination unit that an abnormality has occurred, generates a notification image, which is an image of a space in which the monitoring target is located, including a target area image that is an image showing the target area information and does not include an image of the monitoring target itself, and transmits the notification image to another device;
Equipped with
The notification unit is a monitoring terminal that selects an illustration showing the shape of the monitored object in the area in which the monitored object is located from among multiple types prepared in advance based on the estimation result of the orientation of the monitored object by the target area estimation unit or the attitude estimation unit, and uses the selected illustration as the notification image . The monitoring terminal according to claim 1, which is installed in a monitored facility where the monitored object is active. A posture estimation unit that estimates a posture of the monitoring target,
the target area estimation unit estimates the area using a trained model obtained by machine learning using a first training data set including a teacher image in which the monitoring target is captured and a teacher image in which the monitoring target is not captured;
3. The monitoring terminal according to claim 1, wherein the posture estimation unit estimates the posture using a trained model obtained by machine learning using a second learning data set in which a proportion of teacher images in which the target to be monitored is not captured is relatively smaller than that of the first learning data set. a safety confirmation unit that outputs a voice from a voice output unit to prompt the monitoring target to respond as to whether or not the monitoring target is in a safe state, and generates information on the confirmation result based on the voice input from the voice input unit thereafter;
The monitoring terminal according to claim 1 , wherein the notification unit also transmits the confirmation result to another device. a target area estimation step of estimating target area information indicating an area in which the monitoring target is located from an image of the monitoring target;
a posture estimating step of estimating posture information indicating a posture of the monitoring target;
an abnormality determination step of determining whether or not an abnormality has occurred in the monitored object;
a notification step of generating a notification image, which is an image of a space in which the monitoring target is located, including a target area image that is an image showing the target area information and does not include an image of the monitoring target itself, when it is determined that an abnormality has occurred in the abnormality determination step, and transmitting the notification image to another device;
having
In the notification step, an illustration showing the shape of the monitored object in the area in which the monitored object is located is selected from a plurality of types prepared in advance based on the estimation result of the orientation of the monitored object in the target area estimation step or the attitude estimation step , and used as the notification image .

Images