KR20250022490A - Space-time-based object searching system using meta data of heterogeneous multi-channel and method thereof - Google Patents

Abstract

The present invention discloses a spatiotemporal-based object search system and method using heterogeneous multi-channel metadata, including a search manager which performs spatiotemporal search by comparing a query condition of a specific object with metadata of public filming media and generates an index matching the search result, a data collection unit which collects image data of public filming media matched by using the index, and a data manager which tracks a specific object by using data provided by the data collection unit and generates request information for searching private filming media when tracking of a specific object using private filming media is required.

Description

{SPACE-TIME-BASED OBJECT SEARCHING SYSTEM USING META DATA OF HETEROGENEOUS MULTI-CHANNEL AND METHOD THEREOF}

The present invention relates to a spatiotemporal-based object search system and method using heterogeneous multi-channel images, and more specifically, to a technology for finding an image matching a specific object by jointly using heterogeneous multi-channels including public and private shooting media and providing spatiotemporal-based indexing and search using metadata of the heterogeneous multi-channels.

Control systems using public CCTV are used for traffic and crime prevention. Currently, many public CCTVs for traffic monitoring are used for crime solving, traffic volume analysis, and accident situation inquiry by viewing real-time and past images. However, due to budgetary issues with public CCTV, they cannot be installed in all necessary areas, so there are blind spots. Therefore, there is a need for a method to eliminate blind spots in the event of traffic accidents or other incidents.

Private CCTVs installed by individuals or businesses are installed and operated solely for the purpose of crime prevention for individuals or businesses, so they are only used for incidents that cause damage to individuals or businesses, and the use of personal CCTV video data for incidents of others or for public interest purposes is very limited. In addition, CCTVs installed by individuals or businesses may not be recording, or the quality of the images may be poor, making it impossible to detect a desired object.

Vehicle black box footage is mainly used when a vehicle is involved in a traffic accident. In addition, personal filming media such as body cams, personal cameras, smartphones, and drone-mounted cameras are generally used only for their respective filming and operational purposes, and are difficult to use for public interest purposes, accident evidence footage, and social purposes such as finding missing persons, finding companion dogs, and finding evidence footage of industrial espionage.

Recently, as in Patent Document 1 described as prior art, an intelligent video surveillance system capable of extracting object information is provided, which can identify the situation of an incident by itself through an image of a designated surveillance area, track an object that caused the incident, and easily secure the movement path and tracking image of the object. However, since this prior art is characterized by automatically rotating the camera in the direction of the detected object and automatically tracking the object, such as zooming in, when an abnormality sign or a specific object is detected in a designated surveillance area, there is a problem that it is difficult to detect and track a specific object outside the surveillance area.

In addition to the above, the above invention, as described in the prior art patent document 2, discloses a tracking system that automatically photographs the intruder so that the intruder is visible when an intruder occurs in the surveillance area and continuously tracks the intruder. To this end, a tracking system capable of tracking the movement path of an object is provided, including a plurality of video shooting units that photograph the surveillance area to obtain video information, an object extraction unit that analyzes the video information transmitted from the video shooting units with an image recognition program to generate object information with the coordinates and shooting time of a moving object added, and an object tracking unit that compares and analyzes images of each object through the object information to select object information including the same object image, and analyzes the coordinates and shooting time of the selected object information to track the movement path of the object. Patent document 2 uses a plurality of video shooting units (CCTV) to expand the surveillance area, and suggests a technology for using an object extraction unit that generates object information with the coordinates and shooting time of the object added, and a technique for comparing and analyzing images of each object through the object information to select object information including the same object image, and analyzing the coordinates and shooting time of the selected object information to track the movement path of the object.

However, Patent Document 2 suggests multiple CCTVs to resolve blind spots, but no matter how many CCTVs are installed, it is impossible to resolve all blind spots, and the cost required to install many CCTVs increases exponentially, and since it is a technology that extracts time and location data of a specific object in a technology for tracking a moving object and tracks the object while expanding it to adjacent CCTVs, it is limited to CCTV, and since it does not use spatiotemporal metadata search technology, there is a problem that it is difficult to track objects quickly and accurately.

Korean Patent Registration No. 10-1880504 Korean Patent Registration No. 10-2152318

In order to solve the above problems, the present invention provides spatiotemporal based indexing and search using heterogeneous multi-channel metadata including public and private filming media.

According to an embodiment of the present invention for solving the above-described problem, a spatiotemporal based object search system using heterogeneous multi-channel metadata includes a data collection unit (150) that collects heterogeneous multi-channel metadata including public filming media (200) and private filming media (300); a search manager (110) that, when receiving a search condition for a specific object from a requester terminal (400) requesting an image of a specific object, performs a spatiotemporal search by comparing the metadata with the search condition for the specific object and generates an index matching the search result; and a data manager (190) that receives image data corresponding to the index through the data collection unit, analyzes the image data to detect a specific object, and, when a specific object with mobility is detected, generates tracking request information including the time and coordinates at which the specific object was detected, and provides the tracking request information to the search manager, and is characterized in that it provides a spatiotemporal based search using heterogeneous multi-channel metadata including the public filming media and private filming media.

The above search manager may be characterized by dividing cells in a grid-based manner in a map image, setting a cell number for each cell, matching location information of each shooting medium and unique identification information of the shooting medium with the cell number to index metadata, and setting location information of the shooting medium as a KV value of the metadata by shooting time in consideration of moving shooting medium.

The above data manager may be characterized by estimating the spatial coordinates of an image captured by a photographing medium using the location information of the provider terminal (500) and obtaining the location information of the photographing medium corresponding to the spatial coordinates.

According to an embodiment of the present invention, a spatiotemporal based object search method using heterogeneous multi-channel metadata for searching a specific object using an object search system including a search manager, a data collection unit, and a data manager comprises the steps of: the search manager performing a spatiotemporal search by comparing a query condition of a specific object with metadata of a public photographing medium, and generating an index matching the search result; the step of the data collection unit collecting image data of the matched public photographing medium using the index; and the step of the data manager tracking the specific object using the collected image data, and generating tracking request information for searching the private photographing medium if tracking of the specific object using a private photographing medium is necessary; and the search manager classifies the matching private photographing medium by comparing the metadata of the private photographing medium with the tracking request information, and updates the index, and is characterized in that heterogeneous multi-channels including the private photographing medium are used if necessary.

The above data manager processes image data of a private recording medium to de-identify personal information, generates a processed de-identified image, and provides the de-identified image to a reward manager (120). The reward manager provides the de-identified image to a requester terminal (400) that has requested a search for a specific object, and if the de-identified image is accepted from the requester terminal, transmits reward information indicating a reward for providing the image to a provider terminal (500) that provides the de-identified image, and provides the original image of the de-identified image to the requester terminal.

The present invention provides a spatiotemporal index and search using heterogeneous multi-channel metadata including public and private filming media, thereby minimizing blind spots occurring in a single filming medium and enabling quick and accurate search of traffic accidents and various types of evidence video.

FIG. 1 is a block diagram illustrating an object search system according to an embodiment of the present invention.
Figure 2 is a block diagram illustrating in detail the object tracking server of Figure 1.
Figure 3 is a block diagram illustrating the object tracking server of Figure 2 in more detail by function.
Figure 4 is a block diagram illustrating the object tracking server of Figure 2 in more detail, module by module.
FIG. 5 is a block diagram illustrating the object tracking server of FIG. 2 in more detail for each witness service.
Figure 6 is an example illustrating a procedure for collecting metadata from a data provider.
Figure 7 is an example showing the format of metadata.
Figure 8 is a flow chart illustrating the procedure for processing the query conditions of a data requester.
Figure 9 is an example illustrating the functions of a digital witness finding platform.
Figure 10 is an example illustrating a procedure in which a video provider directly uploads and processes a video.
Figure 11 is an example illustrating the principle of tracking a specific object at a specific point in time.
Figure 12 is an example illustrating the principle of tracking a specific object by trajectory.
Figure 13 is an example illustrating a method for estimating the location of a detected object.
Figure 14 is an example illustrating a conventional smoking gun service.
Figure 15 is an example illustrating a conventional disaster safety text message service.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings and the contents described in the attached drawings, but the present invention is not limited or restricted by the embodiments.

FIG. 1 is a block diagram illustrating an object search system according to an embodiment of the present invention. The object search system (10) may not only search for a specific object, but may also include functions of detection and tracking of the specific object, and therefore may be named an object detection system based on heterogeneous multi-channel CCTV.

The object search system (10) includes an object tracking server (100), a public filming medium (200), a private filming medium (300), a requester terminal (400), and a provider terminal (500). The object tracking server (100) is a server that provides a service for tracking objects by analyzing images acquired from the public filming medium (200) and the private filming medium (300).

Public filming media (200) are filming media operated by the state, such as provinces, cities, and counties, and private filming media (400) are filming media operated by individuals or private companies. Public filming media (200) or private filming media may be mobile filming media or fixed filming media depending on the purpose of use.

A typical fixed filming medium is CCTV. Since the CCTV installation location is fixed and data can be collected through Open API, it is easy to collect metadata and image data. However, since the installation location is limited, there are many blind spots. In addition, representative mobile filming media include black boxes, body caps, smartphones, and drones. However, since the location data of mobile filming media continuously changes, it is difficult to collect location metadata of filming media.

The requester terminal (400) for requesting a video is a terminal for requesting a specific object required for evidence of a traffic accident or an incident, and conditions for the desired video, such as time, location, specific object, and utilization, can be input in the service APP using the technology of the present invention, and representative devices include a smartphone, desktop, tablet, and laptop.

As described above, the present invention has described the functions and types of photographing media and terminals, but it may also be a device equipped with communication functions, and is not limited thereto.

The object tracking server (100) can be named as an object search server by jointly utilizing heterogeneous multi-channels including public filming media (200) and private filming media (300) and providing spatiotemporal indexing and search using metadata of heterogeneous multi-channels.

FIG. 2 is a block diagram illustrating the object tracking server of FIG. 1 in detail, and the object tracking server (100) may be referred to as a digital witness service server. The digital witness service server is a server that performs core functions such as metadata and provided video collection, spatiotemporal indexing and storage, spatiotemporal search based on video provision requester query conditions, requesting video provision consent from a video possession candidate as a search result, deep learning-based confirmation of requested specific objects for the provided video, transaction condition adjustment and matching, and reward processing.

The server (100) includes a search manager (110), a data collection unit (150), and a data manager. The search manager (110) performs a spatiotemporal search by comparing the search conditions of a specific object with metadata of a public photographic medium (200), and generates an index that matches the search results.

Metadata refers to data with properties that are applied with certain rules to efficiently find information such as specific objects. Metadata also refers to data that is used for the purpose of expressing data and for the purpose of quickly finding data. Metadata also acts as an index of information on a computer to quickly find data. Metadata typically includes information about the location or shooting time of the shooting medium, and can also include information about specific media.

The search conditions for a specific object may include conditions related to properties of metadata, and may be conditions generated by the requester terminal (400). Spatiotemporal search means a search related to shooting time and location, and means a search related to search conditions or metadata of a specific object.

The data collection unit (150) collects image data of a matched public filming medium (200) using an index. The data manager (190) tracks a specific object using the data provided by the data collection unit (150), and if tracking of a specific object using a private filming medium (300) is required, it generates request information for searching the private filming medium (300).

For example, when a situation arises where it is difficult to detect or track a specific object through a public filming medium (200), the data manager (190) performs tracking of the specific object using a private filming medium (300) and generates request information to achieve this performance.

The search manager (110) compares the metadata of the private filming medium with the request information, classifies the private filming medium (300) that matches, and updates the index to use heterogeneous multi-channels including the public filming medium (200) and the private filming medium (300), thereby solving the problem of blind spots caused by the conventional public filming medium (200).

The server (100) may further include a reward manager (120), a database (130), and a media controller (140). The reward manager (120) may provide a de-identified image to a requester terminal (400) that has requested a search for a specific object, and when the de-identified image is accepted from the requester terminal (400), reward information indicating a reward for providing the image may be transmitted to a provider terminal (500) that provides the de-identified image, and the original image of the de-identified image may be provided to the requester terminal (400).

Anonymized images refer to images that have been processed to de-identify objects related to personal information. Original images refer to original images that have not been processed to de-identify. Reward information refers to compensation for providing images, and may include information about compensation in various forms such as points, cash, or discounts.

The reward manager (120) can provide the original image to the provider terminal (500) before providing the de-identified image to the requester terminal (400), and can provide the de-identified image to the requester terminal (400) when receiving acceptance information indicating acceptance of the image provision from the provider terminal (500).

Even if the provider includes information related to a specific object in the video, personal information that is difficult to disclose to others may be exposed in the video, and even if such personal information is anonymized, there may be situations where it is difficult to disclose it to others, such as the requester. Therefore, it is necessary to discard the video for the sake of the provider's privacy. Accordingly, if the reward manager (120) receives rejection information from the provider terminal (500) rejecting the provision of the video, the reward manager (120) can discard the video.

The search manager (110) can receive metadata of dynamic shooting media, such as mobile shooting media, in real time through the data collection unit (150) and collect only the image data corresponding to the time and place corresponding to the search conditions or request information.

The data collection unit (150) can receive metadata for each moving location in real time from a dynamic shooting medium and store it in a database. If there is metadata matching the query condition or request information, the data manager (190) can receive and analyze image data corresponding to the matched metadata through the data collection unit (150).

FIG. 3 is a block diagram illustrating the object tracking server of FIG. 2 in more detail by function, FIG. 4 is a block diagram illustrating the object tracking server of FIG. 2 in more detail by module, and FIG. 5 is a block diagram illustrating the object tracking server of FIG. 2 in more detail by witness service. The object tracking server can provide a virtual channel area to collect and analyze multi-channel images.

The media controller (140) may be named a virtual CCTV controller. The media controller (140) is a controller for requesting data from a shooting medium. The media controller (140) may be utilized as a controller for transmitting and receiving metadata and image data from all heterogeneous multi-channel shooting media.

The media controller (140) can perform requests and management of metadata, can control numerous CCTVs and video providing devices as a single CCTV, and can act as an interface between CCTVs and the system.

The media controller (140) can provide a service of finding fixed CCTVs such as private and public CCTVs, mobile CCTVs such as vehicle black boxes, body cams, smartphones, and drone-mounted cameras, and CCTVs that meet user requests.

The data collection unit (150) may be named a virtual CCTV data collection unit. The data collection unit (150) may be manufactured integrally with the media controller (140) and may transmit and receive data of a photographing medium in conjunction with a requester terminal (400) or a provider terminal (500).

For example, since it may be difficult for a private filming medium (300) to generate metadata on its own, metadata can be generated by utilizing an application installed on a provider terminal (500). The provider terminal (500) can generate metadata of the private filming medium (300) and transmit it to the data collection unit (150) in real time. The application is an application for a web or mobile environment.

For example, since it may be difficult for a user who has a private filming medium (300) to input metadata by himself/herself, it is possible to automatically collect GPS metadata of the provider terminal by utilizing an application such as an APP installed on the provider terminal (500). Since metadata of public CCTV is disclosed through Open API, it can be received by the virtual CCTV data collection unit of the present invention, and since the location of private CCTV is fixed, if a CCTV operator, i.e., an image provider, provides one detection area photo of the private CCTV at the time of subscription using the APP or web provided by the present invention, the private CCTV location automatically collected from the APP is stored by indexing the coordinates for the center point of the camera image as object location metadata by utilizing a method of mapping the GPS value of the smartphone location and the detection area photo under the same conditions to a map.

The data collection unit (150) can collect data from heterogeneous multi-channel CCTVs. The data collection unit (150) can perform tasks related to collecting necessary data from CCTVs, and can collect metadata and image data from public filming media (200) such as public CCTVs using Open API, and can collect metadata and image data from private filming media (300) such as private CCTVs using user applications.

The user application is an application used by a user (provider) using private CCTV and can receive requests from the server (100) or transmit its own metadata and image data to the server (100). The user application can be a web or mobile application and can perform the role of tracking the trajectory of a user, such as a vehicle or person, and transmitting it.

The data manager (190) supports indexing, storage, and retrieval of collected metadata, and can perform data management tasks of collecting and storing images and videos provided as user-requested search conditions thereafter. The data manager (190) uses RDMS, but the distributed Key-Value storage presented in Fig. 1 is designed with a NoSQL DBMS structure such as distributed Redis in consideration of scalability, and can store user information such as User ID, CCTV metadata such as filming location and time, black box metadata, and metadata of object trajectory data in the database (130). Meanwhile, the provided videos and images that the user actually requests can be stored in an scalable, low-cost, large-capacity file system such as an object storage.

The database (130) can be functionally classified. The database (130) can include a CCTV data storage manager that effectively distributes and stores video and image data. In addition, the database (130) can store video metadata and indexes as distributed key-values.

Metadata used to find requested video holding media, not video data, is indexed and stored based on the spatiotemporal basis of heterogeneous, multi-channel CCTV, which is advantageous for spatiotemporal, rapid and accurate search. In order to support scalability in situations where global services are launched, a distributed K-V storage is used.

The technology for processing collected metadata is important. In the present invention, since metadata includes location, a two-dimensional space is divided into grid cells, each cell is numbered, heterogeneous multi-channel CCTVs included in each cell are found, and each CCTV ID is indexed by attaching the cell number it belongs to, and the KV (Key-Value) value for scalability and fast and accurate spatiotemporal search is stored as the CCTV ID and time as the Key, and metadata such as the coordinates and status of the CCTV are stored as the Value for each CCTV ID at each time.

In the step of searching for a specific object image of the present invention, when a specific image search requester presents image search conditions such as time, location, and specific object, the spatiotemporal based search manager (110) of the system of the present invention can find Ks, that is, heterogeneous CCTV IDs, that are determined to be located closest to the search conditions by using the KNN (K-Nearest-Neighbors) algorithm when a time and location search condition is input, and the image transaction module performs an image provision request in the order of the CCTV IDs that appear as the closest values, and if the image provision is agreed upon, the requested image can be uploaded using the app or web installed at the time of subscription. The AI-based object recognition manager using Yolov8 automatically classifies the specific object requested by the user from the uploaded image. In order to increase the reliability of image search, the present invention supports a procedure for the manager to finally confirm the specific object with the naked eye in addition to the above-mentioned AI-based automatic search for specific objects.

The search manager (110) can provide a virtual CCTV search that searches for CCTV footage that matches the requested time and location based on the video and video index processor and the user search request. The reward manager (120) provides rewards to video providers and a matchmaking service between requesters and providers.

The search manager (110) can request information on KEYs having the corresponding image from the data manager (190) based on location and direction information through spatiotemporal metadata search desired by the user according to the user's search request.

The server (100) or data manager (110) may include an AI-based object recognition manager, a personal information de-identification and encryption manager, and an object tracking manager.

The AI-based object recognition manager can be used to automatically search for a specific object desired by the requester in the video provided by the video provider, and also to anonymize personal information such as faces and vehicle numbers in the video. The personal information anonymization and encryption manager can process anonymization of regular videos uploaded by data providers for personal information protection in videos provided by video providers, and since a video containing a specific object must be provided when a transaction is concluded, an encryption that can release anonymization processed in the transaction condition matching stage and encryption to restrict accessibility to the video itself can be performed.

The object tracking manager analyzes regular videos uploaded by data providers in response to data requests, etc. for specific mobile objects, automatically estimates location coordinates of the specific objects, or calculates location coordinates of the specific objects by trigonometry using the coordinates of heterogeneous multi-channel CCTVs suggested in the present invention, thereby tracking the objects and predicting their direction of movement. This is a very useful method when searching for missing persons, missing pets, or when specific incident/accident evidence videos are mobile.

The reward manager (120) supports compensation for the transaction of the video requested by the requester. Of course, rewards will also be provided to subscribers who agree to provide metadata of their private CCTV or black box videos to the service platform built based on the present invention, and it can also play a role in processing rewards when providing strange or interesting videos, etc., like YouTube, and becoming popular. The reward manager (120) can play a role in supporting transaction contracts and payments with video providers who provide videos that match the user's request.

Fig. 6 is an example of a procedure for collecting metadata from a data provider. Since public CCTV is provided in real time to all citizens, video data can be collected in real time through Open API. Therefore, the server (100) constantly collects and stores metadata such as equipment ID, time, and location in real time, and searches for CCTV that matches the specific time and location requested by the requester and collects the video.

In the case of fixed private CCTV, you can transfer files from local storage using an app or web, or if you have an Internet connection, you can directly transfer data to the server (100). Since this also has issues with personal information protection, real-time video data transmission traffic, and storage space, only metadata can be transferred to the server (100) in normal times. Also, it can be operated so that even metadata is not provided in normal times. In this case, the present invention searches for a specific object requested by a video requester from metadata providers, and if there is no specific object owner among the providers, it uses communication means such as email, text, and KakaoTalk to target subscribers who have not provided metadata, and suggests time, space, specific object, and compensation conditions so that the subscriber can choose to provide the video.

Mobile private CCTVs such as vehicle black boxes or smartphones can normally generate only time and space trajectory metadata and transmit them to the server (100) using an app installed on the user's mobile phone. Of course, as mentioned above, the GPS coordinates of the mobile phone and the coordinates of the area detected by the camera can differ by up to 200m, so three black box or mobile phone photos taken from different directions for estimating the coordinates in the camera detection area and camera images submitted at the time of subscription are used to calculate an approximate coordinate value of the center of the camera image using the GPS coordinates of the mobile phone.

Figure 7 is an example showing the format of metadata. The metadata format or properties for fixed and mobile public CCTV and private CCTV are as follows. Most vehicle black boxes basically have built-in GPS, but if the black box does not have built-in GPS, it is possible to use the GPS of the smartphone of the black box video provider subscriber.

Figure 8 is a flow chart illustrating a procedure for processing a data requester's query conditions. The data requester can create a query condition for querying a specific object through the requester terminal (400).

When the server (100) receives a query condition for a specific object, such as a request from a requester, it performs a spatiotemporal data search based on the metadata index of the storage, creates a list of image owners found to have matching metadata, and requests the transmission of image data to the image owners in the list.

The spatiotemporal query of the present invention system is designed to search in the order of public CCTV, fixed private CCTV, and then mobile filming media such as vehicle black boxes in order to efficiently find video storage devices. The spatiotemporal search technology can find Ks, i.e., heterogeneous CCTV IDs, that are judged to be located closest to the search conditions by using the KNN (K-Nearest-Neighbors) algorithm when time and location search conditions are input. In the video transaction module, a video provision request is made in the order of the CCTV IDs that appear as the closest values, and if video provision is agreed upon, the requested video can be uploaded using the app or web installed at the time of subscription. At this time, if the K value is 50, candidates searched as video holders are listed in order, so if the first-priority video holder does not agree to video provision, the server (100) can sequentially present videos and transaction conditions to other video holders in the next-priority order until video provision agreement and specific object matching are achieved, and the process can be repeated until the request is matched.

The search manager (110) divides cells based on a grid in a map image, sets a cell number for each cell, matches location information of each shooting medium and unique identification information of the shooting medium to the cell number to index metadata, and, considering a moving shooting medium, sets location information of the shooting medium by shooting time as a KV value of the metadata.

The data manager (190) can estimate the spatial coordinates of an image taken by a photographing medium using the location information of the provider terminal (500), and obtain the location information of the photographing medium corresponding to the spatial coordinates. The spatial coordinates can be coordinates for finding out the direction or angle of view of the image, and can obtain the location information of the photographing medium that is actually moving by referencing the location of each object displayed in the image.

The server (100) can estimate and provide a trajectory before and after a requested time by considering the type of object, road network, object image similarity, etc.

When a provider, such as a video owner, provides a video, the server (100) automatically processes de-identification and encryption, and provides the video to the requester after confirmation by the administrator. This is to prevent indiscriminate leakage of personal information, and an AI-based de-identification module can be used for this purpose.

Personal information such as faces and vehicle numbers are processed using anonymization processing technology that automatically detects specific faces or vehicle numbers through AI-based object recognition and mosaics them or blurs the entire background, leaving only the specific object.

The server (100) can request a data transaction from the provider when the data requester decides on the desired video, and after processing the contract and payment for the video provision, etc., can provide the data requester with an anonymization password for the provided video. If the server (100) does not have the requested video or cannot match a specific object video through metadata-based search, it implements a digital witness search function that notifies all subscribers so that the video transaction can be ultimately successful.

FIG. 9 is an example illustrating the function of a digital witness search platform. The object search system (10) using heterogeneous multi-channel images of the present invention can replace analog banners such as banners at intersections looking for traffic accident witnesses, or banners offering compensation to media outlets, insurance companies, and informants looking for missing dogs and missing persons, and can be implemented and serviced as a crowdsourcing-based digital witness search platform in which all citizens of the nation with heterogeneous multi-channels can participate.

In addition, the present invention can be utilized in various services such as finding missing persons, finding companion dogs, finding evidence videos for traffic accidents and various other incidents, crime prevention for social safety and finding evidence videos in case of accidents, preventing industrial espionage in important technology industries, preventing serious disasters such as port container work and crane work, and finding evidence videos in case of accidents, but is not limited thereto.

Figure 10 is an example illustrating a procedure in which a video provider directly uploads and processes a video. The video provider can actively upload a video for tracking a specific object through an app using a provider terminal (500), and the server (100) can analyze the uploaded video.

FIG. 11 is an example illustrating the principle of tracking a specific object at a specific point in time. The present invention can generate a trajectory for multiple CCTVs by estimating the positions (CCTVs) of a specified object at points in time (T _Ni to T _N-1 ) and (T _N+1 to T _N+i ) based on a specific point in time _T N, and can track the trajectory of a specific object in multiple CCTVs according to the object type and object image similarity by considering a road network.

The server (100) can track a continuous trajectory from T ₁ to T ₅ for an object captured by CCTV at time T ₃ as illustrated in Fig. 11. The server (100) can perform object recognition and tracking for image frames before and after the CCTV video in which the object was detected at time T ₃ .

The server (100) can generate a trajectory using an object detection and object tracking model such as Yolo v8, Bot-Sort, etc., and based on the generated trajectory, can estimate the time and location before or after the current CCTV of the corresponding object, and can search for the next CCTV or black box to be viewed.

FIG. 12 is an example illustrating the principle of tracking a specific object by trajectory. The server (100) can extract an image frame corresponding to an estimated time from a searched CCTV or black box, detect an object, and compare the similarity between the object image detected from the searched CCTV or black box and the image of the object currently detected from the CCTV to search for a video clip of the CCTV or black box that has the most similar object image.

The similarity comparison method can compare visual features using a learned Siamese Neural Networks model, and compares the location of an object at a specific time predicted from a trajectory generated from the current CCTV with the location and time of the object detected from the searched CCTV and black box to identify spatiotemporal features.

The data manager (190) can generate a trajectory of a specific object by tracking the specific object in the (N-1)th image and the (N+1)th image when a specific object is detected in the Nth image, and the search manager (110) can search for the next photographing medium to be searched based on the trajectory of the specific object.

The present invention has the effect of efficiently collecting only the minimum data, such as video, for the time and place that meets the requester's request conditions only when the requester requests it, and from the collected video data, it enables AI-based automatic object recognition and semi-automatic video data detection of data contents requested by the user, such as vehicle numbers, business names, human faces, human motions, business names, etc., and improves the protection of personal information and the speed and accuracy of service processing through anonymization and encryption.

FIG. 13 is an example illustrating a method for estimating the position of a detected object. The data manager (190) can set a grid-based valid detection area when detecting an object in a fixed image of a fixed photographing medium, and can set a valid detection area converted into a grid of a fixed image by using its own position information and direction information when detecting an object in a dynamic image of a dynamic photographing medium, and can continuously track a specific object using images of both a fixed image and a dynamic image based on the position information of each photographing medium.

In order to generate a trajectory of an object in a current CCTV in this method, the GPS coordinates of the object must be assigned, and for this purpose, the pixel location of the object detected in the CCTV image must be mapped to GPS coordinates.

The present invention requires an approximate location rather than an exact GPS location, and can approximately estimate the GPS location for objects detected from fixed cameras such as public/private CCTVs and dynamic cameras such as black boxes, as shown in FIG. 13.

The present invention can provide a map-based UI for setting GPS coordinates for an effective detection area of a camera based on a captured image of a CCTV for a fixed camera, and as shown in (a) of Fig. 13, by setting an effective detection area (x1, y1), (x2, y2), a grid can be generated based on this, and the center of each grid cell can be limited to the position of an object.

The present invention can set the center coordinates of a cell in which a detected pedestrian object is located, as shown in FIG. 13, to the pedestrian location at that time, and can provide a UI that combines a map with metadata (time, location) of an image of a black box for a dynamic camera, and can allow a user to set a valid detection area once when signing up for a service.

The present invention can store the length and width of the valid detection area as black box metadata, generate a grid for the length and width, and then, when detecting the actual object location, dynamically calculate the GPS coordinates of each grid cell based on the vehicle location and heading as shown in (b) of Fig. 13 to assign a location to the object.

The present invention can collect and control metadata and image data from heterogeneous multi-channel CCTVs such as fixed CCTVs such as public CCTVs and private CCTVs installed by individuals and institutions, and mobile CCTVs such as vehicle black boxes, smartphones, body cams, and drone-mounted cameras, and can provide a virtual CCTV system that eliminates blind spots of fixed CCTVs by utilizing spatiotemporal search technology, efficiently searches images for specific times and spaces, and supports transactions with specific object requesters.

This can solve the blind spot problem of video data collection through public CCTV such as a large number of traffic situation monitoring and crime prevention CCTVs, and can quickly and accurately search for specific objects and evidence videos by jointly using CCTV videos of individuals and organizations, vehicle black box videos, body cams, smartphone videos, and drone videos, and can effectively track mobile objects such as missing person searches and solve the problem of difficulty in quickly finding the holder of such videos or obtaining them.

The present invention enables operation as a single CCTV by combining fixed and mobile CCTVs at different times and locations, thereby efficiently finding specific object images occurring outside the surveillance area and efficiently tracking and detecting objects moving outside the surveillance area, thereby solving the problems of the conventional patent document 1.

The present invention can use a crowdsourced data federated collection technique using not only fixed filming media such as CCTV but also mobile filming media such as vehicle black boxes, smartphones, and drone filming, and can use a spatiotemporal search technique for metadata, so that it is possible to eliminate blind spots, enable rapid and accurate object tracking, and solve the problems of the conventional patent document 2. In addition, various conventional problems can be solved as follows.

Fig. 14 is an example illustrating a conventional smoking gun service, and Fig. 15 is an example illustrating a conventional disaster safety text service. In the past, as illustrated in Fig. 14, some similar services were provided for utilizing vehicle black box videos for traffic accident evidence video trading through the "Smoking Gun" app, and as illustrated in Fig. 14, government and local government disaster safety text services such as "Finding Missing Persons" were provided.

However, in the past, when a person who needed a video containing a specific object (missing person, accident vehicle, etc.) in an app or an app made a manual request, the person who viewed the page manually searched to see if the video existed and then manually uploaded the video to the web or app.

In the past, the service effectiveness was very low because there was a problem in that the person who had to provide the video had to check the requester's request from time to time, and even if the request was received in the form of text, etc., the person had to manually check whether he or she had the object video for the specific location and time. This can be proven by the number of downloads of similar service apps, and in the case of the actual Smoking Gun, the number of downloads was not very high.

In other words, the conventional "smoking gun" and the missing person search web service of the disaster safety center notify subscribers of the videos and conditions that the user wants. However, this causes a problem in that even if a subscriber has the evidence video that the user wants, the opportunity to provide the video is missed because unspecified subscribers can easily judge that it is not their business. However, the present invention can solve this problem.

The research report on the development of conventional big data-based intelligent CCTV video search technology is a public security science and technology research and development commissioned by the National Police Agency in 2017, which aims to maximize the utilization of CCTV to support the investigation of serious cases and prevent crimes early. In addition, it aims to develop an intelligent video analysis system that can search and extract specific keywords in the video, thereby reducing the time required for investigation by removing unimportant parts from recorded videos and extracting only scenes in which moving objects appear for use in investigations. The research includes video summary technology that extracts only meaningful parts from videos, and video search technology that recognizes specific objects and scenes in videos that meet specific conditions.

In other words, conventional big data-based intelligent CCTV video search technology targets public CCTVs, so it has limitations in eliminating blind spots, and has limitations in securing golden time through crowd evidence video sourcing, rapid automated processing of information security de-identification, and inducing voluntary participation through rewards. However, the present invention can solve these problems.

The solution principle of the present invention to solve all of the conventional problems as described above is as follows.

First, in order to solve the problem of blind spots in each filming medium such as public CCTV, private CCTV, vehicle black boxes, smartphones, drone footage, and paparazzi, the present invention can apply a virtual CCTV system that collects and utilizes heterogeneous multi-channel footage such as public CCTV, private CCTV for individuals and institutions, vehicle black box footage, smartphone footage, drone footage, and body cams.

Second, in order to quickly search for specific objects that require golden time, such as finding missing persons, finding companion dogs, and finding evidence video of accidents, a technology can be applied to collect metadata such as time and space in real time from multi-channel CCTVs of various types and sizes and to index and distribute storage based on time and space. In addition, a notification and compensation system can be applied to find owners of specific objects for subscribers who do not agree to provide metadata in real time.

Thirdly, in order to resolve the personal information issues of the video and the security, speed, and accuracy of specific object search or trajectory search, metadata such as owner ID, time, and location can be encrypted, and for objects that pose personal information protection issues in the video for an actual specific time and location, anonymization processing and encryption of access rights can be applied, and AI-based object recognition and trajectory tracking technology can be applied to the provided video.

Fourthly, in order to realize a service for finding mobile objects such as missing persons or missing pets, such as missing persons, missing pets, and various evidence videos related to accidents, industrial espionage, and disasters, it is necessary to calculate the location coordinates of a specific object captured on CCTV. In the present invention, a method of calculating an approximate value through trigonometry using front and rear images of heterogeneous multi-channel CCTV and the location coordinates (GPS) of the CCTV itself can be applied.

The present invention aims to apply a technology for automatically searching public CCTV footage based on requested footage information in the first place, in order to quickly process the requests of a large number of requesters, such as finding missing persons, finding companion dogs, and finding footage of accidents, which may be obstacles to service realization, while increasing the matching rate of video provision by video holders. Secondly, if the specific object of the requester matching cannot be found in the public CCTV, a system can be provided that searches for the video holder based on metadata such as private CCTV, vehicle black boxes, and smartphones, and then sequentially guides the video holder to a transaction under a compensation condition suggested by the requester and matches the transaction.

In addition, the present invention applies a system that notifies time and location information and compensation condition information of a specific object to subscribers who do not provide metadata collection, so that if the image owner responds, the same transaction system can be applied thereafter.

Fifthly, in order to solve the problem that the quality of all captured images, including CCTV, deteriorates in bad weather conditions such as fog, rain, and snow, and the performance of automatically classifying specific objects using AI models deteriorates, deep learning-based bad weather removal AI models, such as Dehazing, Deraining, and Desnowing AI models, can be applied.

Finally, in order to induce subscription of users of various video media such as private CCTV, vehicle black boxes, body cams, smartphones, and drone-shot videos of the present invention, various motivation and reward systems are necessary. The system of the present invention can induce subscription and expand transactions by providing rewards such as points to subscribers and metadata providers and content services for subscription inducement, such as fun or bizarre videos from vehicle black boxes or smartphones, like YouTube, thereby creating an advertising revenue model and rewarding video providers. Of course, the platform operator must take steps to ensure that videos provided and shared do not constitute illegal or personal information exposure issues.

In addition, the present invention can be operated as a separate menu for sharing emergency situations and response methods based on civilian crowdsourcing, such as sharing real-time reporting videos of various disaster situations including landslides and floods, to be used as disaster prediction data or to enable citizens to respond during the golden time.

As such, the present invention has the characteristics of continuously collecting and storing heterogeneous multi-channel CCTV metadata and video data so that heterogeneous CCTV video data can be utilized for public purposes or transaction purposes according to user demands, and when a video requested by a user, such as an incident or a missing person, occurs, based on the collected metadata, it has the characteristics of automatically, quickly and accurately searching for public and private CCTVs and mobile camera owners such as vehicle black boxes that match the place and time of the incident or accident that satisfies the user demands, and it has the characteristics of requesting video provision from subscribers who can retain the searched video, anonymizing personal information and processing encryption security for the video data provided when video provision is consented to, and classifying specific objects from the acquired video based on deep learning. In addition, in the case of a situation where a specific object is moving, such as finding a dementia patient or a missing child, finding a missing dog, finding a hit-and-run driver or a fugitive, it has the feature of tracking the trajectory of a moving object based on deep learning, and can match transaction conditions such as rewards provided by the service platform to the provider providing the video and rewards from the video requester.

The present invention has the effect of efficiently jointly collecting and efficiently searching heterogeneous multi-channel video data such as video data from public CCTV, CCTV for individuals and businesses, portable devices such as smartphones, and video data from vehicle black boxes, thereby efficiently securing video required for solving problems such as finding evidence video of accidents, finding missing persons, finding companion dogs, and finding evidence of industrial espionage activities.

The present invention is cost-effective because it enables acquisition of crowd-sourced video data from an unspecified number of different heterogeneous filming media, such as existing vehicle black boxes, private CCTVs, smartphones, drones, and vehicle-mounted filming devices, without budget investment, such as additional installation of CCTVs, and is effective in saving time for building a separate system. It also has very high expandability, and in addition to the aforementioned fields of application, it will be effective in solving social problems, such as preventing school violence, dating violence, theft, and assault, as well as in preventing activities related to the Serious Disasters Act, such as monitoring dangerous workplaces, and supporting evidence video in case of an accident.

10: Object Search System 100: Object Tracking Server
110: Search Manager 120: Reward Manager
130: Database 140: Media Controller
150: Data Collection Department 190: Data Manager
200: Public filming media 300: Private filming media
400: Requester terminal 500: Provider terminal

Claims (5)

A data collection unit (150) that collects heterogeneous multi-channel metadata including public filming media (200) and private filming media (300);
When a search condition for a specific object is received from a requester terminal (400) requesting an image of a specific object, a search manager (110) performs a spatiotemporal search by comparing the metadata with the search condition for the specific object and creates an index that matches the search result.
Including a data manager (190) that receives image data corresponding to an index through the above data collection unit, analyzes the image data to detect a specific object, and if a specific object with mobility is detected, generates tracking request information including the time and coordinates at which the specific object was detected, and provides the tracking request information to the search manager.
A spatiotemporal object search system using heterogeneous multichannel metadata, characterized in that it provides spatiotemporal search using heterogeneous multichannel metadata including the above public and private filming media. In the first paragraph,
The above search manager distinguishes cells based on a grid in a map image, sets a cell number for each cell, matches location information of each shooting medium and unique identification information of the shooting medium to the cell number to index metadata, and sets location information of the shooting medium as the KV value of the metadata by shooting time in consideration of mobile shooting medium. A spatiotemporal object search system using heterogeneous multi-channel metadata. In the second paragraph,
The above data manager is a spatiotemporal object search system using heterogeneous multi-channel metadata, characterized in that it estimates the spatial coordinates of an image taken from a photographing medium using the location information of the provider terminal (500) and obtains the location information of the photographing medium corresponding to the spatial coordinates. A spatiotemporal object search method using heterogeneous multi-channel metadata for searching a specific object using an object search system including a search manager, a data collection unit, and a data manager,
A step in which the above search manager performs a spatiotemporal search by comparing the query conditions of a specific object with the metadata of public filming media, and creates an index matching the search results;
A step in which the above data collection unit collects image data of public filming media matched by using an index, and
The above data manager includes a step of tracking a specific object using the collected image data, and if tracking of a specific object using a private filming medium is required, generating tracking request information for searching the private filming medium.
The above search manager compares the metadata of private filming media with the tracking request information, classifies the matching private filming media, and updates the index.
A spatiotemporal object search method using heterogeneous multi-channel metadata, characterized by utilizing heterogeneous multi-channels including private filming media when necessary. In paragraph 4,
The above data manager processes video data from a private recording medium to de-identify personal information, creates a processed de-identified video, and provides the de-identified video to the reward manager (120).
The reward manager provides a de-identified image to a requester terminal (400) that has requested a search for a specific object, and when the de-identified image is accepted from the requester terminal, it transmits reward information indicating a reward for providing the image to a provider terminal (500) that provides the de-identified image, and provides the original image of the de-identified image to the requester terminal. A spatiotemporal object search method using heterogeneous multi-channel metadata.

Images