KR100876494B1 - Integrated file format structure composed of multi video and metadata, and multi video management system based on the same - Google Patents

Abstract

The present invention analyzes the contents of each video by inputting the multi-video obtained from at least one camera, generates the extracted information and the camera information together as a metadata, and saves them as a file integrated with the multi-video data. Multi-video and metadata based on multi-video and metadata application formats, and provides content-based search, navigation, and browsing capabilities for multi-video using metadata. A file format and system that can be applied to a surveillance video environment, as well as a multi-video integrated file format that can be applied to a vehicle to integrate and store audio and video video data, content analysis metadata, and vehicle status information. And it provides this multi-party video management systems that are based on and how.

Description

Integrated file format structure composed of multi video and metadata, and multi video management system based on the same, and method therefor {A INTEGRATED FILE FORMAT STRUCTURE WITH MULTI-VIDEO DATA AND META DATA, AND THE RESULTING MULTI-VIDEO MANAGEMENT SYSTEM AND METHODS}

The present invention relates to an integrated file format structure including multi-video and its analysis metadata, and to a multi-video management system and method based thereon. More particularly, the present invention relates to obtaining a multi-video and analyzing the acquired multi-video. It is about multi-video acquisition, processing, management, and consumption that represent content information as metadata and store it with multi-video data for use in surveillance video.

In general, a method of acquiring and storing a multi-video and storing the multi-video and displaying the multi-video such as 4 and 16 channels on a split screen has become mainstream.

In order to store such multi-video and play the necessary part, it is common to play the recorded video by accessing and playing the desired video phrase through a function such as fast forward (FF). If a moving object appears at, it is recorded and used to search for the relevant video section.

These conventional techniques do not guarantee content interoperability due to different file storage formats and limited content analysis information, so that video or metadata may be played by preparing a content conversion or related system for video data obtained from different systems. There was a difficulty to do.

On the other hand, when the existing video acquisition device is used in a vehicle, as a supplementary means of safe driving, the video data acquired through the front or rear camera is output to the display device installed next to the driver's seat to secure the rear view or the front view at night. It is used as a device to secure more clearly.

The present invention has been made to solve the above-described problems, an object of the present invention is to obtain a multi-video, to analyze the obtained multi-video to extract the content information to generate as metadata to store with the obtained multi-video The present invention provides a multi-video integrated file format structure that enables multi-video acquisition, processing, management, and consumption for surveillance video, and a multi-video management system and method based thereon.

Another object of the present invention is to install one or more cameras in necessary positions, such as front / rear, left / right, and inside of a vehicle, and input video data from one or more cameras and video metadata (eg, moving object detection and Detection time, moving object trajectory, object color / shape, whether the object changes, etc., vehicle driving information (e.g., position (GPS), speed, direction of travel, acceleration pedal depressed, brake pedal depressed, steering wheel angle, horn depressed) Based on the multi-video integrated file format structure, which can easily search, search, and consume the saved surveillance video by storing the information such as the degree, the room temperature, and the interior / exterior sound of the vehicle in one integrated file format. The present invention provides a multi-video management system and a method thereof.

Still another object of the present invention is to analyze video data obtained from one or more cameras, and to detect and track moving objects in video, motion trajectories of moving objects, position of objects, position change information of fixed objects, and Color video, contour information, texture information, sound information (e.g. object breakdown sound, noise, running sound, scream, explosion sound, horn, fighting sound, music sound, gunshot, etc.) The present invention provides a method of extracting and expressing MPEG-7 metadata or general XML metadata.

Still another object of the present invention is to store metadata analyzed for video data obtained from one or more cameras in a file or database in a local or remote storage device, search for the metadata, and retrieve the retrieved results through a user interface. The present invention provides a method of providing a method through a user interface in the form of a reminder of important information set in the process of obtaining the metadata or the metadata.

In order to achieve the above object, a first aspect of the present invention is a multi-video unified file format structure in which at least one video data and associated generated metadata are configured in one file format, and is used for recording file type information. Ftyp box; A moov box for recording information on metadata having one or more video data and visual information (eg, each video data location, data size, data sample size and location, acquisition camera information, etc.); An mdat box for storing at least one video data and corresponding metadata; And a meta box for recording file-level metadata about creation information (eg, creator, creation location, file size, creation date, etc.) of the entire integration file. It is.

Here, each video and related metadata in the mdat box is preferably stored externally as a separate file.

Preferably, the file format structure of each video and metadata stored in the mdat box is an ISO / IEC 14496-12 ISO Base Media file format.

Preferably, the file format structure of each video and metadata stored in the mdat box includes: a ftyp box for recording type information of a file; A move box for recording information on the video data; An mdat box for recording video data; And a meta box for recording location information and metadata of the video data.

Preferably, information about at least one of video ID, video name, applied content protection index, content type, or content encoding or video related metadata is recorded in the meta box.

Preferably, each video related metadata comprises: video generation metadata for generating each video; And video analysis metadata for analyzing each video.

Preferably, the video generation metadata is metadata that records information on the video equipment, the creation date, and the video acquisition location.

Preferably, the data type for the video generation metadata includes title, creator, creation position coordinates and date (CreationCoordinates), creation tool (CreationTool), and copyright (CopyrightString) of the acquired content. A content type (ContentType); A CreatorType comprising an Agent representing date and time information according to a Gregorian criterion; PlaceType including Place Name, Region Name, PostalAddress, and ElectronicAddress; A time type (TimeType) including a time start point (TimePoint) and a time duration (Duration); And a creation tool type and annotation text including information about a video camera, a setting indicating a camera environment parameter configuration information and a setting value for video acquisition. It consists of

Preferably, the video analytics metadata analyzes motion object detection and tracking, object color information, contour information, texture information, and sound information to extract features and express them in metadata.

Preferably, the data type for the video analytics metadata is a color space type (ColorSpace Type) including an RGB, YCbCr, HSV and HMMD color space, a transformation matrix (LinearMatrix) between each color space, and a monochrome color space (Monochrome). ; A DominantColorType including a ColorSpace, a Color Quantization, a Spatial Coherency of Color Information, and a Value representing a ratio of Dominant Color Data; A video segment type (VideoSegmentType) including semantic information (Semantic), media time (MediaTime), and visual descriptor (VisualDescriptor); A summary type including a video summary name (Name), an identifier ID (SourceID) of the original video, a location (SourceLocator) of the original video, and source information (SourceInformation); A Summary Segment GroupType including a Video Summary Group Name and a Summary Segment representing the Video Summary as a Major Segment; HierarchicalSummaryType including a video summary type (SummaryType), a summary list of subjects (SummaryThemeList), and a summary segment group (SummarySegmentGroup); SequentialSummaryType including a video summary type (SummaryType) and the spatio-temporal characteristics (VisualSummaryComponent) of the original video; SemanticBaseType, SemanticBagType, SemanticType, ObjectType, ObjectType, AgentObjectType, EventType, ConceptType, SemanticStateType Semantic including), semantic place type (SemanticPlaceType), and semantic time type (SemanticTimeType); Media Time including Media Time Start Point (MediaTimePoint), Media Relative Time Start Point (MediaRelTimePoint), Media Time Relative Increment Start Point (MediaRelIncrTimePoint), Media Time Length (MediaDuration), and Media Relative Time Length (MediaIncrDuration); A VisualDescriptor including a RegionLocatorType, a GridLayoutType, a SpatialTemporalLocatorType, a Shape Trajectory Type, a FigureTrajectoryType, and a Parameter Representation Trajectory Type; And an audio descriptor (AudioDescriptor) including an audio locator type (AudioLocatorType), an audio content type (AudioContentType), and an audio content annotation type (AudioContentAnnotationType).

A second aspect of the present invention, the video acquisition unit for acquiring video data through a plurality of cameras installed at different locations; A video content analyzer configured to analyze the content of the obtained video data and extract video and audio feature information; A metadata generator for generating the extracted video and audio feature information in the form of metadata; A file format generator for generating the obtained video data and the generated metadata into one integrated file format; A database for storing the generated unified file format; And a content-based retrieval means for interworking with the database to request a video desired by a user with a specific keyword and to search for and display the corresponding video so that the user can easily view the video.

The video rendering unit may further include a video rendering unit which parses video data obtained in real time from the video acquisition unit, performs rendering, and transmits the image data to be stored in the database.

Preferably, the plurality of cameras can be mounted on the inside / outside of the vehicle.

Preferably, the video data obtained from the video acquisition unit is MPEG-2 Video, H.26x, VC1, Motion JPEG, MPEG-4 Visual, MPEG-4 Part 10 Advanced Video Coding (or H.264), or MPEG- 4 is obtained using at least one compression scheme of Scalable Video Coding.

Preferably, the database is a storage device physically mounted in a vehicle.

Preferably, the database is a storage device on a network accessible via the Internet.

Preferably, the content-based retrieval means comprises: a user interface for requesting a video desired by a user with a specific keyword and for displaying the requested result so that the user can easily view it; A user query processor which receives a specific keyword requested from the user interface and processes and transmits a query of a corresponding user; A content-based retrieval unit interworking with the database and receiving a user's query transmitted from the user's query processing unit to retrieve video data desired by the user; And a search result analysis unit which receives the video data retrieved from the content-based search unit, parses the video list and the corresponding information, and transmits the parsed and analyzed information to the user interface unit.

A third aspect of the invention includes the steps of (a) acquiring video data from at least one camera; (b) analyzing content-based features of each of the acquired video data and generating them as metadata; And (c) generating the obtained video data and the generated metadata in one integrated file format and storing the generated video data in a database.

Preferably, in step (b), the generated metadata may include at least one of a video equipment identifier for generating a video and a creation date, a video acquisition location, a weather condition, a camera parameter value related to video acquisition, or a video copyright. It includes.

Preferably, in the step (b), the content-based feature may include at least one of motion object detection and tracking, motion trajectory of the object, position of the object, change of position of the fixed object, color, contour and texture of the object, or sound information. Contains one piece of information.

Preferably, in step (c), the integrated file format is a file structure for storing at least one video data and metadata, and includes at least one independent video data and metadata in an mdat box. Save it.

Preferably, in step (c), metadata for the at least one video data is stored in a move level meta box and a track level meta box, and the video data is M dot. (mdat) can be stored in the box, or the actual storage location pointer can be stored.

Preferably, in step (c), the metadata for the at least one video data is metadata related to the integrated file generation information (e.g., creation location, date, and annotation information, etc.). Box) and store in a track level meta box, the video data may be stored in an mdat box, or may store an actual storage location pointer.

A fourth aspect of the invention includes the steps of (a ') acquiring video data from at least one camera; (b ') analyzing content-based features of each of the obtained video data and generating them as metadata; (c ') storing the obtained video data and the generated metadata in one integrated file format; (d ') performing a search for the stored integrated file format at the request of a user; And (e ') displaying the video corresponding to the search result so that the user can easily view the video.

Preferably, in step (a '), the video data is MPEG-2 Video, H.26x, VC1, Motion JPEG, MPEG-4 Visual, MPEG-4 Part 10 Advanced Video Coding (or H.264), or MPEG-4 Part 10 Scalable Extension-Obtained using at least one compression scheme among scalable video coding.

Preferably, in step (b '), the video analysis result is output to a display device or a speaker.

Preferably, the step (b ') is to set the area to be monitored and the object to be monitored, the path of monitoring tracking through the user interface, and generates metadata about the object.

Preferably, in step (c '), the integrated file format is stored in a vehicle-mounted storage device or a storage device on a network accessible via the Internet.

According to a fifth aspect of the present invention, (a ") obtaining a plurality of video data through a camera mounted in the front / rear / left / right direction of the vehicle; (b") the display device to display the obtained video data Displaying through; And (c ") analyzing the acquired video data to analyze characteristic information of a vehicle approach or damage and alert the display or speaker device.

A sixth aspect of the present invention provides a recording medium on which a program for executing the above-described multi-video management method is recorded.

According to a seventh aspect of the present invention, at least one camera is installed in a front, rear, left, and right sides of a vehicle, or in any desired direction, and automatically analyzes video input from multi-channels and generates metadata to locally or with multi-video data. It is stored in a remote storage device, and also displays aspect information for securing a view of the rear or blind spot by displaying a multi-input video on a display device inside the vehicle, and through video analysis, objects (eg, people, An object of the present invention is to provide a system and method for improving safety driving by notifying a display device of presence or absence of an animal or an obstacle.

According to the multi-video integrated file format structure of the present invention as described above, and a multi-video management system and method based thereon, by analyzing video data obtained from at least one camera, detecting and tracking a moving object in the video, MPEG-7 metadata is automatically extracted by analyzing the moving video of the moving object, the position of the object, the change of the position of the fixed object, the color information, the contour information, the texture information, the sound information, etc. Alternatively, they can be represented as common XML metadata and stored and managed in a unified file format.

In addition, according to the present invention, metadata analyzed for video data obtained from one or more cameras is stored in a file or database system on a local or remote storage device, retrieved for the metadata, and the retrieved results are retrieved through a user interface. It can be easily provided, there is an advantage that can provide important information set in the process of acquiring metadata in the form of a reminder through the user interface.

In addition, according to the present invention, the multi-video data, video analysis metadata and vehicle driving information metadata that is installed from one or more cameras in the front / rear, left / right, inside or any desired direction of the car and input from multiple channels are integrated files It is saved as a vehicle black box in the format and can be used as driving record and surrounding situation data in case of traffic accident.It can be used to obtain and confirm surrounding situation data in case of vehicle theft or vehicle damage. And by displaying the lateral video data there is an advantage that can provide the safety of the vehicle driving by simultaneously performing blind spot visibility, surrounding approach vehicle alarm, and the like.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, embodiments of the present invention illustrated below may be modified in many different forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art.

1 is a diagram illustrating a form of a file format structure for storing one or more videos and metadata applied to an embodiment of the present invention.

Referring to FIG. 1, a surveillance video integrated file format structure for multiple video and metadata storage according to an embodiment of the present invention includes a ftyp box for recording file type information, and at least one or more files. A move box for indicating video data file information (eg, each video data location, data size, data sample size and location, acquisition camera information, etc.); Mdat box for recording at least one video data (AVC video) and the corresponding metadata, and tracks in the move box for recording location information and metadata in the mdat box of video data (AVC video) It comprises a (trak) -level meta box, and a meta box for indicating information of the MAF file level (for example, a MAF file name, a production date, a production place, and the like).

Here, the position information of the video data (AVC video) in the track-level meta box can be recorded by an iloc (item loaction) box in the meta box, and the iinf is a corresponding video (item). As a box for recording information on data, item ID, item name, applied content protection index, content type, content encoding, or video related metadata can be recorded.

Such a file format structure for multiple video and metadata storage preferably uses the ISO / IEC 14496-12 ISO Base Media file format.

2 is a conceptual diagram for storing multi-video acquisition and related generated metadata in one unified file format according to an embodiment of the present invention.

In FIG. 2, video-related metadata generation is largely divided into video generation metadata and video analysis metadata. Here, the video generation metadata refers to metadata that records information on a video equipment for generating video, a creation date, a video acquisition place, and the like.

The file structure for storing such multi-video data and corresponding metadata is not limited to the ISO / IEC 14496-12 ISO Base Media file format.

FIG. 3 is a table illustrating data types for video generation metadata applied to an embodiment of the present invention in a tabular form. FIG. 4 is a diagram illustrating an example of video generation metadata generated using FIGS. 2 and 3. to be.

3 and 4, data types may be defined and used in a table form as shown in FIG. 3 to generate related metadata in FIG. 2.

That is, the data type for the video generation metadata includes content including title, creator, creation position coordinates and date (CreationCoordinates), creation tool, and copyright (CopyrightString) of the acquired content. CreatorType consisting of ContentType, Agent representing date and time information according to Gregorian criteria, Place Name, Region Name, Postal Address PlaceType including,, and ElectronicAddress, TimeType including Time Start Point and Time Duration, Information about Video Camera (Tool), and Video Contains one or more of the CreationToolType, or annotation text, which contains settings for the camera environment parameter configuration information and settings for acquisition. Done that.

5A and 5B are diagrams showing data types for video analysis metadata applied in an embodiment of the present invention in a tabular form, and FIG. 6 illustrates one image lattice splitting in a video to explain the lattice splitting type of FIG. 5. FIG. 7 is a diagram illustrating an example of video generation metadata generated by using the grid division type of FIGS. 5 and 6.

5 to 7, video analysis applied to an embodiment of the present invention, for example, moving object detection and tracking, object color information, contour information, texture information and sound information (eg, object breakage sound, noise, running) Sounds, screams, explosions, horns, fighting sounds, music sounds, gunshots, etc.) to extract features and express them in the form of metadata.

As shown in FIGS. 5A and 5B, data types for video analytics metadata include RGB, YCbCr, HSV and HMMD color spaces, a transformation matrix between each color space (LinearMatrix), and a monochrome color space (Monochrome). A dominant color type including a color space type (ColorSpaceType), a color space, a color quantization (ColorQuantization), a spatial coherency of color information, and a value representing a ratio of dominant color data. DominantColorType, VideoSegmentType including Semantic, MediaTime, and VisualDescriptor, Video Summary Name, Source ID, The summary type including the location of the original video (SourceLocator) and the information of the source video (SourceInformation), the video summary group name, and the video summary are represented as main segments. Is a summary segment group type (SummarySegmentGroupType) containing a summary segment (SummarySegment), a hierarchical summary type (HierarchicalSummaryType) including a video summary type (SummaryType), a summary list of subjects (SummaryThemeList), and a summary segment group (SummarySegmentGroup). ), A sequential summary type (SummaryType) including the video summary type (SummaryType) and the spatio-temporal characteristics of the original video (VisualSummaryComponent), a semantic base type (SemanticBaseType), a semantic package type (SemanticBagType), a semantic type, and an object. Semantics including type (ObjectType), agent object type (AgentObjectType), event type (EventType), concept type (ConceptType), semantic state type (SemanticStateType), semantic place type (SemanticPlaceType), and semantic time type (SemanticTimeType) Semantic), media time start point (MediaTimePoint), media relative time start point (MediaRelTimePoint), media time Media Time, including Relative Increment Start Point (MediaRelIncrTimePoint), Media Time Length (MediaDuration), and Media Relative Time Length (MediaIncrDuration), RegionLocatorType, GridLayoutType, SpaceTime Visual Descriptor including position type (SpatialTemporalLocatorType), shape trajectory indicator type (FigureTrajectoryType), and parameter expression trajectory type (ParameterTrajectoryType), audiolocator type (AudioLocatorType), audio content type (AudioContentType), and audio At least one of an audio descriptor (AudioDescriptor) including a content annotation type (AudioContentAnnotationType).

Here, the grid structure split type (GridLayoutType) is a data type that can describe visual feature information by dividing an image in a video into a grid structure (see FIG. 6).

FIG. 8 conceptually illustrates detection of motion objects, region definition, and movement trajectory tracking in a video applied to an embodiment of the present invention, and FIG. 9 illustrates the spatiotemporal position type and shape trajectory indicator types of FIGS. 8 and 5B. FIG. 1 is a diagram illustrating an example of video generation metadata generated by using.

Referring to FIGS. 8 and 9, first, a motion zone is automatically detected in an image and a motion region is defined as a quadrangle as an example (can be defined as a polygon or an ellipse, a circle, and an area composed of arbitrary external information). . The movement zone may represent the movement according to time as the trajectory information using the vertex movement information of the rectangular box.

FIG. 10 is a diagram illustrating an example of dominant color information of a moving object region in a video applied to an embodiment of the present invention, and FIG. 11 is a video generated using the dominant color type (DominantColor) of FIGS. 10 and 5A. A diagram illustrating an example of generation metadata.

FIG. 12 is a diagram illustrating an example of a video segmentation and a visual descriptor connected by using the video segment type (VideoSegmentType) of FIG. 5A. FIG. 13 is a video segment type (VideoSegmentType) of FIGS. 12 and 5A. FIG. 5B illustrates an example of video generation metadata generated by using a grid split type (GridLayoutType) and a region locator type (RegionLocatorType).

FIG. 14 is a diagram illustrating an example of selecting and voweling a video main section for generating a video summary applied to an embodiment of the present invention. FIG. 15 uses the hierarchical summary type (HierarchicalSummary) of FIGS. 14 and 5A. FIG. 1 is a diagram illustrating an example of generated video generation metadata. FIG.

FIG. 16 is a diagram illustrating an example of video generation metadata generated by using the sequential summary type of FIG. 5A, and FIGS. 17A to 17C are better than those generated by using various data types of FIGS. 5A and 5B. A diagram illustrating an example of rich video generation metadata.

The metadata generated as described above may be stored and managed together with the video data in an integrated file format or may be managed through a database.

FIG. 18 is a block diagram illustrating a multi-video management system based on a multi-video integrated file format structure according to an embodiment of the present invention. For example, FIG. 18 illustrates a multi-video using surveillance cameras installed inside and outside a vehicle. Although the management system is implemented, the present invention is not limited thereto, and any management system may be implemented using a plurality of cameras installed at different locations.

Referring to FIG. 18, a multi-video management system based on a multi-video integrated file format structure according to an embodiment of the present invention includes a video acquirer 100, a video content analyzer 200, and a metadata generator. 300, a file format generation unit 400, a database (DB) 500, and a content-based retrieval device 600.

Here, the video acquisition unit 100 performs a function of acquiring video data at various positions through the plurality of cameras 110a to 110d installed at different positions.

In addition, the video data obtained from the video acquisition unit 100 is, for example, MPEG-2 Video, H.26x, VC1, Motion JPEG, MPEG-4 Visual, MPEG-4 Part 10 Advanced Video Coding (or H.264), Or it is preferably obtained using at least one compression scheme of MPEG-4 Scalable Video Coding.

On the other hand, the plurality of cameras (110a to 110d) is preferably mounted on the inside / outside of the vehicle 10, for example, up / down and left / right, but is not limited to this, it is mounted in various positions of the vehicle 10 It is possible.

The video content analyzer 200 performs a function of automatically extracting video and audio feature information by analyzing content of video data acquired from the video acquirer 100.

In this case, the video and audio feature information may be, for example, detecting and tracking a moving object in a video, a motion trajectory of an object, a position of an object, a change in the position of a fixed object, a color, an outline and a texture of an object, or sound information (eg, an object). Breakage sound, noise, running sound, screaming, explosion sound, horn, fighting sound, music sound, gunshot, etc.).

The metadata generator 300 receives the video and audio feature information extracted from the video content analyzer 200 and generates the metadata in the form of metadata.

The file format generation unit 400 generates a video data obtained from the video acquisition unit 100 and metadata generated from the metadata generation unit 300 in one integrated file format.

The database 500 performs a function of database-saving and managing the integrated file format generated from the file format generation unit 400.

That is, the database 500 as one content through the file format generation unit 400 for storing the metadata generated from the metadata generation unit 300 and the video data acquired from the video acquisition unit 100 as one file. )

Meanwhile, the database 500 may be a storage device physically mounted in the vehicle 10 or may be a storage device on a network that can be accessed through the Internet.

On the other hand, since the structure of the unified file format is the same as that of the unified file format shown in FIGS. 1 to 17, a detailed description thereof will be omitted.

The content-based retrieval apparatus 600 performs a function of interworking with the database 500 so as to request a video desired by a user with a specific keyword, search for the corresponding video, and display the video for easy viewing by the user.

The content-based retrieval apparatus 600 includes a user interface unit 610 and a user interface unit 610 for requesting a video desired by a user with a specific keyword and displaying the requested result for the user to easily view. The user query processor 620 receives a specific keyword requested from the user and processes and transmits the user's query, and the user's query received from the user query processor 620 in conjunction with the database 500 is provided to the user. The content-based search unit 630 for searching video data and the search result analysis unit for receiving the video data retrieved from the content-based search unit 630, parsing and analyzing the video list and the corresponding information, and transmitting the same to the user interface unit 610. And 640.

Through the user interface 610 of the content-based retrieval apparatus 600 configured as described above, a user may select and view detailed information of a video and a necessary part.

Additionally, the video rendering unit 700 may further include a video renderer 700 that parses video data obtained in real time from the video acquisition unit 100 to perform rendering and store the same in the database 500.

That is, the user may monitor in real time the video data acquired through the video rendering unit 700 mounted in the vehicle 10. In addition, it may not only render the video data obtained through the plurality of cameras (110a to 110d), but may also include a function of the content-based search device 600 to display the video stored in the database 500. have.

As described above, the present invention stores and manages multi-video data, video analysis metadata, and vehicle driving information in the aforementioned integrated file format, provides search and search if necessary, and simultaneously monitors a vehicle mounted on a vehicle. By displaying the front / rear and left / right video data through it, it is possible to provide safety of vehicle driving by simultaneously performing blind spot visibility and surrounding approach vehicle warning. In addition, multi-video data, video analysis metadata, and vehicle driving information metadata stored in the integrated file format can be stored as a vehicle black box and used as driving records and surroundings in case of a traffic accident. There is an advantage that can be used as data to obtain and confirm context data.

In addition, although the conventional video data was simply stored and displayed through a video system screen, the present invention firstly, multi-video acquisition and multi-video data analysis metadata generation, and vehicle driving information metadata and Combining and storing can provide interoperability between file formats.

Second, by implementing this as an intelligent surveillance video system, users can easily access and search the desired video section through content-based search and search for surveillance video data, as well as risk warning through surveillance video analysis.

Third, multi-video data, video analysis metadata, and vehicle driving information metadata can be stored as a vehicle black box in an integrated file format that can be used as a driving record and surrounding situation data in case of a traffic accident. It can be used as a data for acquiring and confirming the data, and the front / rear and left / right video data are displayed on the monitor mounted on the vehicle to secure blind spots and to alert the neighboring vehicles at the same time. Can provide.

19 is a flowchart illustrating a multi-video managing method based on a multi-video integrated file format structure according to an embodiment of the present invention.

18 and 19, a multi-video management method based on a multi-video integrated file format structure according to an embodiment of the present invention, first, from at least one camera (110a to 110d) installed in different locations The video data of the position is obtained and transmitted to the video acquisition unit 100 (S100).

In this case, the video data may be, for example, MPEG-2 Video, H.26x, VC1, Motion JPEG, MPEG-4 Visual, MPEG-4 Part 10 Advanced Video Coding (or H.264), or MPEG-4 Part 10 Scalable Extension. Obtaining using at least one compression scheme of scalable video coding.

On the other hand, the video data obtained in step S100 may further include a process of displaying through a separate display device.

On the other hand, the at least one camera (110a to 110d) is preferably mounted on the inside / outside of the vehicle 10, for example, up / down and left / right, but is not limited to this, in various locations of the vehicle 10 Can be mounted.

Next, the content-based feature of each video data acquired in step S100 is analyzed through the video content analyzer 200 and the metadata generator 300 and generated as metadata (S200).

In this case, the metadata generated in step S200 may include at least one of a video equipment identifier for generating a video, a creation date, a video acquisition location, a weather condition, a camera parameter value related to video acquisition, and a video copyright. .

In addition, the content-based feature in the step S200 is, for example, moving object detection and tracking, the movement trajectory of the object, the position of the object, the change of the position of the fixed object, the color, contour and texture of the object, or sound information (eg, object Breakage sound, noise, running sound, screaming, explosion sound, horn, fighting sound, music sound, gunfire, and the like.

The method may further include outputting the result of the video analysis in step S200 through a separate display device or a speaker.

In addition, in step S200, an area to be monitored and a target object to be monitored and a monitoring trace path may be set through a separate user interface 610, and metadata for the corresponding object may be generated.

In addition, the step S200 may further include analyzing feature information of a vehicle approach or damage by analyzing the video data obtained in the step S100 and alerting the display or speaker device.

Subsequently, the video data obtained in step S100 and the metadata generated in step S200 are generated through the file format generation unit 400 in one integrated file format and stored in the database 500 (S300).

In this case, the integrated file format is a file structure for storing at least one video data and metadata, and stores at least one independent single video data and metadata in an mdat box, or mdat. A file pointer may be stored in the box that specifies the location of at least one independent single video data and metadata (see FIGS. 1-17).

In addition, it is preferable that metadata for at least one video data is stored in a meta box, and the video data is stored in an mdat box or stored in an actual storage location pointer.

In addition, the integrated file format may be stored in a storage device mounted on the vehicle 10 or a storage device on a network accessible through the Internet.

In addition, a process of performing a search for the stored integrated file format by a user's request through the content-based retrieval device 600 (S400), and displaying a video corresponding to the search result so that the user can easily view the video ( Display process may further include (S500).

Meanwhile, the multi-video management method based on the multi-video integrated file format structure according to the embodiment of the present invention may also be embodied as computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored.

For example, the computer-readable recording medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a hard disk, a floppy disk, a removable storage device, a nonvolatile memory (Flash memory). Optical data storage, and the like, and also implemented in the form of a carrier wave (eg, transmission over the Internet).

The computer readable recording medium can also be distributed over computer systems connected over a computer network so that the computer readable code is stored and executed in a distributed fashion.

Although the foregoing has described a preferred embodiment of the integrated file format structure composed of multi-video and metadata according to the present invention, a multi-video management system based on the same, and a method thereof, the present invention is not limited thereto. It is possible to carry out various modifications within the scope of the invention and the accompanying drawings and this also belongs to the invention.

1 is a diagram showing the form of a file format structure for storing one or more videos and metadata applied to an embodiment of the present invention.

2 is a conceptual diagram for storing multi-video acquisition and associated generated metadata in one unified file format according to an embodiment of the present invention.

3 is a table showing data types for video generation metadata applied to an embodiment of the present invention.

4 is a diagram illustrating an example of video generation metadata generated using FIGS. 2 and 3.

5A and 5B are tabular data types for video analytics metadata applied in an embodiment of the invention.

FIG. 6 is a diagram illustrating an image grid division structure in a video to explain the grid division type of FIG. 5; FIG.

FIG. 7 is a diagram illustrating an example of video generation metadata generated by using the grid division type of FIGS. 5 and 6.

FIG. 8 conceptually illustrates detection, region definition and motion trajectory tracking of a moving object in a video applied to an embodiment of the present invention. FIG.

FIG. 9 illustrates an example of video generation metadata generated by using the spatiotemporal position type and shape trajectory indicator type of FIGS. 8 and 5B.

FIG. 10 is a diagram illustrating an example of dominant color information of a moving object region in a video applied to an embodiment of the present invention. FIG.

FIG. 11 illustrates an example of video generation metadata generated by using the dominant color type of FIGS. 10 and 5A.

FIG. 12 illustrates an example of a video segmentation and a visual descriptor connected by using the video segment type of FIG. 5A. FIG.

FIG. 13 is a diagram illustrating an example of video generation metadata generated by using the video segment type of FIGS. 12 and 5A and the grid segmentation type and the locator type of FIG. 5B.

FIG. 14 is a diagram illustrating an example of a video main section selection and a collection for generating a video summary applied to an embodiment of the present invention. FIG.

FIG. 15 illustrates an example of video generation metadata generated using the hierarchical summary types of FIGS. 14 and 5A.

FIG. 16 illustrates an example of video generation metadata generated using the sequential summary type of FIG. 5A. FIG.

17A-17C illustrate an example of richer video generation metadata generated using the various data types of FIGS. 5A and 5B.

FIG. 18 is a block diagram illustrating a multi-video management system based on a multi-video integrated file format structure according to an embodiment of the present invention. FIG.

19 is a flowchart illustrating a multi-video management method based on a multi-video integrated file format structure according to an embodiment of the present invention.

Claims (36)

delete A multi-video unified file format structure comprising at least one video data and associated creation metadata in one file format, A ftyp box for recording type information of the file; A move box for recording information on metadata having one or more video data and visual information; An mdat box for storing at least one video data and corresponding metadata; And A meta box for recording file-level metadata of the creation information of the entire integration file, In the mdat box, each video and related metadata are stored externally as separate files, respectively. The method of claim 2, The file format structure of each video and metadata stored in the mdat box is an ISO / IEC 14496-12 ISO Base Media file format. The method of claim 2, The file format structure of each video and metadata stored in the mdat box includes: a ftyp box for recording type information of a file; A move box for recording information on the video data; An mdat box for recording video data; And And a meta box for recording location information and metadata of the video data. The method of claim 4, wherein The meta box records information on at least one corresponding video data among a video ID, a video name, an applied content protection index, a content type, or content encoding or video related metadata. And an integrated file format structure consisting of metadata. The method of claim 4, wherein Video related metadata, Video generation metadata for generating video; And An integrated file format structure consisting of multivideo and metadata, including video analytics metadata for analyzing the video. The method of claim 6, The video generation metadata, Integrated file format structure consisting of multi-video and metadata, characterized in that the recorded information on the video equipment, the creation date, the location of the video acquisition. The method of claim 6, The data type for the video generation metadata is A content type including a title, a creator, a creation position coordinate and a date, a creation tool, a creation tool, and a copyright string of the acquired content; A CreatorType comprising an Agent representing date and time information according to a Gregorian criterion; PlaceType including Place Name, Region Name, PostalAddress, and ElectronicAddress; A time type (TimeType) including a time start point (TimePoint) and a time duration (Duration); or Multi-video and metadata, characterized in that at least one of the creation tool type (Tool), including information about the video camera (Tool), and configuration information and settings of the camera environment parameter configuration for video acquisition (Setting) Integrated file format structure. The method of claim 6, The video analytics metadata, An integrated file format structure consisting of multi-video and metadata characterized by extracting and expressing features in the form of metadata by analyzing motion object detection and tracking, object color information, contour information, texture information, and sound information. The method of claim 6, The data type for the video analytics metadata is A color space type (ColorSpaceType) including an RGB, YCbCr, HSV and HMMD color space, a transformation matrix between each color space (LinearMatrix), and a monochrome color space (Monochrome); A DominantColorType including a ColorSpace, a Color Quantization, a Spatial Coherency of Color Information, and a Value representing a ratio of Dominant Color Data; A video segment type (VideoSegmentType) including semantic information (Semantic), media time (MediaTime), and visual descriptor (VisualDescriptor); A summary type including a video summary name (Name), an identifier ID (SourceID) of the original video, a location (SourceLocator) of the original video, and source information (SourceInformation); A Summary Segment GroupType including a Video Summary Group Name and a Summary Segment representing the Video Summary as a Major Segment; HierarchicalSummaryType including a video summary type (SummaryType), a summary list of subjects (SummaryThemeList), and a summary segment group (SummarySegmentGroup); SequentialSummaryType including a video summary type (SummaryType) and the spatio-temporal characteristics (VisualSummaryComponent) of the original video; SemanticBaseType, SemanticBagType, SemanticType, ObjectType, ObjectType, AgentObjectType, EventType, ConceptType, SemanticStateType Semantic including), semantic place type (SemanticPlaceType), and semantic time type (SemanticTimeType); Media Time including Media Time Start Point (MediaTimePoint), Media Relative Time Start Point (MediaRelTimePoint), Media Time Relative Increment Start Point (MediaRelIncrTimePoint), Media Time Length (MediaDuration), and Media Relative Time Length (MediaIncrDuration); A VisualDescriptor comprising a RegionLocatorType, a GridLayoutType, a SpatialTemporalLocatorType, a Shape Trajectory Type, a FigureTrajectoryType, and a Parameter Representation Trajectory Type; or An integrated file format structure consisting of multi-video and metadata, characterized in that it comprises at least one of an audio descriptor (AudioLocatorType), an audio content type (AudioContentType), and an audio content annotation type (AudioContentAnnotationType). . delete A video acquisition unit which acquires video data through a plurality of cameras installed at different locations; A video content analyzer configured to analyze the content of the obtained video data and extract video and audio feature information; A metadata generator for generating the extracted video and audio feature information in the form of metadata; A file format generator for generating the obtained video data and the generated metadata into one integrated file format; A database for storing the generated unified file format; Content-based retrieval means for interworking with the database and requesting a video desired by a user with a specific keyword, searching for the video, and displaying the video for easy viewing by the user; And And a video rendering unit for parsing video data obtained in real time from the video obtaining unit, performing rendering, and transmitting the same to be stored in the database. The method of claim 12, The plurality of cameras are mounted on the inside / outside of the vehicle multi-video management system, characterized in that. The method of claim 12, The video data obtained from the video acquisition unit may be MPEG-2 Video, H.26x, VC1, Motion JPEG, MPEG-4 Visual, MPEG-4 Part 10 Advanced Video Coding (or H.264), or MPEG-4 Scalable Video. Multi-video management system, characterized in that obtained by using at least one compression scheme of the coding. The method of claim 12, The video and audio feature information, A moving object detection and tracking, the movement trajectory of the object, the position of the object, the change of the position of the fixed object, the color, contour and texture of the object, or multi-video management system comprising at least one information. The method of claim 12, The structure of the integrated file format, A ftyp box for recording type information of the file; A move box for recording information on metadata having one or more video data and visual information; An mdat box for storing at least one video data and corresponding metadata; And And a meta box for recording file-level metadata for creation information of the entire integrated file. The method of claim 16, The file format structure of each video and metadata stored in the mdat box includes: a ftyp box for recording type information of a file; A move box for recording information on the video data; An mdat box for recording video data; And And a meta box for recording the location information and the metadata of the video data. The method of claim 17, Video related metadata, Video generation metadata for generating video; And And a video analysis metadata for analyzing the video. The method of claim 18, The video generation metadata, Multi-video management system, characterized in that the metadata recording information on the video equipment, the creation date, the location of video acquisition. The method of claim 18, The video analytics metadata, A multi-video management system comprising moving object detection and tracking, object color information, contour information, texture information, and sound information, extracting their features, and expressing them in the form of metadata. The method of claim 12, And the database is a storage device physically mounted in a vehicle. The method of claim 12, And the database is a storage device on a network accessible through the Internet. The method of claim 12, The content-based search means, A user interface unit for requesting a video desired by a user with a specific keyword and displaying the requested result so that the user can easily view the video; A user query processor which receives a specific keyword requested from the user interface and processes and transmits a query of a corresponding user; A content-based retrieval unit interworking with the database and receiving a user's query transmitted from the user's query processing unit to retrieve video data desired by the user; And And a search result analyzing unit receiving the video data retrieved from the content-based searching unit, parsing and analyzing the video list and the corresponding information, and transmitting the parsing result to the user interface unit. delete (a) acquiring video data from at least one camera; (b) analyzing content-based features of each of the acquired video data and generating them as metadata; And (c) generating the obtained video data and the generated metadata in one integrated file format and storing them in a database; In step (b), the generated metadata includes at least one of a video equipment identifier for generating a video and a date of creation, video acquisition location, weather condition, video acquisition related camera parameter value, or video copyright. Multi video management method characterized in that. The method of claim 25, In the step (b), the content-based feature may include at least one of moving object detection and tracking, movement trajectory of the object, position of the object, change of the position of the fixed object, color, contour and texture of the object, or sound information. Multi-video management method comprising a. The method of claim 25, In step (c), the generated metadata is stored in a meta box of moov and track levels, and the obtained video data is stored in an mdat box or actually stored. And storing a location pointer. The method of claim 25, In step (c), the structure of the integrated file format is a file structure for storing at least one video data and metadata, and designates the location of at least one independent video data and metadata in an mdat box. And storing a file pointer. delete delete delete (a ') acquiring video data from at least one camera; (b ') analyzing content-based features of each of the obtained video data and generating them as metadata; (c ') storing the obtained video data and the generated metadata in one integrated file format; (d ') performing a search for the stored integrated file format at the request of a user; And (e ') displaying the video corresponding to the search result so that a user can easily view the same, And outputting the video analysis result to a display device or a speaker in the step (b '). The method of claim 32, The step (b ') is a multi-video management method, characterized in that to set the area to be monitored and the object to be monitored, the monitoring tracking path through the user interface and to generate metadata for the target. The method of claim 32, In the step (c '), the integrated file format is stored in a storage device mounted on a vehicle or a storage device on a network accessible through the Internet. (a ") acquiring a plurality of video data through a camera mounted in the front / rear / left / right directions of the vehicle; (b ") displaying the obtained video data through a display device; and (c ") analyzing the acquired video data to analyze characteristic information of a vehicle approach or damage and alerting the display or speaker device of the feature information. A computer-readable recording medium having recorded thereon a program capable of executing the method of any one of claims 25 to 28 or 32-35.

Images