CN100551014C - Content processing device, method for processing content - Google Patents

Abstract

By the variation that utilizes the captions in image, comprise to come the theme in the detected image content, and come the content segmentation based on each theme.At first, in picture material, detect the scene change point that significantly changes scene by image switching.Next, be formed on before the scene change point and the average image of one second afterwards frame, and, in high-precision mode, use this average image to detect and captions on scene change point, whether occurred.Detect fragment that equal static captions wherein occur, to create index information about the time cycle of each fragment of wherein having shown equal static captions.

Description

Content processing device, method for processing content

technical field

The present invention relates to a content processing device configured to perform processing such as indexing on video content obtained by, for example, recording a television program, and to a method and a computer program for processing the content. Specifically, the present invention relates to a content processing device configured to determine a scene change of a recorded television program on the basis of the title (i.e., theme) of the television program, and to perform segmentation or classification of the scene, and to processing content Methods and Computer Programs.

More particularly, the present invention relates to a content processing device configured to detect theme changes on the basis of telops included in video content, perform segmentation of video content on the basis of the detected theme, and perform indexing , and relates to methods and computer programs for processing content. In particular, the present invention relates to a content processing device configured to detect a theme change on the basis of a relatively small amount of data by using subtitles included in video content, and to a method and computer program for processing content.

Background technique

In today's information science, the importance of broadcasting is immeasurable. In particular, television broadcasting has a great influence on viewers because sound and images are directly transmitted to the viewers. Broadcasting technology includes a wide range of technologies such as processing, transmitting and receiving signals, and processing audio and video information.

The household penetration rate of televisions is very high, and television programs broadcast from various television stations are viewed by the general public. As an alternative to watching broadcast content, viewers can record content and play back the recorded content at any chosen time.

Recently, advances in digital technology have made it possible to store large amounts of audiovisual data. For example, hard disk drives (HDDs) with a capacity of tens to hundreds of gigabytes are available relatively cheaply, and HDD-based recording devices and personal computers (PCs) capable of recording and playing back television programs are available on the market. . HDD is a random access device. Therefore, when playing a program recorded on the HDD, it is not necessary to play the programs in the recorded order as known video tapes, but any recorded program (or any scene or segment in the program) can be played directly. A viewing mode in which a receiving device such as a television, or a video recording and playing device receives and temporarily stores broadcast content in a large storage device such as a hard disk device, and then plays the stored content is called "server broadcast". By using a server broadcast system different from conventional television systems, viewers do not have to watch a program as it is broadcast, but can watch it at any chosen time.

The increase in hard disk capacity of server broadcast systems has allowed viewers to record up to tens of hours of television programming. However, it is basically impossible for viewers to watch all of the television content recorded on the hard disk. If the viewer can retrieve only the scenes of interest and perform digest viewing, the viewer may be able to efficiently and effectively use the recorded content.

Images must be indexed for scene retrieval and summary viewing of recorded content. As a method of video indexing, a method is known in which a scene change point corresponding to a frame in which a video signal largely changes is detected, and indexing is performed.

For example, a scene change detection method is known for detecting that the scene of an image has changed ( For example, refer to Patent Document 1). When forming a histogram, assign constants to a predetermined layer and its adjacent layers, and add said constants; calculate a new histogram by normalization; by using the newly calculated histogram, in every two consecutive image fields Or detect scene changes in image frames. In this way, scene changes can be accurately detected even in attenuated images.

Many scene changes are included in television programs. Typically, time periods corresponding to specific titles (ie, topics) are treated, and video content is segmented and categorized as suitable for summary viewing. However, even when the same title continues, scenes change frequently. Therefore, video indexing methods that rely only on scene change points will not necessarily provide the indexing desired by the user.

There has been proposed a video-sound content compiling device configured to detect a video cut position by using video data, perform sound clustering using sound data, and integrate video data and sound Indexing is performed on data, and video content is compiled, retrieved, and selected based on index information (for example, refer to Patent Document 2). According to this video-sound content compiling device, index information (for distinguishing voice, silence, and music) obtained from audio information is linked with scene change points. In this way, meaningful segments of images and sounds can be detected as scenes, and less meaningful scene change points can be ignored. However, because there are many scene change points in one television program, it is not possible to segment the video content on the basis of different themes.

Generally, as a method of producing and editing television broadcasts such as news programs and variety programs, a method of displaying subtitles expressly or implicitly indicating titles of programs in corners of image frames is employed. The subtitles displayed in the image frames can be used as an important clue for specifying or evaluating the theme of the broadcast program during the display period of the subtitles. Therefore, subtitles are extracted from video content, and video indexing is performed in which the content of displayed subtitles is defined as an index.

For example, there has been proposed a broadcast program content menu generating device configured to: detect subtitles included in an image frame as characteristic image segments of the image frame, and automatically generate a subtitle representing the broadcast by extracting image data corresponding to only the subtitle A menu of contents of a program (for example, refer to Patent Document 3). Typically, to detect subtitles in a frame, edge detection must be performed. However, edge computing adds a high processing burden. For a device used to perform edge detection for each image frame, a large amount of calculation is required. In addition, the main purpose of the device is to automatically generate program menus for news programs using subtitles extracted from video data, not to specify changes in topics in news programs on the basis of detected subtitles, or to use topics to add image index. In other words, no solution is provided to the problem of performing image indexing based on information about subtitles detected in image frames.

[Patent Document 1]

Japanese Unexamined Patent Application Publication No. 2004-282318

[Patent Document 2]

Japanese Unexamined Patent Application Publication No. 2002-271741

[Patent Document 3]

Japanese Unexamined Patent Application Publication No. 2004-364234

Contents of the invention

An object of the present invention is to provide an excellent content processing apparatus capable of properly performing video indexing of recorded video content and segmenting the video content into scenarios; and providing methods and computer programs for processing content.

Another object of the present invention is to provide an excellent content processing device configured to detect theme changes in video content by using subtitles included in images, segment content by each theme, and perform indexing and providing methods and computer programs for processing content.

Another object of the present invention is to provide an excellent content processing device configured to detect title changes on the basis of a relatively small amount of data by using subtitles included in video content; and to provide a method and computer program for processing content .

By considering the above problems, a first aspect of the present invention provides a content processing device configured to process video content data including image frames in temporal order, the device comprising: a scene change detection unit configured to process Detecting a scene change point in the video content of the image frame, the scene change point is a point between two image frames where the scene of one image frame is significantly different from the scene of the other image frame; the subject detection unit is configured to process Detecting a segment corresponding to a theme and a plurality of consecutive image frames in which the same fixed subtitle is displayed in the subtitle area of the video content; and an index storage unit configured to store the The index information of the time period corresponding to the segment.

It has become common to receive and temporarily store broadcast content such as a television program in a receiving device, and then play the content. The increase in the capacity of hard disks has made it possible to record television programs corresponding to tens of hours. Therefore, it is effective to retrieve only scenes of interest to the viewer from the recorded content, and to allow the viewer to perform digest viewing. To enable scene retrieval and summary viewing of recorded content, images must be indexed.

Conventionally, a method of indexing by detecting scene change points from video content has been known. However, due to the many scene changes included in a television program, the indexing is not necessarily optimal for the viewer.

For broadcast television programs such as news programs and variety shows, subtitles representing the theme of the program are often displayed in the four corners of an image frame. Accordingly, subtitles can be extracted from video content, and the display content of the subtitles can be used as an index. However, to extract subtitles from video content, edge detection processing must be performed for each image frame. This is a problem due to the large number of calculations that must be performed.

Therefore, the content processing apparatus according to the present invention first detects a scene change point included in video content to be processed, and then detects whether subtitles are displayed in image frames immediately before and after the scene change point. If subtitles are detected, detect segments where the same fixed subtitle is displayed. In this way, the amount of edge detection processing performed to extract subtitles is reduced, thereby reducing the processing load imposed on detecting subjects.

For example, the subject detection unit generates an average image of image frames corresponding to periods of one second before and after a scene change point, and detects subtitles included in the average image. If the subtitles are displayed continuously before and after the scene change point, parts of the subtitles will remain sharp in the average image, while other parts will be blurred. In this way, the accuracy of subtitle detection can be improved. Subtitle detection can be performed by performing, for example, edge detection.

The theme detection unit compares the subtitle detected in the average image with the subtitle displayed in the subtitle area of the image frame preceding the scene change point in the segment in which the same fixed subtitle is displayed, and defines the point at which the subtitle disappears as the subtitle of the theme. starting point. Similarly, the subject detection unit compares the subtitle detected in the average image with the subtitle displayed in the subtitle area of the image frame after the scene change point in the segment in which the same fixed subtitle is displayed, and defines the point at which the subtitle disappears end point of the topic. Euclidean between the average colors between image frames can be determined by calculating the average color of each color component in the subtitle area of each image frame compared to the image detected in the average image Whether the distance exceeds a predetermined threshold determines whether the subtitle has disappeared from the subtitle area with a small processing burden. Of course, the point at which subtitles disappear can be detected more accurately by using a known method of detecting scene changes.

However, there is a problem in that, when calculating the average color in the subtitle area, the influence of the background color other than the subtitle included in the subtitle area is large. Thus, as an alternative method, a method of determining whether subtitles exist using edge information has been proposed. In other words, an edge image in the subtitle area of the frame to be compared is determined, and the existence of the subtitle in the subtitle area is determined on the basis of the comparison result of the edge images in the subtitle area of the frame. More specifically, the edge image in the subtitle area of the frame to be compared is determined, and when the number of pixels in the edge image detected in the subtitle area is significantly reduced, it is determined that the subtitle has disappeared, and when the change in the number of pixels is small , to make sure subtitles are displayed continuously. In addition, when the number of pixels of the edge image increases significantly, this can be determined as the appearance of a new subtitle.

When subtitles are changed, the number of pixels of the edge image may not change very much. Even when the variation in the number of pixels of the edge images in the subtitle area between frames is small, when the logical AND of each edge pixel corresponding to each edge image, and the number of edge pixels in the resulting image is significant When reduced (for example, by a third or less), changes in subtitles, ie starting and ending positions of subtitles, may also be evaluated.

The theme detection unit determines the length of the segment on the basis of the start point and the end point, and determines the segment as corresponding to a predetermined theme if the length of the segment is longer than a predetermined amount of time. In this way, false detection can be prevented.

The subject detection unit may determine whether the subtitle is a necessary subtitle based on the size of a subtitle area in which the subtitle is detected in the frame, and position information. The position where the subtitles appear in the image frame, and the size of the subtitles are determined according to commonly established conventions of broadcasters. By detecting subtitles with reference to the position where the subtitles appear in the image frame and the size of the subtitles on the basis of such an established convention, erroneous detections can be reduced.

A second aspect of the present invention provides a computer program written in a computer-readable format for performing on a computer system processing of video content comprising image frames in temporal order, the processing comprising the steps of: Detecting a scene change point in the video content, the scene change point is a point between two image frames in which the scene of one image frame is significantly different from the scene of the other image frame; in the step of detecting the scene change point immediately after On the basis of the image frames before and after the scene change point detected, in the video content to be processed, a segment corresponding to a plurality of consecutive image frames displaying the same fixed subtitle in its subtitle area is detected to detect the gap between the image frames whether a subtitle is displayed in the subtitle area; storing index information indicating a time period corresponding to the segment detected in the step of detecting the segment; and when a theme is selected from the index information stored in the storing step, playing the The index information represents the segment of the video content corresponding to the start time and end time.

The computer program according to the second aspect of the present invention defines a computer program written in a computer-readable format to execute predetermined processing in a computer system. In other words, by installing the computer program according to the second aspect of the present invention to the computer system, cooperative operations are performed on the computer system so that the same operations as those of the content processing apparatus according to the first aspect of the present invention can be realized.

The present invention provides: an excellent content processing device configured to detect a theme change of a video content on the basis of subtitles included in the video content, to perform segmentation of the video content on the basis of the detected theme, and to perform Indexing; and methods and computer programs for processing content.

The present invention provides: an excellent content processing device configured to detect title changes on the basis of a relatively small amount of data by using subtitles included in video content; and a method and computer program for processing content.

For example, in accordance with the present invention, recorded television programs may be segmented on a thematic basis. By segmenting and indexing TV shows by topic, users can watch TV shows in an efficient manner such as digest viewing. For example, a user can check the beginning of a topic while replaying recorded content, and if the topic does not interest them, the user can skip to the next topic. In addition, editing operations such as storing only selected subjects are easy when storing recorded video content on a DVD.

Other objects and advantages of the present invention will be described in detail below with reference to the accompanying drawings.

Description of drawings

1 illustrates a schematic diagram of a functional structure of a video content processing device according to an embodiment of the present invention;

FIG. 2 illustrates a schematic diagram of a subtitle area included in an example scene of a television program;

3 illustrates a flowchart of a subject detection process for detecting segments of video content in which the same fixed subtitle is displayed;

4 illustrates a method of detecting subtitles in an average image acquired from images immediately after and before a scene change point;

5 illustrates a method of detecting subtitles in an average image acquired from images immediately after and before a scene change point;

6 illustrates a method of detecting subtitles in an average image acquired from images immediately after and before a scene change point;

7 illustrates a method of detecting subtitles in an average image acquired from images immediately after and before a scene change point;

8 illustrates an example of the structure of a subtitle detection area in an image frame having an aspect ratio of 720×480 pixels;

FIG. 9 illustrates a situation in which a starting position of a subject is detected from a sequence of frames;

FIG. 10 illustrates a flowchart showing a process of detecting a starting position of a subject from a sequence of frames;

FIG. 11 illustrates a situation of detecting an end position of a subject from a sequence of frames;

FIG. 12 illustrates a flowchart showing a process of detecting an end position of a subject from a sequence of frames.

reference sign

10Video content processing equipment

11 image storage unit

12 scene change detection unit

13 subject detection unit

14 index storage units

15 playback units

Detailed ways

Embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 illustrates a schematic diagram of the functional structure of a video content processing device 10 according to an embodiment of the present invention. The video content processing device 10 shown in the figure includes an image storage unit 11 , a scene change detection unit 12 , a theme detection unit 13 , an index storage unit 14 , and a playback unit 15 .

The image storage unit 11 demodulates and stores broadcast waves, and stores video contents downloaded from information sources via the Internet. For example, the image storage unit 11 may be constituted by a hard disk recorder.

Scene change unit 12 retrieves video content subjected to subject detection from image storage unit 11, tracks scenes (scenes or sets) included in consecutive images, and detects scene change points where scenes change significantly due to switching of images.

For example, the image storage unit 11 may employ the method of detecting a scene change disclosed in Japanese Unexamined Patent Application Publication No. 2004-282318 assigned to the assignee. More specifically, scene change points are determined by generating histograms representing image components of two consecutive fields or frames, and detecting changes in the scene when the sum of the differences of the calculated histograms is greater than a predetermined threshold. When generating the histogram, constants are assigned to the corresponding layer and its neighbors, and the constants are added. Then, with normalization, the result of another histogram is calculated. By using these newly generated histograms, scene changes can be detected in every two images on the screen. Thus, scene changes can be accurately detected even in attenuated images.

The theme detection unit 13 detects a section in which the same fixed subtitle is displayed in the video content subjected to theme detection, and outputs the detected section as a section corresponding to a specific theme in a television program.

In television programs such as news programs and variety shows, subtitles displayed in image frames can be used as important clues for specifying or evaluating the subject matter of segments in the television program in which subtitles are displayed. However, the computation required to detect and extract subtitles is very heavy. Therefore, according to this embodiment, segments in which the same fixed subtitle is displayed are detected on the basis of scene change points detected in the video content in such a manner that the number of image frames on which edge detection must be performed is reduced as much as possible. . Segments in which the same fixed caption is displayed can be thought of as segments of a television program that correspond to a particular theme. When performing segmentation, indexing, and summary viewing of video content, the segment can be properly handled as a single block. Details of the subject detection processing will be described below.

The index storage unit 14 stores time information on each segment detected by the image storage unit 11 in which the same fixed subtitle is displayed. The following table shows an example structure of time information stored in the index storage unit 14. In this table, a record for each detected fragment is provided. In each record, the title corresponding to the subject of the segment, the start time of the segment, and the end time of the segment are recorded. For example, index information can be written in a standard structured description language, such as Extensible Markup Language (XML). The title of the theme may be the title of video content (or TV program), or character information of displayed subtitles.

Table 1

content name Start time [seconds] end time [seconds]

Video 1 20 45 60 80 Video 2 10 25 30 45

The playback unit 15 retrieves video content instructed to be played from the image storage unit 11, and decodes and demodulates the retrieved video content to output the video content as images and sounds. According to this embodiment, the playback unit 15 retrieves appropriate index information from the index storage unit 14 on the basis of the content name in order to play video content and links the index information to the video content. For example, when a theme is selected from the index information managed by the index storage unit 14, the corresponding video content is retrieved from the image storage unit 11, and a section from the start time to the end time indicated by the index information is played.

Next, the theme detection processing performed by the theme detection unit 13 to detect a segment in video content in which the same fixed caption is displayed will be described in detail.

According to this embodiment, frames immediately after and between the scene change point detected by the scene change detection unit 12 are used to detect whether subtitles are displayed in the image frame. When a displayed subtitle is detected, since a segment in which the same fixed subtitle is displayed is detected, edge detection processing for extracting subtitles can be reduced. Therefore, the processing load imposed upon detecting a subject can be reduced.

For example, in a television program with various genres, such as a news program and a variety show, subtitles are displayed to gain understanding and support, and to arouse interest, or to attract viewers' attention. In many cases, fixed subtitles are displayed in one of four areas on the screen, as shown in FIG. 2 . Generally, fixed subtitles have the following characteristics:

1) Used as a representative of the title of a broadcast television program (title, etc.);

2) When TV programs are about the same title, they are displayed consecutively.

For example, in a news program, when a particular news item is broadcast, the title of the news item may be continuously displayed. The theme detection unit 13 detects such a section of a program in which a fixed subtitle is displayed, and adds an index to the detected section corresponding to a specific theme. The theme detection unit 13 is also capable of generating a thumbnail of a detected fixed subtitle, or recognizing characters of a displayed subtitle to obtain character information of a title corresponding to a specific theme.

FIG. 3 illustrates a flowchart showing a theme detection process performed by the theme detection unit 13 to detect sections of video content in which the same fixed subtitle is displayed.

First, the video frame at the first scene change point is retrieved from the video content to be processed (step S1). An average image is generated from image frames corresponding to one second before and one second after the scene change point (step S2). Then, subtitle detection is performed on the averaged image (step S3). If the subtitles continue to be displayed after and between scene change points, parts of the subtitles of the average image will remain sharp while other parts will be blurred. Therefore, detection accuracy of subtitles can be improved. The image frames used to generate the average image are not limited to the image frames one second before and after the scene change point. More than two image frames may be used as long as the image frames for obtaining the average image are obtained from points before and after the scene change point.

4 to 6 illustrate a process of detecting subtitles from an average image generated from image frames before and after a scene change point. Since the scene of one image frame varies significantly from that of the other image frames, the frame obtained by averaging the two image frames is blurred as if the image frames were alpha blended. If the same fixed subtitle is continuously displayed before and after the scene change point as shown in Figure 5, the subtitle part of the average image remains sharp and stands out from the blurry background. Therefore, subtitles can be extracted with high precision by performing edge detection processing. If subtitles are displayed in only one of the image frames before and after the scene change point as shown in Figure 6 (or, if the subtitles displayed in one image frame are different from those displayed in other frames), the average image The subtitle portion of the will be blurred in the same way as the background. In this way, subtitles are not falsely detected.

Typically, subtitles are brighter than the background. Therefore, a method of detecting subtitles using edge information may be employed. For example, YUV conversion is performed on the input image, and, then, edge calculation is performed on the Y component. To perform edge computing, the subtitle information processing method described in Japanese Unexamined Patent Application Publication No. 2004-343352 assigned to the assignee, or the artificial (artificial ) image extraction method.

If subtitles are detected from the averaged image (step S4), a rectangular area satisfying the following conditions is determined as an actual subtitle.

1) An area larger than a predetermined area (for example, larger than 80×30 pixels)

2) Areas that do not cover more than one subtitle area (refer to Figure 2)

The position where the subtitles appear and the size of the subtitles within the image frame are determined according to conventions generally established by broadcasters. By detecting subtitles on the basis of such established conventions with reference to the position where the subtitles appear and the size of the subtitles in the image frame, erroneous detections can be reduced. FIG. 8 illustrates an example of the structure of a subtitle detection area in an image frame having an aspect ratio of 720×480 pixels.

When subtitles are detected, the subtitle areas of the detected subtitles are sequentially compared one by one with the subtitle areas in the image frame before the scene change point. The image frame immediately following the image frame in which the subtitle disappears from the subtitle area is determined as the starting point of the segment corresponding to the specific theme (step S5).

FIG. 9 illustrates a situation where the start position of the subject is detected from the sequence of frames in step S5. As shown in the figure, the comparison of the subtitle area is performed sequentially for each frame forward from the detection of the scene change point of the subtitle in step S3. Then, when a frame in which subtitles disappear from the subtitle area is detected, the immediately following frame is detected as the start position of the subject.

In FIG. 10 , the process of detecting the starting position of a subject from a sequence of frames is shown in the form of a flowchart. First, when there is a frame before the current frame position (step S21), the frame is obtained (step S22), and the subtitle areas of these frames are compared (step S23). Then, if there is no change in the subtitle area ("No" in step S24), subtitles are continuously displayed. Thus, the process returns to step S21 to repeat the above-mentioned process. If there is a change in the subtitle area ("YES" in step S24), the subtitle has disappeared. Thus, the frame immediately following this frame is output as the start position of the subject, and the processing routine is completed.

Similarly, subtitle regions of detected subtitles are compared with subtitle regions in image frames after the scene change point one by one in order. The image frame immediately preceding the image frame in which the subtitle disappears from the subtitle area is determined as the end point of the segment corresponding to the specific theme (step S6).

FIG. 11 illustrates a situation of detecting the end position of a subject from a sequence of frames. As shown in the figure, the comparison of the subtitle area is performed sequentially for each frame backward from the detection of the scene change point of the subtitle in step S3. Then, when a frame in which subtitles disappear from the subtitle area is detected, the immediately following frame is detected as the end position of the theme.

FIG. 12 illustrates a flowchart showing a process of detecting an end position of a subject from a sequence of frames. First, when there is a frame after the current frame position (step S31), the frame is obtained (step S32), and the subtitle areas of these frames are compared (step S33). Then, if there is no change in the subtitle area ("No" in step S34), subtitles are continuously displayed. Thus, the process returns to step S31 to repeat the above-mentioned process. If there is a change in the subtitle area ("YES" in step S34), the subtitle has disappeared. Thus, the frame immediately preceding this frame is output as the end position of the subject, and the processing routine is completed.

When detecting the subtitle disappearing position as shown in FIGS. 9 and 11, by sequentially comparing the subtitle areas of the frames forward and backward from the scene change point as the starting position one by one, the position where the subtitle has disappeared can be accurately detected. . In order to reduce the processing load, the approximate position where the subtitle disappears can be detected through the following steps.

1) Comparing I pictures in a coded picture (such as MPEG) including alternately arranged I pictures (intra-coded pictures) and a plurality of P pictures (inter-frame forward predictive coded pictures)

2) Compare image frames per second

For example, it may be determined whether the subtitle has been removed from the subtitle by calculating the average color of the RGB components of the subtitle area of the compared image frames and determining whether the Euclidean distance (Euclidean distance) of the average color between the image frames exceeds a predetermined threshold value. The subtitle area disappears. In this way, it is possible to determine whether subtitles have disappeared while requiring only a small processing burden. In other words, it is determined that the subtitle has disappeared at the n-th image frame before or after the scene change point satisfying the following equation (1), wherein RO _avg , GO _avg , and BO _avg represent the The average color of the subtitle area (ie, the average of RGB components), Rn _avg , Gn _avg , and Bn _avg represent the average color of the subtitle area in the nth image frame from the scene change point. For example, the threshold is 60.

When the fixed subtitle disappears in the segment where no scene change occurs, the average image will include a clear background, while the subtitle will be blurred, as shown in Figure 7. In other words, the results are opposite to those shown in FIG. 5 . The same is true when fixed subtitles appear in segments where no scene changes occur. In order to more accurately detect the point where the subtitle disappears, the method of detecting a scene change disclosed in Japanese Unexamined Patent Application Publication No. 2004-282318 can be employed for the subtitle area.

Here, there is a problem in that, when the average color is calculated in the subtitle area, the influence of the background color other than the subtitle included in the subtitle area is large, which lowers the detection accuracy. Thus, as an alternative method, a method of determining whether subtitles exist using edge information has been proposed. In other words, the edge images in the subtitle areas of the frames to be compared are determined, and the presence of subtitles in the subtitle areas is determined on the basis of the comparison result of the edge images in the subtitle areas of the frames. More specifically, an edge image in the subtitle area of the frame to be compared is determined, and when the number of pixels in the edge image detected in the subtitle area is significantly reduced, it may be determined that the subtitle has disappeared. On the contrary, when the change in the number of pixels is small, it can be determined that subtitles are continuously displayed.

For example, SC represents the scene change point, Rect represents the subtitle area at SC, and EdgeImg1 represents the edge image of the Rect at SC. EdgeImgN represents the edge image in the subtitle area Rect of the n-th frame counted from SC (towards the start or end of the time axis). The edge image is binarized by a predetermined threshold (eg, 128). In step S23 in the flowchart shown in FIG. 10 and step S33 in the flowchart shown in FIG. 12 , the numbers of edge points (pixel numbers) of EdgeImg1 and EdgeImgN are compared. When the number of edge points is significantly reduced (eg, one-third or less), it can be evaluated that subtitles have disappeared (however, when the number of edge points has been significantly increased, it can be evaluated that subtitles have appeared).

When the number of edge points is not much different between EdgeImg1 and EdgeImgN, it can be evaluated that subtitles are continuously displayed. However, there is a possibility that subtitles have changed even though the number of edge points has not changed greatly. Thus, when a logical AND of each pixel in EdgeImg1 and EdgeImgN is obtained, and the number of edge points in the resulting image is significantly reduced (for example, by a third or less), it is evaluated that the subtitle has changed, That is, this is the start or end point of the subtitle. In this way, detection accuracy can be improved.

Next, the subtitle start point determined in step S5 is subtracted from the subtitle end point determined in step S6 to determine the subtitle display time. Then, by determining the subtitle display time as a section corresponding to a specific theme only when the subtitle is displayed for a predetermined amount of time (step S7), the possibility of false detection can be reduced. It is also possible to obtain genre information on television programs from an electronic program guide (EPG), and change the threshold of subtitle display time according to the genre. For example, because the subtitle display time is relatively long for a news program, the threshold may be set to 30 seconds, however, for a variety show, the threshold may be set to 10 seconds.

The subtitle start point and end point of the section identified in step S7 as corresponding to the specific theme are stored in the index storage unit 14 (step S8).

The subject detection unit 13 contacts the scene change detection unit 12 to confirm whether the video content includes a scene change point after the subtitle end point detected in step S6 (step S9). When no scene change point is found after the subtitle end point, the entire processing routine is completed. When a scene change point is found after the subtitle end point, the frame of the next scene change point is retrieved (step S10), the process returns to step S2, and the above-described theme detection process is repeated.

In step S4, when subtitles are not detected at the scene detection point to be processed, the theme detection unit 13 contacts the scene change detection unit 12 to confirm whether a subsequent scene change point is included in the video content (step S11). When subsequent scene change points are not included, the entire processing routine is completed. Conversely, when a subsequent scene change point is included, the frame of the next scene change point is retrieved (step S10), the process returns to step S2, and the above-described subject detection process is repeated.

According to the present embodiment, the subtitle detection process is performed on the assumption that subtitle areas are provided at the four corners of the television screen, as shown in FIG. 2 . However, in many television programs, the current time is continuously displayed in one of these areas. To prevent erroneous detection, character information of the detected subtitle may be obtained, and if the feature is recognized as a number, the detected subtitle may be determined not to be an actual subtitle.

In some cases, subtitles may disappear from the screen, and after a few seconds, the same subtitles may reappear. In such a case, by treating subtitle display as continuous subtitle display (i.e., continuing the same theme) when the following conditions are met, even when subtitle display is discontinuous, that is, subtitle display is interrupted, it is possible to prevent additional index of.

1) Equation 1 is satisfied in the subtitle area before the subtitle disappears and after the subtitle reappears

2) Before the subtitle disappears, and in the subtitle area after the subtitle reappears, the number of pixels of the edge image is substantially the same, and when the logical "AND" of each corresponding pixel in the edge image is obtained, the pixels of the edge image are basically the same number,

3) The amount of time the caption disappears is equal to or less than a threshold (eg, 5 seconds)

For example, genre information of TV programs can be obtained from EPG, so that the threshold of interruption time can be changed according to the genre of TV programs (such as news programs or variety shows).

Industrial Applicability

In the foregoing, the present invention has been described in detail with reference to specific embodiments. However, it is obvious that those skilled in the art can modify or change these embodiments within the scope of the present invention.

In this specification, the case where indexing is performed on video content obtained mainly by recording television programs is described, but the gist of the present invention is not limited. The content processing apparatus according to the present invention can appropriately perform indexing of various video contents that are generated and compiled for purposes other than television broadcasting and that include subtitle regions representing topics.

The present invention has been disclosed by way of example in nature, and what has been described in this specification should not be construed as limiting. The essence of the present invention should be inferred from the scope of the claims.

Claims (21)

1, a kind of contents processing apparatus is used to handle the video content that comprises chronological picture frame, and this equipment comprises:

Scene change-detection unit is used for detecting scene change point at video content to be processed, this scene change point be wherein scene because the switching of frame and the point of marked change;

Subject detection unit is used for detecting the fragment of video content to be processed, and this fragment is a plurality of continuous images frames that same fixedly captions wherein occur; And

Index storage unit, be used to store with detected by described subject detection unit, wherein the relevant index information of time of the fragment of same fixedly captions appears,

Wherein, whether and then described subject detection unit is used by before the detected scene change point in scene change-detection unit and frame afterwards, occur in the corresponding position to detect captions.

2, contents processing apparatus as claimed in claim 1 also comprises:

Broadcast unit is used for when the displaying video content, will be linked at the index information and the video content of index storage unit place management.

3, contents processing apparatus as claimed in claim 2, wherein, when from index information, having selected theme in index storage unit management, broadcast unit play and the output video content in, from by the zero-time of index information representative fragment to the corresponding video content of concluding time.

4, contents processing apparatus as claimed in claim 1, wherein, described subject detection unit is created in before the scene change point and the average image of the frame in the predetermined period of time afterwards, and this average image is carried out captions detect.

5, contents processing apparatus as claimed in claim 4, wherein, described subject detection unit:

The caption area of scene change point frame forward relatively, and will be in the original position that is the theme of the frame detection of captions after the frame that caption area disappears wherein and then, and

The caption area of scene change point frame backward relatively, and will and then wherein the frame before the frame that disappears from caption area of captions detect the end position that is the theme.

6, contents processing apparatus as claimed in claim 5, wherein, described subject detection unit is calculated average color to each color component in the caption area of each frame that will compare, and pass through to determine whether the Euclidean distance of the average color between the picture frame surpasses predetermined threshold, and determine whether captions disappear from caption area.

7, contents processing apparatus as claimed in claim 5, wherein, edge image in the caption area in each frame that described subject detection unit is determined to compare, and the existence of captions in definite caption area on the basis of the comparative result of the edge image in the caption area of frame.

8, contents processing apparatus as claimed in claim 7, wherein, edge image in the caption area in each frame that described subject detection unit is determined to compare, when the number of pixels of detected edge image in caption area significantly reduces, determine that captions disappear, and in the variation of number of pixels hour, determine that same captions occur continuously.

9, contents processing apparatus as claimed in claim 8, wherein, when the variation of the number of pixels of detected edge image in caption area hour, described subject detection unit obtain to the logical of corresponding each edge pixel of each edge image, and the number of the edge pixel in result images determines that captions change when significantly reducing.

10, contents processing apparatus as claimed in claim 5, wherein, described subject detection unit is determined the captions time of occurrence from detected captions original position to end position, and and if only if the time of occurrence of these captions just definite theme when longer than predetermined amount of time.

11, contents processing apparatus as claimed in claim 5, wherein, described subject detection unit detects in frame on the basis of the size of caption area of captions or positional information therein, determines whether captions are necessary.

12, a kind of content processing method is used for handling the video content that comprises chronological picture frame in being configured in the content-processing system of computer, and this method comprises:

Scene change-detection step, wherein, the scene change-detection parts that comprise in computer detect scene wherein in video content to be processed because the switching of frame and the scene change point of marked change;

The topic detection step, wherein, the topic detection parts that comprise in computer use in scene change-detection step before the detected scene change point and frame afterwards, detect at the scene change point place captions whether occur, and detect the fragment that wherein occurs same fixedly captions before detecting the scene change point of captions and in a plurality of successive image frames afterwards; And

The index stores step, wherein, the index stores component stores that in computer, comprises with wherein appear at described topic detection step in the relevant index information of time of fragment of detected same fixedly captions.

13, content processing method as claimed in claim 12 also comprises:

Play step, when from the index information of the index stores step, storing, having selected theme, play and output from by the zero-time of the index information representative of corresponding video content fragment to the concluding time.

14, content processing method as claimed in claim 12 wherein, in the topic detection step, is created in before the scene change point and the average image of the frame of predetermined period afterwards, and this average image is carried out captions detect.

15, content processing method as claimed in claim 14,

Wherein, in the topic detection step,

The caption area of scene change point frame forward relatively, and will be and then the original position that is the theme of the frame detection of captions after the frame that caption area disappears wherein, and

The caption area of scene change point frame backward relatively, and will be and then wherein the frame before the frame that disappears from caption area of captions detect the end position that is the theme.

16, content processing method as claimed in claim 15, wherein, in the topic detection step, the average color of each color component in the caption area of each frame that calculating will be compared, and whether surpass predetermined threshold by the Euclidean distance of determining the average color between the picture frame, and determine whether captions disappear from caption area.

17, content processing method as claimed in claim 15, wherein, in the topic detection step, the edge image in the caption area in each frame of determining to compare, and on the basis of the comparative result of the edge image in the caption area of frame, determine the existence of the captions in the caption area.

18, content processing method as claimed in claim 17, wherein, in the topic detection step, edge image in the caption area in each frame of determining to compare, and when the number of pixels in the detected edge image in caption area significantly reduces, determine the disappearance of captions, and work as the continuous appearance of hour definite identical captions of variation of number of pixels.

19, content processing method as claimed in claim 18, wherein, in the topic detection step, the variation of the number of pixels in detected edge image in caption area hour, acquisition to the logical of corresponding each edge pixel of each edge image, and when the number of the edge pixel of result images significantly reduces, determine the variation of captions.

20, content processing method as claimed in claim 15, wherein, in the topic detection step, determine the captions time of occurrence from detected captions original position and end position, and and if only if the time of occurrence of these captions is just determined theme when longer than predetermined amount of time.

21, content processing method as claimed in claim 15 wherein, in the topic detection step, detects in frame therein on the basis of the size of caption area of captions or positional information and determines whether captions are necessary.

Images