KR101833943B1 - Method and system for extracting and searching highlight image - Google Patents

Abstract

Disclosed are a method and a system for extracting and searching a highlight scene of a video based on context. The method, which is realized by a computer, comprises the steps of: selecting a candidate frame from a video; calculating a candidate value indicating feature information of images for each of the candidate frames; and extracting some candidate frames as representative images of the video based on the candidate value.

Description

METHOD AND SYSTEM FOR EXTRACTING AND SEARCHING HIGHLIGHT IMAGE < RTI ID = 0.0 >

The following description relates to a technique for providing the main scene of a moving picture.

The rapid increase of users of broadband networks enables the development of new services and diversification of service items through the network. One of the most common services using such a communication network is a video providing service.

A part of video frames constituting a video is extracted and utilized as summary information of the video. As an example of extracting an image frame, there is a method of using a first image frame or a last image frame of a moving image. In order to express a change with time of a moving image, a plurality of image frames are used instead of one image frame do.

For example, Korean Patent No. 10-0547370 (filed on Jan. 20, 2006) entitled " Summary Image Extracting Apparatus and Method Using Object Context Information, and Video Summarization and Index System Using the Same " A technique of extracting a changing shape and position and utilizing it as a representative frame is disclosed.

The present invention provides a method and system for extracting a main scene based on a context of a moving picture.

The present invention provides a method and system for providing context-based key scenes of a moving picture as a scene search function so that the overall contents and flow of a moving picture can be easily grasped.

A computer-implemented method comprising: selecting a candidate frame in a moving picture; Calculating a candidate value indicating feature information of the image for each of the candidate frames; And extracting, as a representative image of the moving image, some of the candidate frames based on the candidate value.

According to an aspect of the present invention, the calculating step may calculate the candidate value using a value indicating a time interval between adjacent frames for each of the candidate frames.

According to another aspect of the present invention, the value indicating the time interval is given a larger value as the inter-frame time interval is longer, and the extracting step may further include the step of extracting a predetermined number of candidate frames or candidate values The candidate frame can be extracted as the representative image.

According to another aspect of the present invention, the calculating step may calculate the candidate value by further using at least one of a value indicating image quality and a value indicating feature information of a caption included in an image for each of the candidate frames.

According to another aspect of the present invention, the calculating step may calculate the candidate value using a feature value of the caption region after detecting a caption region in which a character string exists in each of the candidate frames.

According to another aspect of the present invention, detecting the caption area may detect the caption area using ML-LBP (Multi-Block Local Binary Pattern) for the candidate frame.

According to another aspect of the present invention, the detection of the caption area may include detecting the number of LBPs having different values and the number of pixels whose pixel values are equal to or larger than a threshold value using ML-LBP (Multi Block Local Binary Pattern) It is possible to detect the caption area on the basis of the LBP feature point weights.

According to another aspect of the present invention, the selecting includes: extracting a key frame or a frame at a predetermined time interval from the moving image; Calculating a scene change value between frames for each of the extracted frames; And selecting at least some of the extracted frames as the candidate frame based on the scene change value.

According to another aspect, the method may further include providing a scene search function of the moving image using the representative image.

According to another aspect of the present invention, the step of providing the scene search function may include the representative image as a thumbnail for the segment movement.

A computer program stored in a computer-readable medium for executing a method for providing a main scene, the method comprising: selecting a candidate frame from a moving picture; Calculating a candidate value indicating feature information of the image for each of the candidate frames; Extracting some of the candidate frames as a representative image of the moving image based on the candidate value; And providing a scene search function of the moving picture using the representative image. The present invention also provides a computer program stored in a computer-readable medium.

A computer-implemented system comprising: a candidate selector for selecting a candidate frame in a moving picture; And a representative extraction unit for calculating a candidate value indicating feature information of the image for each of the candidate frames and extracting a frame of the candidate frames as a representative image of the moving image based on the candidate value. System.

According to embodiments of the present invention, it is possible to extract a substantial main scene based on the context of a moving picture.

According to the embodiments of the present invention, the main content of the moving picture and the flow of the moving picture can be easily grasped by providing the main scene based on the context of the moving picture as the scene searching function.

1 is a block diagram for explaining an example of the internal configuration of a computer system according to an embodiment of the present invention.
2 is a diagram illustrating an example of components that a processor of a computer system according to an embodiment of the present invention may include.
3 is a flowchart illustrating an example of a main scene providing method that can be performed by a computer system according to an embodiment of the present invention.
4 is an exemplary diagram for explaining a process of setting a main area in an image according to an embodiment of the present invention.
5 to 9 are exemplary diagrams for explaining image quality measurement items in an embodiment of the present invention.
FIG. 10 is a flowchart illustrating a series of processes for calculating caption feature values according to an exemplary embodiment of the present invention.
11 to 14 are exemplary diagrams for explaining a process of detecting a caption area in an image according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The present invention relates to a technique for providing a main scene of a moving picture, and more particularly to a method for extracting a main scene based on a context of a moving picture and providing an extracted main scene to a scene searching function, ≪ / RTI >

Embodiments, including those specifically disclosed herein, achieve context-based key scene extractions and thereby achieve significant advantages in terms of efficiency, convenience, diversity, accuracy, cost savings, and the like.

The present invention is to extract a main scene from a moving picture using various algorithms for recognizing the context of a moving picture and to provide it as a scene searching function.

One of the functions of the video player is the scene search function, which allows you to quickly and easily move the main scene of the movie. By using the scene search function, the user can search the scene of the video mainly for the main scene in which scene change occurs, and can easily grasp the overall contents of the video through the scene search function.

Conventionally, the time-based reference scene extraction algorithm is used for the scene search function. However, because the time-based reference scene extraction algorithm is a mechanical scene extraction method at regular intervals, it is impossible to move to an important scene according to the context of the video.

In contrast, in the case of the scene search function based on the context of the moving picture, the main scene is extracted according to the weight based on the feature information of the image and is provided as a scene search function, so that the entire flow of the moving picture can be reflected.

1 is a block diagram for explaining an example of the internal configuration of a computer system according to an embodiment of the present invention. For example, a main scene provision system according to embodiments of the present invention may be implemented through the computer system 100 of FIG. 1, the computer system 100 includes a processor 110, a memory 120, a persistent storage 130, a bus 140, an input / output interface 150 and a network interface 160.

Processor 110 may include or be part of any device capable of processing a sequence of instructions. The processor 110 may comprise, for example, a processor and / or a digital processor within a computer processor, a mobile device, or other electronic device. The processor 110 may be, for example, a server computing device, a server computer, a series of server computers, a server farm, a cloud computer, a content platform, a mobile computing device, a smart phone, a tablet, a set top box, The processor 110 may be connected to the memory 120 via a bus 140.

The memory 120 may include volatile memory, permanent, virtual or other memory for storing information used by or output by the computer system 100. Memory 120 may include, for example, random access memory (RAM) and / or dynamic RAM (DRAM). The memory 120 may be used to store any information, such as the state information of the computer system 100. The memory 120 may also be used to store instructions of the computer system 100 including, for example, instructions for providing a main scene. Computer system 100 may include one or more processors 110 as needed or where appropriate.

The bus 140 may comprise a communication infrastructure that enables interaction between the various components of the computer system 100. The bus 140 may, for example, carry data between components of the computer system 100, for example, between the processor 110 and the memory 120. The bus 140 may comprise a wireless and / or wired communication medium between the components of the computer system 100 and may include parallel, serial, or other topology arrangements.

The persistent storage device 130 may be a component such as a memory or other persistent storage device as used by the computer system 100 to store data for a predetermined extended period of time (e.g., as compared to the memory 120) Lt; / RTI > The persistent storage device 130 may include non-volatile main memory as used by the processor 110 in the computer system 100. The persistent storage device 130 may include, for example, flash memory, hard disk, optical disk, or other computer readable medium.

The input / output interface 150 may include a keyboard, a mouse, voice command inputs, displays, or interfaces to other input or output devices. Configuration commands and / or input for providing a moving picture may be received via the input / output interface 150.

The network interface 160 may include one or more interfaces to networks such as a local area network or the Internet. The network interface 160 may include interfaces for wired or wireless connections. Configuration commands and / or services or content associated with the video may be received via the network interface 160.

Also, in other embodiments, the computer system 100 may include more components than the components of FIG. However, there is no need to clearly illustrate most prior art components. For example, the computer system 100 may be implemented to include at least some of the input / output devices connected to the input / output interface 150 described above, or may include a transceiver, a Global Positioning System (GPS) module, Databases, and the like. More specifically, when the computer system 100 is implemented in the form of a mobile device such as a smart phone, an acceleration sensor, a gyro sensor, a camera, various physical buttons, buttons using a touch panel, An input / output port, a vibrator for vibration, and the like may be further included in the computer system 100.

FIG. 2 is a diagram illustrating an example of a component that a processor of a computer system according to an exemplary embodiment of the present invention may include; FIG. 3 is a diagram illustrating a main scene Fig. 2 is a flowchart showing an example of a method of providing a service;

2, the processor 110 may include a candidate selecting unit 210, a representative extracting unit 220, and a scene search unit 230. The components of such a processor 110 may be representations of different functions performed by the processor 110 in accordance with control commands provided by at least one program code. For example, the candidate selector 210 may be used as a functional representation that the processor 110 operates to control the computer system 100 to select a candidate frame in a motion picture. The components of the processor 110 and the processor 110 may perform the steps S310 to S340 included in the main scene providing method of FIG. For example, the components of processor 110 and processor 110 may be implemented to execute instructions in accordance with the at least one program code described above and the code of the operating system that memory 120 contains. Wherein at least one program code may correspond to a code of a program implemented to process the main scene providing method.

The main scene providing method may not occur in the order shown, and some of the steps may be omitted or an additional process may be further included.

In step S310, the processor 110 may load the program code stored in the program file for the main scene providing method into the memory 120. [ For example, a program file for a method of providing a main scene may be stored in the persistent storage device 130 described with reference to FIG. 1, and the processor 110 may receive a program file stored in the persistent storage device 130 The computer system 110 may be controlled such that the program code is loaded into the memory 120. [

At this time, each of the candidate selector 210, the representative extractor 220, and the scene searcher 230 included in the processor 110 and the processor 110 may include a corresponding part of the program code loaded into the memory 120 And may be different functional representations of the processor 110 for executing subsequent steps S320 through S340. For the execution of steps S320 through S340, the processor 110 and the components of the processor 110 may process an operation according to a direct control command or control the computer system 100. [

In step S320, the candidate selector 210 may control the computer system 100 to select a candidate frame among the frames of the moving picture, which is a moving picture for extracting a main scene. For example, the candidate selector 210 may select a key frame of a moving picture as a candidate frame or a frame extracted at a predetermined time interval (e.g., 1 second) as a candidate frame. When a key frame is used as a candidate frame, if the time interval between frames is equal to or less than a predetermined time (for example, 1 second), the candidate frame can be excluded. The candidate selecting unit 210 extracts a key frame or a frame at a predetermined interval from the moving image, calculates a scene change value such as a brightness change and a color histogram change between the extracted frames, At least some of the frames may be selected as candidate frames. For example, the candidate selecting unit 210 may combine the inter-frame brightness variation and the color histogram variation into a certain ratio (e.g., 1: 2) It can be used as a frame.

In step S330, the representative extraction unit 220 calculates a candidate value indicating the feature information of the selected image for each of the candidate frames selected in the moving image, and extracts some of the candidate frames as the representative image based on the candidate value have. For example, the candidate frames selected in the moving picture have independent candidate values, and a certain number of frames may be extracted as the representative images in order of decreasing candidate values by sorting them based on the candidate values. As another example, if the candidate value in the candidate frame has a value equal to or larger than a predetermined value, it is also possible to extract the frame as a representative image. The representative extracting unit 220 is an example for calculating a candidate value including feature information of an image based on the context of a moving image. The representative extracting unit 220 may include at least one of a temporal interval value between candidate frames, an image quality measurement value, A candidate value can be calculated by a combination of values. Here, the time interval-based value is a value indicating a time interval between candidate frames, the image quality measurement value is a value obtained by measuring the image quality of the candidate frame, and the caption feature value is a feature value of the caption included in the image of the candidate frame Quot ;. < / RTI > The factors that can be used to calculate the candidate value, i.e., the time interval based value between candidate frames, the image quality measurement value, and the subtitle feature value will be described below.

In operation S340, the scene search unit 230 may provide a scene search function of the corresponding moving image using the representative image extracted from the moving image. The scene search unit 230 may select a representative image extracted based on the context of the moving image as a main scene of the moving image, and provide the selected scene as a scene search function. The scene search function includes a function of moving a playback section along with a scene search of a moving image. The representative image based on the context can be configured as a main scene of a moving image and utilized as a thumbnail for moving the section.

Hereinafter, each factor used for calculating the candidate value of the candidate frame will be described in detail.

(1) Time interval based value between candidate frames

A value based on a time interval between candidate frames is a value indicating a time interval between adjacent candidate frames, for example, a previous candidate frame. The longer the time interval between the candidate frames is, the larger the relative value is. A small value is given. When the time interval between candidate frames is short, it is highly likely that a frame within the same scene or a frame similar to the image information is selected as the representative image. Therefore, in order to exclude the representative image, the representative extracting unit 220, The candidate value can be calculated using the time interval-based value to be given.

(2) Image quality measurement value

The representative extractor 220 may extract at least one of an exposure value Exposure, a sharpness, a vividness, a low depth of field (DOF), a contrast value, and a prominence The image quality measure can be calculated by a combination of values.

Among the image quality measurement items, the exposure value represents the average brightness value of the main area in the image, and it becomes a deduction when it is too dark or bright. Sharpness indicates the extent to which the main areas of the image are clearly visible, and vividness is similar to sharpness, but the higher the primary color value and ratio, the higher the score. The low depth indicates the depth of field of the image, and a high score is given to a shallow depth image or a simple background image. The contrast value indicates the contrast of the image, and the higher the contrast in the subject, the higher the score. Finally, the prominence shows the contrast between the background of the image and the subject. The more clearly the contrast and color of the subject and the background are distinguished, the higher the score is.

(2-1) Setting the image main area

First, it is necessary to set the main area in the image of the candidate frame in order to calculate the image quality measurement value. The representative extraction unit 220 can search a salient area in the image by using a saliency map model that finds a visible region in the image in a similar visual manner to a human. In other words, the main area means a part where the visual saliency is strong in the image, and it is likely to include a main subject or a subtitle. For example, the representative extraction unit 220 performs Gaussian modeling of the distribution of the visual protrusion map of the image, thereby extracting a main area 401 including a main subject, a subtitle, and the like from the image 400 as shown in FIG. 4 Can be set. If the main area 401 is small, there is a high possibility that the image is highly focused on the subject. The image quality measurement item may acquire a meaningful value obtained in the main area 401, which is a partial area rather than the entire area of the image 400 as the case may be.

(2-2) Exposure value (Exposure)

The exposure value, which is one of the image quality measurement items, is an item that can judge whether or not the exposure of the image is appropriate. 5, the representative extraction unit 220 extracts arithmetic mean (Ma) and geometric mean (Mg) of brightness values and contrast and border line intensity (Mg) for the main region 501 in the image 500 edge strength. At this time, the arithmetic mean (Ma) of brightness values can be set as a base score. It is also possible to consider the geometric mean (Mg) of the brightness value in calculating the exposure value of the main area, since the brightness difference between pixels is not large in case of exposure overexposure. In addition, since exposure and overexposure are low in color contrast, contrast, etc., the contrast is considered when calculating the exposure value of the main area. Similarly, when the exposure value of the main area is calculated, Need to be considered.

In one example, the exposure value can be calculated through Equation (1).

[Equation 1]

Exposure value = base × scale_factor_1 × scale_factor_2 × scale_factor_3

Here, base denotes a quadratic function of the arithmetic mean (Ma) of the brightness values, and scale_factor_1 denotes a weight that is given a larger value as the geometric mean (Mg) of the brightness value is lower than the arithmetic mean (Ma) scale_factor_2 means a weight given a higher value as the contrast is higher, and scale_factor_3 means a weight given a larger value as the border strength becomes higher.

(2-3) Sharpness

One of the image quality measurement items, sharpness, is an item that can discriminate whether or not it is a clear image without blurring. For example, the representative extracting unit 220 may calculate the sharpness of each pixel of the entire image or the main region, and then calculate an average of an upper certain category (for example, 1%) of the absolute values of the pixel sharpness, The sharpness can be obtained. The pixel sharpness can be defined as pixel sharpness = | I-J |, where I is the original image and J is the image blurred the original image I. The sharpness of the whole image or the main area can be an average value of the upper part of the pixel sharpness.

(2-4) Vividness

One of the items of image quality measurement is brightness, which indicates the degree and degree of primary color. As shown in FIG. 6, the luminosity is converted into a hue saturation value (HSV) (color component H, saturation component S, brightness component V) color space 610 and expressed as a product of a saturation component S and a brightness component V, At this time, an average of a part of the pixel (for example, 50%) can be used. The representative extraction unit 220 may calculate the saturation component S of each of the pixels for the entire image or the main area and then calculate the average value of the upper part of the saturation component S And can use it as the life of the corresponding image.

(2-5) Low Depth (Low DOF (depth of field))

Low Depth, another of the image quality metrics, is a measure of the degree of simplicity of the background, such as shallow depth of field (DOF) images. For example, the representative extraction unit 220 can set a major region in an image and then quantify the relative size of the major region, and use the reduced region as the low depth. In other words, as shown in FIG. 7, the representative extracting unit 220 can calculate the low depth by quantifying the relative size of the main region 701 with respect to the size of the entire frame 700. The low depth can be defined as Equation (2).

&Quot; (2) "

Low depth = 1- (size of main area) / (total size of image)}

(2-6) Contrast value (Contrast)

The contrast value, which is another item of the image quality measurement item, indicates the degree of the color contrast and the contrast of the entire image. 8, the representative extractor 220 obtains a covariance matrix in the CIE LAB color space 810 and calculates a covariance matrix of the sum of all diagonal components (eigenvalues) ), Which can be used as a contrast value. Euclidean distance between colors can reflect the actual color difference better than RGB space. The larger the contrast value, the larger the volume occupied by the pixel cloud. In other words, the volume representing the contrast value can be calculated as the sum of the eigenvalues in the matrix after performing the Gaussian modeling on the pixel cloud, that is, by obtaining the covariance matrix.

(2-7) Prominence

Prominence, another of the image quality measurement items, is an item indicating the degree of distinctness of the subject and background such as color and contrast. For example, the representative extracting unit 220 may calculate the prominence using the difference in characteristics of the image of the main region and the background in the entire frame. 9, the representative extraction unit 220 divides the image 900 into a main area 901 and a background area 903 including an object, and extracts the main image 901 and the background area 903 in the main area 901 and the background area 903, respectively, The average pixel value difference over the CIE LAB color space, and the contrast and border strength. At this time, the average pixel value difference can be set as a default value. For example, the prominence can be calculated through Equation (3).

&Quot; (3) "

Prominence = base × max (scale_factor_1, scale_factor_2)

Here, base denotes the average pixel value difference between the main area 901 and the background area 903, and scale_factor_1 denotes a weight that gives a larger value as the contrast ratio between the main area 901 and the background area 903 increases Scale_factor_2 means a weight value that is given a larger value as the ratio of the boundary line strength between the main area 901 and the background area 903 is higher.

Therefore, the representative extractor 220 may calculate the image quality measurement value of the candidate image by a combination of at least one of exposure value, sharpness, vividness, low depth, contrast value, and prominence.

(3) Subtitle feature value

Subtitles existing in the image convey information and act as an eye-lining element in the frame image, so the subtitle feature value can be utilized to extract the representative image among the candidate frames. For example, the representative extraction unit 220 derives a caption area in a frame image to calculate a feature value in the corresponding area. The representative extraction unit 220 extracts a size value, a position value, a motion value, and a color information of a caption, , And frequency information, as shown in FIG.

The representative extraction unit 220 can detect a caption area in which a character string exists in the entire area of the frame image. Detecting a caption region in an image is a task of classifying each pixel of an image into a text region or a non-text region. To this end, the representative extraction unit 220 may use a machine learning method for learning a classifier using various features that can express characteristics of the caption region, or may classify the caption region as a non-caption region You can use an expert system that defines the rules. For example, the subtitle region has a sharp change in the intensity of pixels compared to the non-subtitle region, and since the outline is highly dense and regular, the mean difference feature of pixel intensities in a local image around the pixel to be classified, It is possible to classify subtitles and non-subtitles by using features such as standard deviation or histogram of oriented gradients. After the pixel-by-pixel caption / non-caption classification, it is possible to obtain the final caption region corresponding to the character string by grouping adjacent pixels among the pixels classified as the caption. Then, the representative extractor 220 calculates a size value, a position value, a motion value, a color information, a frequency information, and the like of the corresponding region with respect to the detected caption region when the caption region is finally detected in the frame image, The subtitle characteristic value can be obtained.

FIG. 10 is a flowchart illustrating a series of processes for calculating a caption feature value by deriving a caption region from a frame image. The caption feature value calculation process of FIG. 10 may be included in step S330 of the main scene providing method illustrated in FIG.

In step S1001, the representative extraction unit 220 may control the computer system 100 to receive a candidate frame.

In step S1002, the representative extraction unit 220 may convert the color image corresponding to the candidate frame into a brightness image (e.g., a gray image). For example, the representative extraction unit 220 may correct the light source of the color image by applying a light source correction algorithm to the color image, and then convert the corrected color image into a gray image.

In step S1003, the representative extraction unit 220 may perform binarization based on the hue change on the brightness image. That is, the representative extraction unit 220 may generate a color change-based binary image by detecting a color change pixel in a brightness image.

In step S1004, the representative extraction unit 220 may perform boundary detection based binarization on the brightness image. For example, the representative extraction unit 220 may generate a boundary detection based binary image by detecting a boundary in a brightness image using a Canny algorithm or the like.

In step S1005, the representative extraction unit 220 may obtain a composite image of the two binary images by combining the color change-based binary image and the boundary detection-based binary image.

In step S1006, the representative extraction unit 220 may detect a candidate region that is a region of interest (ROI) for detecting a caption in the composite image.

In step S1008, the representative extracting unit 220 may detect a caption area where a character string actually exists in the candidate area based on ML-LBP (Multi-Block Local Binary Pattern). At this time, the representative extracting unit 220 may extract a ML-LBP feature point by applying a feature extraction technique (e.g., wavelet transform) to the input frame, and utilize the ML-LBP feature point for detecting the caption area (S1007).

The wavelet transform used in image processing can be considered as simply performing subband decomposition. In other words, a low-pass filter and a high-pass filter are used to divide the frequency band of an image and encode each band. The process of dividing the image band using the filter is called wavelet decomposition. Since the image is a two-dimensional signal, it is decomposed using a low-pass filter and a high-pass filter, both horizontally and vertically. As a result, four different bands are generated. As shown in Fig. 11, the upper left image has a low frequency band (LL band) in both the horizontal and vertical directions, the lower left image has a low frequency in the horizontal direction, (LH band), the upper right image has high frequency in the horizontal direction, the low frequency band (HL band) in the vertical direction, and the lower right image has the high frequency band (HH band) in both the horizontal and vertical directions.

Since the LL band contains almost all the information in the LL band, the LL band is regarded as another new image and the wavelet decomposition can be applied to the LL band again. have. As described above, when wavelet-decomposing only the low-frequency band (LL band) of the wavelet-decomposed image repeatedly, four different bands are generated in the low-frequency band as shown in Fig.

The representative extractor 220 may calculate LBP feature point weights for each of the candidate regions, and then determine an area having a weight value equal to or greater than a predetermined size as a caption area. For example, as shown in FIG. 13, when a pixel having a pixel value larger than that of the reference pixel is compared with eight surrounding pixels based on one pixel, and a pixel smaller than the reference pixel is assigned a binary value of 0, You can have a branch pattern. The representative extractor 220 may calculate LBP feature point weights by using LBP for all the pixels in the LH band and the HL band among the wavelet-decomposed images. For example, the LBP feature point weight may be defined as Equation 4 have.

&Quot; (4) "

Weight = lambda x {nLBP / 256}

Where nLBP is the number of LBPs having different values in the candidate region (having a high value for a complex region), and? Is a refinement coefficient indicating the number of effective pixels ). ≪ / RTI > For example, the refinement coefficient can be defined as: " (5) "

&Quot; (5) "

? = {PxCnt ( _pi , HL) + PxCnt ( _pi , LH)} / {2Nxk}

Where N is the number of pixels, i is the position of the pixel [0, N], PxCnt () is the number of pixels above the threshold, and κ is a constant representing the image complexity.

Therefore, the representative extracting unit 220 can detect the caption area 1405 in which the character string actually exists by utilizing the ML-LBP in the candidate frame 1400 as shown in Fig.

10, in step S1009, the representative extraction unit 220 extracts at least one of the feature values of the corresponding region, that is, the size value, the position value, the motion value, the color information, and the frequency information, The subtitle feature value can be calculated by a combination of these. For example, the representative extractor 220 may use the LBP feature point weight of the caption area detected in the candidate frame as the caption feature value of the corresponding image.

The representative extractor 220 may calculate a candidate value using a candidate inter-frame time interval based value for each of the candidate frames, and may further calculate at least one of the image quality measurement value and the subtitle characteristic value The candidate value can be calculated. When the candidate value is calculated by combining two or more factors among the time interval-based value, the image quality measurement value, and the caption feature value, the combination ratio of each factor is applied constantly, or the factor of each factor It is also possible to apply the combination ratios (weights) differently.

Therefore, in the present invention, a candidate frame is selected from a moving image, a candidate value including characteristic information of the selected image is calculated for each selected candidate frame, and a certain frame among the candidate frames is extracted as a representative image based on the candidate value have.

As described above, according to the embodiments of the present invention, it is possible to extract a substantial main scene based on the context of a moving picture, and to provide a main scene based on the context of the moving picture as a scene search function, .

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit, a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be embodied in any type of machine, component, physical device, computer storage media, or device for interpretation by a processing device or to provide instructions or data to the processing device have. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

In a computer implemented method,
Selecting a candidate frame in a moving picture;
Calculating a candidate value indicating feature information of the image for each of the candidate frames; And
Extracting, as a representative image of the moving image, some of the candidate frames based on the candidate value
Lt; / RTI >
Wherein the calculating step comprises:
Setting a salient area according to a distribution of a saliency map in an image of the candidate frame; And
Calculating the candidate value using the feature value for the main region
≪ / RTI > The method according to claim 1,
Wherein the calculating step comprises:
And calculating the candidate value using a value indicating a time interval between adjacent frames for each of the candidate frames
≪ / RTI > 3. The method of claim 2,
The value indicating the time interval is given a larger value as the time interval between frames is longer,
Wherein the extracting comprises:
Extracting a predetermined number of candidate frames or candidate frames whose candidate values are equal to or more than a predetermined value based on the candidate values as the representative images
≪ / RTI > 3. The method of claim 2,
Wherein the calculating step comprises:
And calculating the candidate value by further using at least one of a value indicating image quality and a value indicating characteristic information of a caption included in an image for each of the candidate frames
≪ / RTI > The method according to claim 1,
Wherein the calculating step comprises:
Calculating a candidate value for each of the candidate frames using a feature value of the caption region after detecting a caption region in which a character string exists;
≪ / RTI > 6. The method of claim 5,
To detect the caption area,
Detecting the caption area using ML-LBP (Multi-Block Local Binary Pattern) for the candidate frame
≪ / RTI > 6. The method of claim 5,
To detect the caption area,
Detecting the LBP feature point weight based on the number of LBPs having different values and the number of pixels whose pixel values are equal to or larger than a threshold value using ML-LBP (Multi Block Local Binary Pattern) for the candidate frame
≪ / RTI > The method according to claim 1,
Wherein the selecting comprises:
Extracting a key frame or a frame at a predetermined time interval from the moving image;
Calculating a scene change value between frames for each of the extracted frames; And
Selecting at least some of the extracted frames as the candidate frame based on the scene change value
≪ / RTI > The method according to claim 1,
Providing a scene search function of the moving picture using the representative image
≪ / RTI > 10. The method of claim 9,
The step of providing the scene search function may include:
And configuring the representative image as a thumbnail for moving the moving image
≪ / RTI > A computer program stored in a computer readable medium for executing a method for providing a main scene,
The main scene providing method includes:
Selecting a candidate frame in a moving picture;
Calculating a candidate value indicating feature information of the image for each of the candidate frames;
Extracting some of the candidate frames as a representative image of the moving image based on the candidate value; And
Providing a scene search function of the moving picture using the representative image
Lt; / RTI >
Wherein the calculating step comprises:
Setting a salient area according to a distribution of a saliency map in an image of the candidate frame; And
Calculating the candidate value using the feature value for the main region
A computer program stored on a computer readable medium. In a computer implemented system,
A candidate selecting unit for selecting a candidate frame in the moving image; And
A representative extracting unit for calculating a candidate value indicating feature information of the image for each of the candidate frames, and extracting a frame of the candidate frames as a representative image of the moving image on the basis of the candidate value;
Lt; / RTI >
The representative extracting unit extracts,
Setting a salient area according to a distribution of a saliency map in an image of the candidate frame,
Calculating the candidate value using a feature value of the main region
Lt; / RTI > 13. The method of claim 12,
The representative extracting unit extracts,
And calculating the candidate value using a value indicating a time interval between adjacent frames for each of the candidate frames
Lt; / RTI > 14. The method of claim 13,
The value indicating the time interval is given a larger value as the time interval between frames is longer,
The representative extracting unit extracts,
Extracting a predetermined number of candidate frames or candidate frames whose candidate values are equal to or more than a predetermined value based on the candidate values as the representative images
Lt; / RTI > 14. The method of claim 13,
The representative extracting unit extracts,
And calculating the candidate value by further using at least one of a value indicating image quality and a value indicating feature information of a caption included in the image for each of the candidate frames
Lt; / RTI > 13. The method of claim 12,
The representative extracting unit extracts,
Calculating a candidate value for each of the candidate frames using a feature value of the caption region after detecting a caption region in which a character string exists;
Lt; / RTI > 17. The method of claim 16,
The representative extracting unit extracts,
Detecting the LBP feature point weight based on the number of LBPs having different values and the number of pixels whose pixel values are equal to or larger than a threshold value using ML-LBP (Multi Block Local Binary Pattern) for the candidate frame
Lt; / RTI > 13. The method of claim 12,
Wherein the candidate selector comprises:
Extracting a key frame or a frame at a predetermined time interval from the moving image, calculating a scene change value between frames with respect to each of the extracted frames, and calculating at least some frames of the extracted frames based on the scene change value Selecting the candidate frame
Lt; / RTI > 13. The method of claim 12,
A scene search unit for providing a scene search function of the moving image using the representative image,
≪ / RTI > 20. The method of claim 19,
Wherein the scene search unit comprises:
And configuring the representative image as a thumbnail for moving the moving image
Lt; / RTI >

Images