JPH04209074A - Moving area extraction system - Google Patents

Abstract

PURPOSE:To separate an image into a moving area and a stationary area at a high speed with high accuracy by accurately extracting the moving picture in the moving picture with a small information quantity. CONSTITUTION:Image data of one frame inputted from an image pickup part 1 is divided by a DCT(discrete cosine transformation) part 6 into plural blocks and a frequency component of each block is used as edge information on edge parts of the block. Then an arithmetic part 7 compares edge information of an aimed block of a certain frame with pieces of edge information of the same block which is one frame after and circumferential blocks. The probability that the pieces of edge information of both blocks match and the probability of coincidence with the distance between both the blocks are found, the moving direction and movement quantity of the edge parts of the aimed block are decided according to both the probability values, and the moving area is extracted according to the direction and quantity to separate the moving area and stationary area in the image. Consequently, the moving area can be extracted at a high speed with high accuracy by using the small information quantity.

Description

[Detailed description of the invention]

[00011

[Industrial Application Field] This invention is a method for efficiently recording and transmitting moving images by separating the moving and still areas of a moving image and processing the moving and still areas separately. Regarding the area extraction method, it is particularly suitable for application to videophones that transmit moving images at low transmission rates, imaging devices that have an automatic moving subject detection function, and the like. [0002] [0002] Conventionally, as a device for separating a moving area and a static area from a moving image, for example, Japanese Patent Laid-Open No. 60-25410
No. 7 (name: "Automatic tracking device for cameras") has been proposed. This device focuses on changes in color signals within a specific tracking field of the tracked subject and detects whether the tracked subject is moving or not and the direction of movement.The device moves the focus area to follow the movement of the subject. In the position after moving,
Focus detection or focus adjustment is performed by a known means such as "mountain climbing servo system". [0003] Also, in the paper "Moving region extraction method J using background memory (Preliminary draft of the 1989 Image Coding Symposium. P89-P90. A method has been proposed in which the outline of a moving region is extracted from the difference between this background memory and an input frame using a background memory that is updated and stored as if it were an input frame.[0004]

[Problems to be Solved by the Invention] However, with conventional devices that detect moving areas by focusing on changes in color signals within the tracking field of view, it is difficult to detect when the color of the target object and the color of the background are similar. However, there is a disadvantage that the movement of the subject, that is, the moving area cannot be detected. This is because the accuracy of identifying a moving subject cannot be improved due to the ambiguity of color information. [0005] Furthermore, in the method of extracting a moving region using background memory, a memory with a huge storage capacity is required to store all the pixels of one frame, and furthermore, if only the difference between two frames is taken, the movement direction cannot be measured. There is a disadvantage that it is not possible to determine the [0006] An object of the present invention is to provide a moving region extraction method that can accurately extract a moving region of a moving image with a small amount of information, and can separate moving regions and static regions at high speed and with high precision. shall be. [0007]

[Means for Solving the Problems] The moving area extraction method according to the present invention extracts one frame of image data from one block n
Divide into multiple blocks of ×n pixels, extract edge information of the edge part in each block from specific frequency components included in each block, and combine the edge information of the block of interest in a certain frame with the same block one frame later. and the edge information of the surrounding blocks, and from the probability that the edge information of the target block matches the edge information of each block one frame later, and the probability that both blocks match with respect to the distance between both blocks, the edge information of the target block is calculated. The moving direction and amount of the edge portion of the image are determined, and the moving area is extracted from the judgment result to separate the moving area and the static area in the image. [0008]

[Operation] The present invention focuses on edge portions in an image and recognizes moving areas in the image by detecting movement of the edge portions between frames. To detect the movement of edge parts between frames, one frame of image data is divided into multiple blocks, and edge information of the edge parts of each block is extracted from specific frequency components contained in each block and compared. To do this. [0009] Edge information consists of n×n forming each block.
A specific transform coefficient among n×n transform coefficients obtained by performing two-dimensional orthogonal transform on pixel image data is used. [00101 Then, the edge information of the block of interest in a certain frame is sequentially compared with the edge information of the same block and its surrounding blocks one frame later, and the probability that both blocks are the same is calculated from the ratio of the edge information of both blocks. demand. This probability (membership value) is obtained from a predefined membership function. [0011] Also, focusing on the distance between both blocks, the probability that both blocks match (membership value) with respect to the distance between both blocks is determined from another predefined membership function. This is because even if the edge information is the same, if the distance between both blocks is large, it is considered a coincidence. [0012] Then, from both messageship values, it is inferred to which block the edge portion of the block of interest has moved one frame later. This inference method is an inference method that is very similar to the way humans think by applying fuzzy theory, and it is possible to judge a moving region at high speed and with high precision using a small amount of information such as edge information of each block. [0013] The image data of the moving area determined in this way is faithfully recorded and transmitted, and the image data of the static area is thinned out every few frames and recorded and transmitted, thereby reducing the memory capacity of the recording unit and improving the transmission speed. It is possible to aim for [0014] Furthermore, it is possible to determine whether it is a movement of the image or a shake of the imaging device based on the proportion of the moving area in the image, and if it is determined that it is a shake of the imaging device, the movement is reversed with respect to the entire image ** By adding vectors, blur can be corrected. [0015]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a block diagram showing an embodiment in which the present invention is applied to an imaging apparatus having a function of automatically tracking a moving subject. [0016] This device has an imaging section 1 made of a solid-state imaging device such as a CCD that converts a subject image formed through a lens into an electrical signal, and a video signal output from this imaging section 1 is sent to an AD conversion section 2. The image data is converted into digital image data and temporarily stored in the frame memory 3. The image data stored in the frame memory 3 is compressed and encoded by the signal processing section 4 and recorded in the recording section 5. [0017] Also, the output of the signal processing unit 4 is simultaneously converted into DCT
(Discrete Co51ne Transfer
m: Discrete cosine transform) unit 6. DCT
As shown in FIG. 2, the unit 6 performs two-dimensional DCT on each block of image data divided into a plurality of blocks each consisting of n×n pixels (for example, 8×8 pixels). [0018] DCT is a type of orthogonal transformation in the frequency domain, and converts one block of image data into fij (i,
j=o, 1°..., n-1), the obtained n×
n DCT coefficients Fuv(u, v=0.1...
, n-1) is defined as follows: Cw=1/205 (w=o) =1 (w≠0). [00191 FIG. 3 shows an 8×8 matrix of DCT coefficients Fuv. Each coefficient of the DCT coefficient Fuv represents a component obtained by decomposing one block of image data fij into spatial frequencies, and in the figure, the coefficient Foo at the upper left represents a value (DC component) proportional to the average value of the image data fij, As the variables U and V increase, they represent higher frequency components (AC components). Further, the coefficients below the coefficient Foo represent the magnitude of the high frequency component of the longitudinal wave, and the coefficients to the right thereof represent the magnitude of the high frequency component of the transverse wave. Therefore, by observing the DCT coefficient Fuv, it is possible to simultaneously observe the magnitudes of longitudinal waves, transverse waves, and their composite waves included in the image in the block. [00201 The DCT coefficients obtained by the DCT section 6 are supplied to the calculation section 7. The calculation unit 7 focuses on a specific coefficient among the DCT coefficients and outputs a control signal according to a change in this coefficient. The focus control section 8 receives this control signal and drives the motor within the imaging section 1 to move the position of the lens. [00211 Thus, the imaging unit 1. AD conversion section 2. Frame memory 3. Signal processing section 4. DCT section 6. The position of the lens is adjusted by a closed loop formed by the calculation unit 7 and the focus control unit 8, and the position where a specific coefficient has the maximum value is determined to be the in-focus position of the lens. [0022] By the way, the edge portions of an image generally contain high frequency components, but the frequency components of the obvious edge portions that form the outline of an object are, in the case of DCT coefficients, as shown in FIG.
The coefficient F1o in the shaded area appears in Fot. Therefore, this coefficient is set as a specific coefficient, and the position of the lens where this coefficient has the maximum value is set as the in-focus position. [0023] In the calculation unit 7, the coefficient F detected in this way
The maximum values of 1o and F o t are memorized, and if the newly input coefficient changes by more than a certain value, it is determined that the content of the subject has changed and the focusing operation is redone. [0024] Next, the motion area detection operation in the calculation unit 7 will be explained. Detection of a moving area, that is, detection of the amount and direction of movement of a subject, uses the above-mentioned specific coefficients as edge information of the block in question (hereinafter referred to as the block of interest), and uses this edge information and the same block that is input one frame later. This is performed by comparing the edge information of the block and its surrounding blocks (hereinafter referred to as comparison blocks). [0025] For this detection, fuzzy theory is applied and an inference method that is extremely close to human thinking is used. That is, 0
If the edge information of both blocks is similar and both blocks are located close to each other, it is considered that there is a high possibility that the subject of the block of interest has moved to the comparison block. ■So, what about other blocks? ■ Overall, the subject probably moved like this. As in,
Reasoning is based on the ambiguous thought process that humans use when performing pattern matching. [0026] Specifically, it is performed as follows. First, the edge information of the block of interest and the same block input one frame later are compared, and if the edge information of both blocks is clearly different, it is determined that the edge has moved to another block. Next, in order to know where the edge portion has moved, a block having edge information similar to the edge information of the block of interest is searched among the blocks surrounding the same block one frame later. [0027] At this time, if the edge information ratio of both blocks is less than a certain value δ (for example, 1/4) or more than a certain value γ (for example, 4/7), the edge information of both blocks is It is assumed that the objects are clearly different, and the probability that the objects in both blocks are the same is zero. It is also assumed that if the edge information of both blocks is exactly the same, the probability that the objects in both blocks are the same is 1. [00281 Assuming that the probability that the subject in both blocks is the same changes linearly with the ratio of edge information in both blocks, the probability (membership value) is calculated using the value δ and the value γ as the base, as shown in FIG. It becomes a mountain-shaped membership function with 1 as the apex. [0029] Furthermore, when focusing on the distance between both blocks and sequentially comparing the block of interest and blocks surrounding the same block one frame later, even if the edge information of both blocks has the same value, If the blocks are far apart from each other, it is considered that there is a low possibility that the subject moved to that block, and if the blocks are separated by a certain distance λ, it is considered a coincidence, and the probability that the subject moved there is considered to be low. Assume that is zero. [00301 If the distance is zero, that is, the same block has the same edge information, it is assumed that the subject remained there without moving, and the probability that the same subject is captured is set to 1. Assuming that the probability that the same subject is photographed decreases linearly as the distance increases, as shown in Figure 5,
It becomes a linear membership function connecting the distance λ and the probability (membership value) 1. [00311 The probability that satisfies both of these, that is, the lower probability, is considered to be the probability that the subject appearing in the block of interest was captured in the comparison block (that is, the probability that the subject moved to or remained in the comparison block). If this is calculated for all blocks surrounding the block of interest after one frame, a probability distribution as shown in FIG. 6 will be obtained. It is thought that the greatest possibility is that the object that was in the block of interest one frame ago has moved to the projection H of the center of gravity G of this distribution. Therefore, next, the movement of the subject is detected by using the H block as the block of interest and comparing it with the image signal one frame later. [0032] When recording image data in the recording unit 5, image data in a moving area is recorded faithfully, and image data in a still area is thinned out every several frames and recorded. Therefore, when the image is still, only shooting is performed and recording is in standby mode, and when the calculation unit 7 detects that the subject has moved, the recording unit records the image data of the frame. A control signal is sent to 5. [0033]Furthermore, according to the present invention, it is possible to determine whether the subject is moving or camera shake during photographing based on the ratio of the moving area to the entire screen. In other words, if the moving area is smaller than a certain percentage, it can be assumed that the subject is moving and the background remains unchanged. If the entire image is moving, it can be considered to be camera shake, and the camera shake can be corrected by correcting the entire screen in the opposite direction of the motion vector. [0034]

[Effects of the Invention] According to the present invention, a motion area can be determined by detecting an edge portion of an object from a specific high-frequency component in image data and inferring the movement of this edge portion in a form similar to human thinking. Therefore, it is possible to extract moving regions at high speed and with high accuracy using a small amount of information. [0035] Furthermore, according to the present invention, by recording and transmitting image data only when the subject moves, it is possible to reduce the memory capacity of the recording unit and improve the transmission speed. [0036] Furthermore, when the present invention is applied to an imaging device, it is possible to provide an imaging device with excellent operability by changing the focus area following the movement of the subject and continuing automatic focusing operation. can. In this case, since the direction and amount of movement (motion vector) of the entire screen are determined, if it is determined that the image is due to camera shake, the hand-held image can be corrected by taking into consideration the opposite motion vector.

[Brief explanation of the drawing]

FIG. 1 is a configuration diagram showing an embodiment in which the present invention is applied to an imaging device.

FIG. 2 is a diagram showing how one frame of image data is divided into blocks.

FIG. 3 is a diagram showing a matrix of two-dimensional DCT coefficients.

FIG. 4 is a membership function for the ratio of edge information.

FIG. 5 shows membership functions for distances between blocks.

FIG. 6 is a probability distribution that the subject of the block of interest has moved to another block.

[Explanation of symbols]

1 Imaging unit 2 AD conversion section 3 Frame memory 4 Signal processing section 5 Recording Department 6 DCT section 7 Arithmetic section 8 Focus control section

Claims (5)

[Claims] Claims: 1. One frame of image data is divided into a plurality of blocks each consisting of n×n pixels, and edge information of edge portions in each block is obtained from specific frequency components contained in each block. and compare the edge information of the block of interest in a certain frame with the edge information of the same block and its surrounding blocks one frame later,
Based on the probability that the edge information of the block of interest matches the edge information of each block one frame later, and the probability that the two blocks match with respect to the distance between the two blocks, the moving direction and movement of the edge portion in the block of interest are determined. A moving area extraction method characterized by determining the amount, extracting a moving area from the determination result, and separating a moving area and a still area in the image. 2. In claim 1, the edge information included in each block is obtained by subjecting image data of n×n pixels of the one block to two-dimensional discrete cosine transformation.
A moving area extraction method characterized by using a specific coefficient among ×n transformation coefficients. 3. The moving image forming apparatus according to claim 1, wherein the image data of the moving area is recorded and transmitted for each frame, and the image data of the still area is thinned out and recorded and transmitted every several frames. Region extraction method. 4. The moving area extraction method according to claim 1, wherein it is determined from the ratio of the moving area in the image whether the movement of the image is a movement of the image itself or a shake of the imaging means. 5. In claim 4, when the movement of the image is determined to be a shake of the imaging means, a motion vector opposite to the direction and amount of movement of the moving area is added to the image data. A moving area extraction method characterized by correcting the above-mentioned blurring.