KR102784696B1 - Image processing device and image analyzing method - Google Patents

Abstract

The present embodiments disclose an image processing device and an image analysis method thereof.
Another image analysis method according to one embodiment of the present invention may include the steps of: selecting an image frame corresponding to a reference frame rate from a series of input image frames; dividing the selected image frame into a plurality of subframes; scaling the resolution of each of the subframes to correspond to the reference resolution; and time-division analyzing the scaled subframes.

Description

Image processing device and image analyzing method thereof {Image processing device and image analyzing method}

These embodiments relate to an image processing device and an image analysis method thereof.

Video analytics, which provides functions such as Motion Detection, Cross, and Entering provided by the camera, may be limited by the VA function that the device can provide, i.e., CPU or Codec. The performance of these video analytics functions is greatly affected by the resolution and frame rate, which are the data sizes of the input video, and the larger the resolution and the higher the frame rate for video analysis, the higher the CPU and memory required.

Japanese Patent No. 3785068

These embodiments provide a method for analyzing high-resolution images using limited VA resources (CPU, memory, etc.) equipped in a device.

An image analysis method of an image processing device according to one embodiment of the present invention comprises: a step of selecting an image frame corresponding to a reference frame rate from a series of input image frames; a step of dividing the selected image frame into a plurality of subframes; a step of scaling the resolution of each of the subframes to correspond to the reference resolution; and a step of time-division analyzing the scaled subframes.

In one embodiment, subframes in areas with higher weights may be more similar to the reference resolution.

In one embodiment, subframes in higher weighted regions may require increased analysis time.

In one embodiment, the division ratio and analysis time of the subframe can be determined based on the time division analysis results.

An image processing device according to one embodiment of the present invention includes a frame skipping unit for selecting an image frame corresponding to a reference frame rate from a series of input image frames; a scaling unit for dividing the selected image frame into a plurality of subframes and scaling the resolution of each of the subframes to correspond to the reference resolution; and an analysis unit for time-divisionally analyzing the scaled subframes.

These embodiments can provide image analysis functions at high resolution through time division of input images in cases where there are limited resources (CPU, memory) for processing image analysis functions.

FIG. 1 is a block diagram schematically illustrating an image processing device according to one embodiment of the present invention.
FIG. 2 is a block diagram schematically illustrating the configuration of an image processor according to one embodiment of the present invention.
Figures 3 to 5 are examples explaining image analysis according to an embodiment of the present invention.
Figure 6 is a flowchart schematically explaining an image analysis method according to one embodiment of the present invention.

Hereinafter, the present invention will be described in detail according to a preferred embodiment with reference to the attached drawings.

High-resolution cameras are greatly increasing their resolution and frame rates, such as encoding and outputting images of 4K, 12MP, and higher as input. However, increasing the CPU and memory resources required by the image analysis engine mounted on the camera to match the resolution and frame rate incurs greater costs.

For example, if the image analysis resource that can detect 10 faces for a 640x480@10FPS image is Ra, then to detect faces for a 4000x3000@30FPS image with a resolution of 12MP, about 40 times the resolution and 3 times the FPS processing capability are required, so 120xRa resources are required. As a result, 1200 face detection results can be produced. However, applying this many resources to an embedded system such as a camera is not suitable because it requires a lot of cost.

The embodiment of the present invention can analyze high-resolution images using limited VA resources (CPU, Memory) equipped in a camera. For example, if a VA engine has only 10FPS processing capability and 30FPS is input, two image frames are discarded and the third image frame is selected so that the VA engine can process it. In addition, since the higher the input resolution, the more CPU resources are required for VA processing, so it is converted into a small-sized image through scaling down. The smaller the area deteriorated by scaling down, the higher the accuracy. Therefore, the more similar the input image is to the size of the image processed by the VA engine, the lower the error rate in the image analysis result.

An embodiment of the present invention crops an input image into time-based, i.e., time-based, portions that can be processed by the VA engine and applies the cropped image to the VA engine as input. Since the scaled-down ratio of the reduced image input is smaller, the error rate of the image analysis result can be reduced.

However, a problem may occur in which the image analysis results that could be processed at 10 FPS in one area are reduced to 1 FPS. Accordingly, the present invention adjusts the area of the image input to the VA engine using the weight for the occurrence result, thereby reducing the result error due to the image analysis parallax.

FIG. 1 is a block diagram schematically illustrating an image processing device according to one embodiment of the present invention.

Referring to FIG. 1, the image processing device (1) may include an image sensor (10) and an image processor (30). The image processing device (1) may further include a display (50). In another embodiment, the image processing device (1) may be connected to the display (50) by wire or wirelessly.

The image processing device (1) may be various devices such as a surveillance camera including a visual camera, a thermal imaging camera, a special-purpose camera, a wireless communication device, a PDA (personal digital assistant), a laptop computer, a desktop computer, a camcorder, a digital camera, a CCTV, an action camera, a digital recording device, a network-enabled digital television, a mobile phone, a cellular phone, a satellite telephone, a camera phone, a two-way communication device, etc. Alternatively, the image processing device (1) may be an image processing system in which at least one of an image sensor (10) and an image processor (30) is implemented separately and is connected by wire or wirelessly to transmit and receive data.

The image sensor (10) may include a photoelectric conversion element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The image sensor (10) captures a scene in front to obtain image information. A lens (not shown) that receives an optical signal may be provided at the front of the image sensor (10).

The image processor (30) may be implemented with a variety of hardware and/or software configurations that execute specific functions. For example, the image processor (50) may mean a hardware-embedded data processing device having a physically structured circuit to execute a function expressed by a code or command included in a program. As an example of such a hardware-embedded data processing device, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like may be included, but the scope of the present invention is not limited thereto.

The image processor (30) can reduce noise in an image frame and perform signal processing for improving image quality, such as gamma correction, color filter array interpolation, color matrix, color correction, and color enhancement. In addition, the image processor (30) can compress image data generated by performing signal processing for improving image quality to generate an image file, or can restore image data from the image file. The compression format of the image includes a reversible format or an irreversible format. In addition, the image processor (30) can also functionally perform color processing, blur processing, edge emphasis processing, image interpretation processing, image recognition processing, and image effect processing. The image processor (50) can output an image frame that has undergone image processing to a memory and/or a display.

The image processor (30) can generate metadata that analyzes the image frame. The image processor (30) can divide a series of frames input according to the frame rate of the image sensor (10) into a plurality of subframes and analyze the subframes in a time-division manner. The image processor (30) can change the resolution and analysis time of the subframe by setting weights for each region of the image frame and determining the importance of the corresponding subframe.

Although not shown, the image processing device (1) may further include a storage means and a communication unit.

The storage means performs a function of temporarily or permanently storing data, instructions, programs, program codes, or a combination thereof processed by the image processor (30). The storage means can store image frames and metadata output from the image processor (30). The storage means may be a storage medium of at least one type among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, an SD or XD memory, etc.), a RAM, a ROM, and web storage that performs a storage function on the Internet.

The communication unit may be a device including hardware and software necessary to transmit and receive signals, such as control signals or data signals, through wired or wireless connections with external devices, but the scope of the present invention is not limited thereto.

The display (50) provides the user with a result image (e.g., a video sequence) that is a series of image frames output from the image processor (30), thereby allowing the user to monitor the displayed image. The display (50) can provide the user with visual information and/or auditory information.

FIG. 2 is a block diagram schematically illustrating the configuration of an image processor according to one embodiment of the present invention.

Referring to FIG. 2, the image processor (30) may include a first image processing unit (20) and a second image processing unit (40).

The first image processing unit (20) can analyze the input image frame and generate an event if the event occurrence condition set is satisfied. The event can include events set by the user, such as the appearance of an object, occurrence of a specific image (e.g., appearance of a face that cannot be recognized), occurrence of movement in a set area, occurrence of an abnormal sound source (e.g., car tire friction sound (skid), glass breaking sound, alarm sound, collision sound, etc.) in the case of a sound source, occurrence of a specific sound source (e.g., shouting, screaming, crying, etc.) by the user, occurrence of a sound exceeding a threshold value, etc., but is not limited thereto. The first image processing unit (20) can generate metadata including event information.

The first image processing unit (20) may include a frame skip unit (201), a first scaling unit (203), and an analysis unit (205).

The frame skip unit (201) can convert the input frame rate to a reference frame rate by skipping a portion of a series of input image frames. The input frame rate may be a frame rate output by the image sensor (10). The reference frame rate may be determined by the image analysis capability of the analysis unit (205). The frame skip unit (201) can lower the input frame rate to the reference frame rate by skipping image frames at a predetermined interval if the input frame rate is higher than the reference frame rate. The frame skip unit (201) can select and output image frames that have not been skipped.

The first scaling unit (203) can divide the image frame output from the frame skip unit (201) into a plurality of subframes and scale each subframe. Since the higher the resolution (size), the more resources are required for analysis, the first scaling unit (203) divides the image frame with a high resolution into a plurality of subframes with a low resolution.

The first scaling unit (203) can crop the image frame to generate a predetermined number of subframes. The first scaling unit (203) can crop the image frame based on a time that the analysis unit (205) can process. For example, the first scaling unit (203) can set the number of frames that the analysis unit (205) can process within a predetermined time as a maximum value and generate subframes having a number less than or equal to the maximum value. Each of the subframes can have the same or different resolution. The resolution of the subframe can be determined according to the importance corresponding to the weight set in the region cropped from the image frame to the subframe. The importance increases as the weight of the subframe in the region increases, so the resolution can be determined so that the scale down ratio described later is reduced.

The first scaling unit (203) can scale down each of the subframes to a predetermined resolution. The first scaling unit (203) can scale the subframes to a reference resolution that is optimal for image analysis by the analysis unit (205). The first scaling unit (203) can sequentially output the scaled down subframes to the analysis unit (205).

The analysis unit (205) can time-division analyze the input subframes within a given time period and output the analysis results. The analysis unit (205) can time-division analyze the subframes according to the analysis time and analysis order allocated to each of the subframes. The analysis time allocated to the subframes can be determined according to the weight of the area to which the subframe corresponds. The importance of a subframe in an area with a higher weight increases, so the analysis time increases and the number of analyses can increase within a given time period. The analysis results can be output as metadata. The weight can be determined by the event occurrence frequency or user specification. For example, the weight can be set higher in an area with a higher event occurrence frequency.

The user can change the video frame division ratio (i.e., the resolution of the subframe) and/or the analysis time per subframe based on the event distribution of the video frame according to the analysis result of the analysis unit (205).

The smaller the area degraded by scale down, the higher the accuracy of image analysis, so the closer the resolution of the subframe is to the reference resolution, the lower the scale down rate, which reduces the error rate of image analysis. Therefore, areas with high weights in the image frame may be cropped to a resolution similar to the reference resolution, and/or the analysis time may increase.

The second image processing unit (40) can encode an input image frame to generate a data stream.

The second image processing unit (40) may include a second scaling unit (401), an encoder (403), and a stream generation unit (405).

The second scaling unit (401) can scale down the input image frame to a desired resolution (size) using a resizer. The scale down ratio of the second scaling unit (401) can be the same as or different from the scale down ratio of the first scaling unit (401).

The encoder (403) can compress scaled down image frames to generate compressed images. The degree of compression can be proportional to the scale down ratio. Various compression methods such as JPEG, MJPEG, H.264, and H.265 can be applied.

The stream generation unit (405) can generate a data stream (DS) of a compressed image.

The image processing device (1) can generate and output a packet including metadata output from the analysis unit (205) and a data stream output from the stream generation unit (405).

Figures 3 to 5 are examples explaining image analysis according to an embodiment of the present invention.

Below, an example is described in which a video frame having a resolution of 4000x3000 is input at 30FPS to an image processor (30) capable of performing 10 face detections in a video frame when a video frame having a resolution of 640x480 is input at 10FPS.

Referring to FIG. 3, the frame skipping unit (201) can discard two video frames from three video frames and select one video frame to correspond to the analysis frame rate of the video processor (30).

The first scaling unit (203) can divide the selected image frame into 3x3 to generate 9 subframes. The subframes can include 6 subframes having a resolution of 1340x1000 and 3 subframes having a resolution of 1320x1000. The first scaling unit (203) can scale down the subframes to a resolution of 640x480 to reduce the number of pixels to about 1/5 or less. The first scaling unit (203) can divide the 9 scaled-down subframes into Scene1, Scene2, ..., Scene9 and output them to the analysis unit (205) in sequence.

The analysis unit (205) can perform face detection by dividing a set time based on the analysis frame rate of the image processor (30) into time periods according to the number of subframes and analyzing the subframes. For example, the analysis unit (205) can perform face detection by dividing time t into t1 to t9 corresponding to nine subframes and analyzing the corresponding subframes during each time interval. Here, t1 to t9 can each be 1/9 second. That is, the analysis unit (205) can output 90 face detection results per second as metadata (MD).

The display (50) decodes the data stream output from the image processor (30) and displays a restored image frame, and the image frame can display face detection information included in the metadata in each area corresponding to nine subframes.

Referring to FIG. 4, when there are many face detections in area A of a video frame and few face detections in area B, the user can change the split ratio of the video frame to increase the face detection accuracy in area A more than area B. Accordingly, the first scaling unit (203) can crop area A of the video frame into three areas with a resolution of 1000x1000 to generate three subframes (Scene2, Scene3, Scene7), crop area B into one area with a resolution of 3000x3000 to generate one subframe (Scene1), and crop the remaining areas into three areas with a resolution of 1000x1000 to generate three subframes (Scene4, Scene5, Scene6).

In addition, the user can set the analysis time of area A to be higher than the analysis time of the remaining areas. Accordingly, the three subframes (Scene2, Scene3, and Scene7) corresponding to area A are allocated twice the analysis time of the remaining subframes, so that the analysis unit (205) can perform face detection for the subframes of Scene1, Scene4, Scene5, and Scene6 during t1, t4, t5, and t6 (each 1/9 second), perform face detection for the subframe of Scene2 during t2 and t8 (total 2/9 seconds), perform face detection for the subframe of Scene3 during t3 and t9 (total 2/9 seconds), and perform face detection for the subframe of Scene7 during t7 and t10 (total 2/9 seconds).

That is, in the previous video frame, it was divided into 9 subframes and face detection was performed at equal time intervals, but in the current video frame, it is divided into 7 subframes and face detection can be performed for twice the time in some subframes due to two face detections.

Referring to Fig. 5, there is no face detection in areas C and E of the video frame, and there are many face detections in area D. The user can set the weight of area D to 0.8, and set the weights of areas C and E to 0.1, respectively. Accordingly, areas C and E can be divided so that face detection is performed at 1 FPS, and area D can be divided so that face detection is performed at 8 FPS.

The image analysis method of the present invention can be applied to various image analysis functions such as face detection, automatic number-plate recognition (ANPR), People Counting, Queue Length Counting, and Loitering.

Fig. 6 is a flow chart schematically explaining an image analysis method according to one embodiment of the present invention. The image analysis method illustrated in Fig. 6 can be performed by the image processing device illustrated in Fig. 1, and detailed descriptions of contents overlapping with those described above are omitted.

Referring to FIG. 6, the image processing device (1) can compress a series of image frames input from an image sensor (10) to create a data stream and simultaneously create metadata through analysis of the image frames.

For metadata generation, the image processing device (1) can convert the input frame rate into a reference frame rate by skipping a portion of a series of input image frames (S61). If the input frame rate is greater than the reference frame rate that the analysis engine can process, a lot of analysis resources are required, so the number of image frames to be analyzed can be reduced by skipping the image frames according to the reference frame rate. The number of image frames to be skipped can be determined by the reference frame rate.

The image processing device (1) can divide a non-skipped image frame into a plurality of subframes and scale each subframe (S63). Scaling is an operation of scaling down the resolution of the subframe to the reference resolution, since a large amount of analysis resources are required when the resolution of the subframe is greater than the reference resolution that the analysis engine can process. The lower the scale down ratio, the lower the image deterioration rate, so the resolution of the subframe corresponding to the area with a high weight can be similar to the reference resolution.

The image processing device (1) can analyze the scaled subframes in sequence (S65). The image processing device (1) can analyze the subframes according to the analysis time and analysis order allocated to each of the subframes. Therefore, the image frames have different analysis times and analysis accuracy for each region. The user can adjust the division ratio of the next image frame (i.e., the resolution of the subframe) and the analysis time of each subframe generated by the division based on the analysis result of the current image frame or the accumulated information of the analysis result of the image frame.

Embodiments of the present invention can improve the accuracy of a specific area (i.e., reduce the scale down ratio) by utilizing the distribution of the event results that occur, or improve the accuracy for an event by increasing or decreasing the time distribution.

Although the present invention has been described with reference to the embodiments shown in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be determined by the technical idea of the appended claims.

Claims (5)

A step of selecting a video frame from a series of input video frames so as to correspond to a reference frame rate;
A step of dividing the selected image frame into a plurality of subframes having a resolution;
A step of scaling down the resolution of each of the above subframes to a reference resolution; and
A step of time-division analyzing subframes scaled down to the above reference resolution;
An image analysis method of an image processing device, wherein the resolution of each subframe is determined in a step of dividing the selected image frame into a plurality of subframes so that the scale down ratio of a subframe with a higher importance is lower. In the first paragraph,
An image analysis method of an image processing device, wherein a subframe in an area with a high weight has a resolution similar to the reference resolution. In the first paragraph,
An image analysis method of an image processing device, in which the analysis time increases as the subframe of the area with a high weight increases. In the first paragraph,
An image analysis method of an image processing device, wherein the division ratio and analysis time of the subframe are determined based on the results of the above time division analysis. A frame skipping unit that selects a video frame from a series of input video frames so as to correspond to a reference frame rate;
A scaling unit that divides the selected image frame into a plurality of subframes having a resolution and scales down the resolution of each of the subframes to a reference resolution; and
An analysis unit for time-divisionally analyzing subframes scaled down to the above reference resolution is included;
An image processing device, wherein the resolution of each subframe is determined in a step of dividing the selected image frame into a plurality of subframes so that the scale down ratio of a subframe with a higher importance is lower.

Images