KR20140137564A - 3 Dimensional Image Sensing System for Human Tracking and Its Method - Google Patents

Abstract

The present invention relates to a three-dimensional image sensing system for a human figure and a method thereof, and more particularly, to a three-dimensional image sensing system for a human figure, which comprises a camera device for photographing a character image and a character image transmitted from the camera device, And an image sensing device for transmitting the camera control information to the camera device after extracting the camera control information, wherein the image sensing device comprises: a data receiver for receiving a person image acquired from the camera device; A data extracting unit for extracting a face recognition image from a portrait image transmitted from the data receiving unit to confirm face direction information, extracting depth information from the portrait image and separating skeleton data; A position correcting unit for matching the skeleton data with each landmark of the human body to convert the coordinate values of the camera coordinate system to coordinate values of the world coordinate system to perform position correction of the camera apparatus to obtain position correction data; A data analyzer for extracting and transmitting one or more pieces of person state information among position information of a current person, posture information of a skeleton, face direction information or motion information based on the face direction information, the skeleton data, and the position correction data; A controller for outputting camera control information for determining an operation state of the camera device based on the person state information transmitted from the data analyzer; And a data transmission unit for transmitting the camera control information to the camera apparatus. Therefore, the present invention can automatically photograph the position and behavior of the person while tracking the position of the person, and can record the contents of the lecture more effectively while tracking the position of the instructor, the position of the hand, The noise generated during the operation of the photographer or the camera operation is minimized, so that the deterioration of the concentration of the learner due to the camera operation noise generated during the lecture can be minimized.

Description

{3 Dimensional Image Sensing System for Human Tracking and Its Method}

The present invention relates to a three-dimensional image sensing system and method for tracking a person, and more particularly, to a system and method for three-dimensional image sensing using a three-dimensional person image, which extracts face direction information and posture information of a skeleton, Dimensional image sensing system and method for tracking a person, which can be automatically photographed without being tracked.

In recent years, video lectures through computers or televisions that do not have time and space constraints have been widely used instead of offline lectures that go directly to academies.

In this video lecture, a learner watches a video and learns it. When learning a video through a television, a user simply looks at the screen and hears the sound. In learning a computer-based video content, The user executes the input using the mouse and the keyboard, watches the monitor image, and hears the sound through the speaker.

Particularly, educational video contents recording video of instructor's lectures are being developed in recent years as a way of providing educational video contents to users through servers while the Internet has been developed and the number of Internet users is increasing.

In order to create video contents, a camera is installed in a lecture room in which a lecturer lectures in order to record the lecture contents of a lecturer, and a camera photographer skilled in the operation of a camera zoom in ) Or zoom out, and the instructor was photographed while changing the position of the camera to record the video of the lecture contents.

Such a conventional method of shooting a steel requires a photographer who is skilled in manipulating the camera to photograph the lecture contents of the lecturer. Since the photographer must take a picture while changing the position of the camera according to the motion of the lecturer, The photographer was inconvenient that he had to shoot the camera manually while very busy.

In order to solve the above problems, technologies for automatically tracing a lecturer and taking a lecture are proposed.

As a prior art document, the disclosure of Patent Document No. 10-2004-0107954 (published on December 23, 2004) discloses a description of an automatic instructor tracking system and a tracking method.

1 is a diagram illustrating an automatic trainer tracking system according to an embodiment of the present invention.

Referring to FIG. 1, the automatic trainer tracking system and tracing method includes a plurality of sensors 100 arranged in a line at a predetermined height and at regular intervals on one side of a pulley so as to detect the position of a lecturer, a zooming / And a controller 200 for receiving the sensing signal of the sensor 100 and moving the video camera 300 to the lecturer position, and a controller 200 for receiving the sensing signal of the sensor 100 and moving the video camera 300 to the lecturer position, .

In the automatic lecture tracking system and tracking method of the related art, a plurality of sensors 100 are constituted by infrared ray sensors. The infrared ray sensors include a light emitting portion and a light receiving portion formed integrally with each other, emitting infrared rays forward through a light emitting portion, The emitted infrared ray senses the object by receiving the infrared ray reflected from the object to the light receiving section.

The sensing distance of the infrared sensor is adjusted to within about 1.5m considering the front and rear width of the pulley through the adjustment volume. Further, in the case where a lectern is installed on the right or left side of the robots, the detection distance of the corresponding sensors is adjusted so as not to detect lectures.

However, it is difficult to precisely control the sensing distance of a general sensor, and a sensor that measures an accurate sensing distance is expensive, which is costly. In addition, since the position of a lecture can be changed, it is troublesome to adjust the detection distance of each sensor 100 every time the position of the strong end is changed.

The present invention extracts face direction information and posture information of a skeleton from a three-dimensional portrait image obtained by synthesizing a color image and a depth image, and automatically captures the position and behavior of the person while tracking the position and behavior of the person, It is possible to record lecture contents more effectively while tracing the position of the lecturer in the production of image contents, and to minimize the learner's concentration loss due to the noise generated during the lecture because the movement of the camera photographer or the noise generated during the camera operation is minimized A three-dimensional image sensing system for tracking a person and a method therefor.

Among the embodiments, the three-dimensional image sensing system for tracking characters includes a camera device for photographing a portrait image and a portrait image transmitted from the camera device, extracting person state information, and then transmitting camera control information to the camera device And an image sensing device. The image sensing apparatus may further include: a data receiving unit that receives a portrait image acquired from the camera apparatus; A data extracting unit for extracting a face recognition image from a portrait image transmitted from the data receiving unit to confirm face direction information, extracting depth information from the portrait image and separating skeleton data; A position correcting unit for matching the skeleton data with each landmark of the human body to convert the coordinate values of the camera coordinate system to coordinate values of the world coordinate system to perform position correction of the camera apparatus to obtain position correction data; A data analyzer for extracting and transmitting one or more pieces of person state information among position information of a current person, posture information of a skeleton, face direction information or motion information based on the face direction information, the skeleton data, and the position correction data; A controller for outputting camera control information for determining an operation state of the camera device based on the person state information transmitted from the data analyzer; And a data transmission unit for transmitting the camera control information to the camera apparatus.

The camera device may include at least one of a color (RGB) image sensor, an infrared sensor, and a depth sensor, and a color image and a depth image may be synthesized through the at least one sensor to capture and transmit a three- .

The data extracting unit may include a face recognizing unit for recognizing face direction information by detecting a facial color area in the portrait image to generate a face recognition image, selecting an eye area in the face recognition image and detecting a direction to the eye area, And a skeleton extracting unit for extracting a depth image by estimating a geometric change of each landmark of the human body or depth information using an additional sensor in the portrait image, and separating the skeleton data from the depth image.

The control unit may further include a PTZ control module for generating PTZ (Pan Tilt Zoom) control information according to the position and behavior information of the person based on the person state information. The PTZ control information and the off- , Operation state information for determining any one of an operation state, a waiting operation, a recognition operation, and a tracking operation, and generates and outputs camera control information.

Among the embodiments, the three-dimensional image sensing method for tracking a person includes an image sensing device for analyzing a portrait image transmitted from a camera device for capturing a portrait image to extract person state information, and then transmitting camera control information to the camera device A method of image sensing for tracking a person, the method comprising: receiving a character image acquired from the camera device; Extracting a face recognition image from the portrait image to confirm face direction information, extracting depth information from the portrait image, and separating skeleton data;

A position correcting step of matching the skeleton data with each landmark of the human body to convert the coordinate values of the camera coordinate system into coordinate values of the world coordinate system to obtain the position correction data of the camera device; Extracting and transmitting one or more pieces of person state information among position information of a current person, posture information of a skeleton, face direction information, or motion information based on the face direction information, the skeleton data, and the position correction data; A control step of outputting camera control information for determining an operation state of any one of an off operation, a standby operation, a recognition operation, and a tracking operation based on the person state information; And a data transmitting step of transmitting the camera control information to the camera apparatus.

The position correction step may include: a correction judgment step of checking whether the position of the camera device is corrected by matching the skeleton data with each landmark of the human body; And a correction unit that corrects the position of each component in the camera apparatus based on a reference correction value when it is determined that the position correction is necessary in the correction determination step and corrects the correction coefficient obtained through the correction to the correction determination step Performing step; And a data acquiring step of acquiring position correction data of the camera apparatus by converting the coordinate value of the camera coordinate system into the coordinate value of the world coordinate system when it is determined that the position correction is not necessary in the correction judgment step have.

The control step may further include PTZ (Pan Tilt Zoom) control information calculated based on the position and behavior information of the person based on the person state information in the camera control information.

The three-dimensional image sensing system and method of the present invention extracts face direction information and skeletal attitude information from a three-dimensional face image synthesized with a color image and a depth image, It is possible to capture the contents of the lecture more effectively while tracking the location of the instructor, the position of the hand and the direction of the face in the production of the educational image contents, So that the deterioration of the concentration of the learner due to the camera operation noise generated during the lecture can be minimized.

1 is a diagram illustrating an automatic trainer tracking system according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a person tracking image sensing system according to an exemplary embodiment of the present invention.
FIG. 3 is a view for explaining the face direction recognition process of the face recognition unit of FIG. 1;
4 is a flowchart illustrating an image sensing method for tracking a person according to an exemplary embodiment of the present invention.

The description of the present invention is merely an example for structural or functional explanation, and the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, the embodiments are to be construed as being variously embodied and having various forms, so that the scope of the present invention should be understood to include equivalents capable of realizing technical ideas. Also, the purpose or effect of the present invention should not be construed as limiting the scope of the present invention, since it does not mean that a specific embodiment should include all or only such effect.

Meanwhile, the meaning of the terms described in the present invention should be understood as follows.

The terms "first "," second ", and the like are intended to distinguish one element from another, and the scope of the right should not be limited by these terms. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" to another element, it may be directly connected to the other element, but there may be other elements in between. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that there are no other elements in between. On the other hand, other expressions that describe the relationship between components, such as "between" and "between" or "neighboring to" and "directly adjacent to" should be interpreted as well.

It should be understood that the singular " include "or" have "are to be construed as including a stated feature, number, step, operation, component, It is to be understood that the combination is intended to specify that it does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Commonly used predefined terms should be interpreted to be consistent with the meanings in the context of the related art and can not be interpreted as having ideal or overly formal meaning unless explicitly defined in the present invention.

FIG. 2 is a block diagram for explaining a configuration of a person tracking image sensing system according to an embodiment of the present invention, and FIG. 3 is a view for explaining a face orientation recognition process of the face recognition unit of FIG.

Referring to FIG. 2, the person tracking image sensing system includes a camera device 110 and an image sensing device 120.

The camera device 110 includes at least one of a color (RGB) image sensor 111, an infrared sensor 112, or a depth sensor 113, And transmits the generated three-dimensional portrait image to the image sensing device 120. [0033] FIG.

The image sensing device 120 analyzes the three-dimensional portrait image transmitted from the camera device 110 and extracts the person state information including the current position of the person, posture information of the skeleton, face direction information, and behavior information, Generates control information based on the status information, and transmits the control information to the camera device 110 so that the PTZ control and the operation state of the camera device 110 are controlled.

The image sensing apparatus 120 includes a data receiving unit 121, a data extracting unit 122, a position correcting unit 123, a data analyzing unit 124, a controller 125 and a data transmitting unit 126.

The data receiving unit 121 receives the three-dimensional character image obtained from the camera device 110 and transmits the three-dimensional character image to the data extracting unit 122.

The data extracting unit 122 extracts the face recognition image and the depth image by receiving the three-dimensional person image from the data receiving unit 121. The data extracting unit 122 includes a face recognizing unit 122a and a skeleton extracting unit 122b.

The face recognition unit 122a detects a face color region in the portrait image, generates a face recognition image by distinguishing the face of the person from the surrounding image, selects the eye region in the face recognition image, Thereby confirming the face direction information.

That is, as shown in FIG. 3, the face recognizing unit 122a selects the eye region in the face recognition image and determines that the face direction is the front direction when the two eyes face the front face (refer to (a) When one eye is pointing to the right in the eye region, the face direction is determined to be the right direction (see (b)), and when one eye is pointing to the left in the eye region, the face direction is determined as the left direction ( c)

At this time, the face recognizing unit 122a stores facial color data including facial color feature information for each country in order to detect the facial color area, and extracts only the facial color area corresponding to the facial color data in the three-dimensional portrait image.

The skeleton extracting unit 122b extracts depth information by extracting the depth image using the geometric changes of the land-marks of the human body in the portrait image, and separates the skeleton data from the depth information.

The position correcting unit 123 matches ten or more skeleton data obtained from the skeleton extracting unit 122b with each landmark of the human body and performs a linear transformation on the coordinates of the camera coordinate system, And obtains the position correction data by correcting the positions of the sensors 111, 112 and 113 in the camera apparatus. The position correcting unit 123 can obtain the position correction data such as the movement and the angle of the camera device 110 by an algorithm using a conversion relation between a general camera coordinate system and a world coordinate system.

The data analyzing unit 124 analyzes the position information of the current person based on the face direction information, the skeleton data, and the position correction data from the face recognizing unit 122a, the skeleton extracting unit 122b and the position correcting unit 123, Position information, face orientation information, or motion information, and transmits the extracted person state information to the controller 125.

The control unit 125 controls the camera device 110 based on the person state information transmitted from the data analysis unit 124 to obtain camera control information for determining the operation state of the camera device 110, Output.

The control unit 125 may further include a PTZ control module (not shown) for generating PTZ (Pan Tilt Zoom) control information according to the position and behavior information of the person, As well as PTZ control information.

The data transmitting section 126 transmits the camera control information to the camera apparatus 110. [

The image sensing device 120 may further include a data extractor 122, a data analyzer 124, and a memory (not shown) that stores data transmitted and received by the controller 125.

4 is a flowchart illustrating an image sensing method for tracking a person according to an exemplary embodiment of the present invention.

Referring to FIG. 4, the image sensing apparatus for tracking a person receives a three-dimensional portrait image obtained by synthesizing a color image and a depth image from a camera apparatus 110, The recognition image and the depth image are extracted (steps S1 and S2)

The image sensing apparatus 120 detects the eye region in the face recognition image and extracts the face direction information by confirming whether the face direction is the front, right, or left direction (Step S3)

The image sensing apparatus 120 extracts skeleton data using the API provided by the depth sensor 113 from the portrait image. (Step S4) The image sensing apparatus 120 receives skeleton data including ten skeleton information from the human body And converts the coordinate values of the camera coordinate system to coordinate values of the world coordinate system to determine whether the positions of the sensors 111, 112, and 113 need to be corrected (step S5)

When it is determined that the position correction is required, the image sensing device 120 corrects the position of the sensor 111, 112, 113 or the PTZ driving device based on a reference correction value previously inputted, It is again determined whether the position correction of the sensors 111, 112, 113 or the PTZ driving device is necessary by using the obtained correction coefficient (steps S6 to S8)

The image sensing device 120 acquires the position correction data when it is determined that the position correction of the sensors 111, 112, 113 or the PTZ driving device is not necessary, and integrates and analyzes the face orientation information, skeleton data, (Step S9), the position information of the current person, the posture information of the skeleton including the position information of the hand, the face direction information, or the motion information.

The image sensing apparatus 120 outputs camera control information including operation state information for determining the operation state of any one of the OFF operation, the standby operation, the recognition operation, and the tracking operation of the camera apparatus 110 based on the person state information (Step S10)

At this time, the image sensing apparatus 120 may further include PTZ (Pan Tilt Zoom) control information calculated based on the position and behavior information of the person based on the person state information in the camera control information.

The image sensing device 120 may transmit the camera control information to the camera device 110 so that the operation state of the camera device 110 can be changed according to the location where the person is located (step S11)

As described above, the system and method for tracking a human figure according to an embodiment of the present invention can more effectively capture the content of a lecture while tracing a position of a lecturer, a position of a hand, and a face direction of a lecturer.

For example, the image tracking system and method for tracing a person traces the position of a teacher's hand during a lecture, and when a specific learner is instructed, the image of the learner can be photographed. The face direction of the lecturer is automatically detected, Voice or facial expressions of the teacher, or the board or instructional materials used by the instructor.

In addition, the image sensing system and method for tracking a person can automatically track the instructor by controlling the camera device 110 according to the position and behavior of a lecturer, so that the noise of the camera photographer, So that it is possible to minimize deterioration of concentration of the learner due to the camera operation noise generated during the lecture.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

110: camera device 120: image sensing device
121: Data receiving unit 122: Data extracting unit
123: position correcting unit 124: data analyzing unit
125: control unit 126: data transmission unit

Claims (7)

And an image sensing device for analyzing a character image transmitted from the camera device to extract person state information and then transmitting camera control information to the camera device, the system comprising:
The image sensing apparatus includes:
A data receiving unit for receiving a portrait image acquired from the camera device;
A data extracting unit for extracting a face recognition image from a portrait image transmitted from the data receiving unit to confirm face direction information, extracting depth information from the portrait image and separating skeleton data;
A position correcting unit for matching the skeleton data with each landmark of the human body to convert the coordinate values of the camera coordinate system to coordinate values of the world coordinate system to perform position correction of the camera apparatus to obtain position correction data;
A data analyzer for extracting and transmitting one or more pieces of person state information among position information of a current person, posture information of a skeleton, face direction information or motion information based on the face direction information, the skeleton data, and the position correction data;
A controller for outputting camera control information for determining an operation state of the camera device based on the person state information transmitted from the data analyzer; And
And a data transmission unit for transmitting the camera control information to the camera device.
The camera device according to claim 1,
A color image (RGB) image sensor, an infrared sensor, and a depth sensor, wherein the color image and the depth image are synthesized through the at least one sensor, and the three-dimensional person image is captured and transmitted 3D image sensing system for character tracking.
The apparatus of claim 1,
A face recognizing unit for recognizing facial direction information by detecting a face color region in the portrait image to generate a face recognition image, selecting an eye region in the face recognition image, and detecting a direction of the eye region; And
And a skeleton extracting unit for extracting depth information by estimating a geometric change of each landmark of the human body or a depth image using an additional sensor in the human image and separating the skeleton data from the depth information, 3D image sensing system for character tracking.
The apparatus of claim 1,
And a PTZ control module for generating PTZ (Pan Tilt Zoom) control information according to the position and behavior information of the person based on the person state information,
Wherein the PTZ control information includes operation state information for determining one of an operation state of the PTZ control information, a turn-off operation of the camera apparatus, a standby operation, a recognition operation, and a tracking operation. Dimensional image sensing system.
A person tracking image sensing method performed by an image sensing device for analyzing a portrait image transmitted from a camera device for capturing a portrait image and extracting person state information and then transmitting camera control information to the camera device,
A data receiving step of receiving a portrait image acquired from the camera device;
Extracting a face recognition image from the portrait image to confirm face direction information, extracting depth information from the portrait image, and separating skeleton data;
A position correcting step of matching the skeleton data with each landmark of the human body to convert the coordinate values of the camera coordinate system into coordinate values of the world coordinate system to obtain the position correction data of the camera device;
Extracting and transmitting one or more pieces of person state information among position information of a current person, posture information of a skeleton, face direction information, or motion information based on the face direction information, the skeleton data, and the position correction data;
A control step of outputting camera control information for determining an operation state of any one of an off operation, a standby operation, a recognition operation, and a tracking operation based on the person state information; And
And a data transmitting step of transmitting the camera control information to the camera apparatus.
6. The method according to claim 5,
A correction judgment step of checking whether the position of the camera apparatus is corrected by matching the skeleton data to each landmark of the human body;
And a correction unit that corrects the position of each component in the camera apparatus based on a reference correction value when it is determined that the position correction is necessary in the correction determination step and corrects the correction coefficient obtained through the correction to the correction determination step Performing step; And
And a data acquiring step of acquiring position correction data of the camera apparatus by converting a coordinate value of the camera coordinate system into a coordinate value of a world coordinate system when it is determined that the position correction is not required in the correction determination step, A three dimensional image sensing method for character tracking.
6. The method according to claim 5,
Wherein the camera control information further includes PTZ (Pan Tilt Zoom) control information calculated based on the position and behavior information of the person based on the person state information.

Images