KR102431386B1 - Method and system for interaction holographic display based on hand gesture recognition - Google Patents

Abstract

According to the present invention, in an interactive hologram display system based on hand gesture recognition, a sensor for gesture recognition, receiving depth image data from the sensor, generating hologram data based on hand gesture recognition, and displaying the generated hologram data in real time An interaction hologram display system including a hand gesture interaction processing device configured to transmit, and a hologram display device configured to provide a holographic display based on hologram data received from the hand gesture interaction processing device may be provided.

Description

Interaction hologram display method and system based on hand gesture recognition

The present invention relates to an interactive hologram display method and system based on hand gesture recognition. More specifically, it relates to a hologram display system capable of interacting with a user's hand gesture using deep learning and image processing for a depth image, and a method therefor.

Recently, there have been many attempts to interact with a device such as a computer or a mobile phone with a human hand. As a representative product, Leap Motion has excellent hand detection performance and allows you to control a computer through gesture interaction and play simple games through gesture interaction. For example, Leap Motion has developed a system that can be applied to a computer to control the computer without a mouse or keyboard, such as raising or lowering the volume and flipping the screen. In this way, by using gesture interaction, you can easily control other devices. Also, recently, people are taking pictures through the gesture of folding and unfolding the palm of the hand without touching the picture of the cell phone. As such, research on interaction with other devices through hand gesture recognition is being actively conducted.

In addition, 3D display technology for realistically experiencing content is being actively researched by global companies such as Microsoft and Google by launching 3D display products. 3D display technology can be largely divided into a method of wearing glasses and a method of wearing glasses. The method of wearing glasses is a method that can be seen only by wearing something in front of the eyes, such as glasses, and the glasses-free method is a method of observing a 3D display without a device worn in front of the eyes. Among the glasses-free methods, the digital hologram display method has the advantage that, unlike other glasses-free methods, fatigue due to focus-convergence mismatch does not occur.

The holographic display technology is a method that enables a digital holographic display to be observed from anywhere in 360 degrees. Since the holographic display is a method that uses light diffraction and interference, all lights must be turned off for observation. Because of these conditions, it can operate in a dark environment without light, and in order to control the image displayed on the hologram display, it must be controlled by a computer.

Therefore, there is a need for a new hand gesture recognition system that can easily control a hologram using gesture interaction in a dark environment.

Republic of Korea Patent Registration No. 10-1092909

An object of the present invention is to provide an interactive hologram display method and system for interacting a hologram display with a user's hand gesture using deep learning and image processing for a depth image.

Another object of the present invention is to provide an interactive hologram display method and system capable of accurately and real-time interaction with an intuitive gesture in a dark environment for observing the hologram display.

Another object of the present invention is to provide a method and system for displaying a hand gesture interaction hologram that can define and recognize a more intuitive gesture using joint information.

The problems to be solved of the present invention are not limited to the above-mentioned contents, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In one embodiment of the present invention, there is provided an interactive hologram display system based on hand gesture recognition, comprising: a sensor for gesture recognition; a hand gesture interaction processing device configured to receive depth image data from the sensor, generate hologram data based on hand gesture recognition, and transmit the generated hologram data in real time; and a hologram display device configured to provide a hologram display based on the hologram data received from the hand gesture interaction processing device.

In addition, the hand gesture interaction processing apparatus may include an interaction control unit configured to perform hand gesture recognition, and the interaction control unit may be configured to receive a depth image from the sensor.

In addition, the interaction controller may perform pre-processing of the depth image, and the pre-processing of the depth image may include a background subtraction process for removing depth information of a background and structures other than the user.

In addition, the background removal process may remove the background and the structure by using only depth information in which depth information between a first frame and subsequent frames exceeds a predetermined threshold value.

In addition, the pre-processing of the depth image may include an ROI (region of interest) setting process, and may be configured to detect a hand only within the set ROI.

In addition, the interaction control unit may acquire joint information of the entire hand through a deep learning model that has previously learned a data set of the image after the preprocessing.

In addition, the interaction control unit may be configured to set a bounding box based on the center of the hand based on joint information of the entire hand, and to perform gesture recognition when the movement of a finger crosses the bounding box. have.

In addition, the interaction control unit may be configured to recognize up, down, left, and right gestures based on whether the index or middle finger moves away from the center of the hand by more than a predetermined reference.

In addition, the interaction control unit may be configured to recognize an enlargement or reduction gesture based on whether the value of the distance accumulated in the arrangement changes by more than a predetermined reference using an arrangement that continuously stores the distance between the tip of the thumb and the index finger. have.

In addition, the hand gesture interaction processing apparatus may further include a real-time hologram data transmitter configured to transmit the hologram data to the hologram display apparatus.

In another embodiment of the present invention, there is provided a method for providing an interactive hologram display based on hand gesture recognition, the method comprising: acquiring depth image data from a gesture recognition sensor; Receiving depth image data from the gesture recognition sensor in a hand gesture interaction processing device, generating hologram data based on hand gesture recognition, and transmitting the generated hologram data in real time; and providing a hologram display in a hologram display device based on the hologram data received from the hand gesture interaction processing device.

In addition, the method further comprises the step of performing pre-processing of the depth image in the hand gesture interaction processing apparatus, wherein the pre-processing of the depth image is background removal ( background subtraction) processing may be included.

In addition, the background removal process may remove the background and the structure by using only depth information in which depth information between the previous frame and the next frame exceeds a predetermined threshold value.

In addition, the pre-processing of the depth image may further include an ROI (region of interest) setting processing step, and may be configured to detect a hand only within the set ROI.

In addition, the method may further include acquiring joint information of the entire hand through a deep learning model that has previously learned a data set from the pre-processed image in the hand gesture interaction processing apparatus.

In addition, in the hand gesture interaction processing device, a bounding box is set based on the center of the hand based on joint information of the entire hand, and gesture recognition is performed when the movement of the finger exceeds the bounding box. It may include further steps.

The method may further include recognizing up, down, left, and right gestures based on whether the index finger or middle finger moves away from the center of the hand by more than a predetermined reference in the hand gesture interaction processing apparatus.

In addition, by using an array that continuously stores the distance between the tip of the thumb and the index finger in the hand gesture interaction processing device, the zoom-in or zoom-out gesture is recognized based on whether the value of the distance accumulated in the array changes by more than a predetermined reference. It may include further steps.

Also, the hand gesture interaction processing apparatus may transmit hologram data to the hologram display apparatus through a real-time hologram data transmitter.

In another embodiment of the present invention, a computer-readable recording medium in which a program for implementing the above-described method is stored may be provided.

According to the present invention, the user can easily observe the interaction using the user's hand gesture, which is difficult to proceed in a dark environment observing the hologram display, by accurately interacting with the hologram in the desired shape with an intuitive gesture without other tools. have.

In addition, according to the present invention, it is possible to observe and control the appearance of the hologram display desired by the surgeon in a situation where it is difficult to touch something with his hand, for example, through a gesture only.

In addition, according to the present invention, it is possible to provide an interactive hologram display method and system for interacting a hologram display with a user's hand gesture using deep learning and image processing for a depth image.

Further, according to the present invention, it is possible to provide a method and system for displaying a hand gesture interaction hologram capable of defining and recognizing a more intuitive gesture using joint information.

Effects of the present invention are not limited to the above-mentioned contents, and other technical effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is an exemplary view showing an actual appearance of a hologram display device.
1 is a block diagram illustrating a gesture interaction operation according to an embodiment of the present invention.
2 is a block diagram illustrating an interactive hologram display system according to an embodiment of the present invention.
3 is an exemplary diagram for explaining a pre-processing of a depth image according to an embodiment of the present invention.
4 is an exemplary diagram illustrating a background removal result according to an embodiment of the present invention.
5 is a conceptual diagram for explaining the structure of a deep learning model CrossInfoNet for finding joint information of a hand using only depth information according to an embodiment of the present invention.
6 is an exemplary view for showing the ROI setting result, hand joint information, and the processing result of the bounding box according to an embodiment of the present invention.
7 is an exemplary diagram for explaining a gesture recognition operation for up, down, left, right, enlargement, reduction, and continuous rotation (left and right) according to an embodiment of the present invention.
8 is an exemplary view showing the state of the hologram in the basic rotation up, down, left, right, and basic state according to an embodiment of the present invention.
9 is an exemplary diagram illustrating a basic state and a state of a hologram that has been enlarged and reduced according to an embodiment of the present invention.
10A and 10B are result tables showing experimental results according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily carry out the present invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. And, in order to clearly describe the embodiment of the present invention in the drawings, parts irrelevant to the description are omitted.

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression may include the plural expression unless the context clearly dictates otherwise.

In the present specification, terms such as "comprise", "have" or "include" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one It may be understood that this does not preclude in advance the possibility of the presence or addition of other features or numbers, steps, operations, components, parts, or combinations thereof, or other features or more.

In addition, the components shown in the embodiment of the present invention are shown independently to represent different characteristic functions, and it does not mean that each component is made of separate hardware or a single software component. That is, each component is listed as each component for convenience of description, and at least two components of each component are combined to form one component, or one component can be divided into a plurality of components to perform a function. Integrated embodiments and separate embodiments of each of these components are also included in the scope of the present invention without departing from the essence of the present invention.

In addition, the following embodiments are provided to more clearly explain to those of ordinary skill in the art, and the shapes and sizes of elements in the drawings may be exaggerated for more clear description.

Hereinafter, with reference to the accompanying drawings, a preferred embodiment according to the present invention will be described.

1 is a block diagram illustrating a gesture interaction operation according to an embodiment of the present invention.

Most high-performance hand detection models use color information. However, since color information cannot be used in the holographic display environment, only the depth information is used, and only the necessary depth information is obtained through background subtraction and ROI setting processing in consideration of the characteristics of the hologram display installed in many other structures. A hand is detected, a gesture is defined using the joint information of the detected hand, and the gesture can be recognized and output when an appropriate gesture action is taken. In addition, the system can be designed to operate with a delay between gestures of 1 second.

More specifically, referring to FIG. 1 , first, depth image data may be acquired from a gesture recognition sensor such as a depth camera. (S110) For example, the gesture recognition sensor may use various depth cameras such as Azure Kinect. It may be a sensor product including

Pre-processing including background subtraction and region of interest (ROI) setting processing may be performed on the depth image obtained from the sensor (S120). By calculating the difference between the first frame taken while the camera is turned on and the subsequent frames, leaving only the depth information above a certain standard threshold and processing all the rest as 0, the depth information of structures and backgrounds other than the user in the image can be removed The purpose of the ROI setting step is to prevent hand estimation from depth information that may not be removed in the background subtraction step.

In addition, hand detection can be performed quickly and efficiently through ROI processing during pre-processing. For example, since the location of the hand where the user performs the actual hand motion is always constant, in order to prevent the depth information from being stolen by other structures, the location where the ROI is set is determined using a mouse click, and the location is fixed. By using it as an ROI area, it is possible to increase the processing speed by reducing the amount of data being processed by not looking for a hand outside the ROI.

Next, hand detection can be performed using the image received after pre-processing. (S130) For example, CrossInfoNet (a deep learning model that finds joint information of the hand) that has previously learned the NYU data set from the image received after the pre-processing performed above. ), the joint information of the hand can be found, and the structure of the deep learning model CrossInfoNet will be described later with reference to FIG.

In addition, using the joint information found, it is possible to set and draw a bounding box drawn around the user's hand, and this bounding box operates as a reference threshold to operate a gesture to cross the bounding box. Only then can the gesture be configured to be output.

Next, a gesture interaction is performed, so that the hologram display can be controlled based on the gesture interaction. (S140) For example, a gesture can be defined and a gesture can be recognized using joint information found with a CrossInfoNet deep learning model, etc. .

First, for example, in the case of up, down, left, and right gestures, the user's index and middle fingers can be defined as swiping in each direction, and the index and middle fingers are separated from the center of the hand by more than a reference threshold. Then, each gesture may be output.

Also, for example, in the case of an enlargement/reduction gesture, the distance information between the thumb and the index finger may be continuously stored in a heap-shaped array while the image is turned on. If the distance between the thumb and the index finger stored here increases, an enlargement gesture may be output, and when the distance between the thumb and index finger is decreased, a reduction gesture may be output.

In addition, for example, in the case of a continuous left and right rotation gesture, it can be defined as a form of swiping left and right while avoiding all hands. A right continuous rotation gesture may be output.

In this case, in order to avoid overlap between the up, down, left, and right gestures and the left and right continuous rotation gestures, the dot product value of the vector with the center of the hand for all five fingers may be used. The dot product value of the vector is changed to a value between -1 and 1 through normalization, and a value close to 1 is obtained when the finger is spread out, and a value close to -1 when the finger is bent. Duplicates between them can be removed.

The gesture output through this process communicates with the server through UDP/IP communication, for example, and when a binary code of the output gesture is transmitted, a corresponding command message is transmitted to the hologram display, and the hologram display, which was in the default state, corresponds to the corresponding command message. A gesture may be performed to show a new display, and this process may be performed in real time.

2 is a block diagram illustrating an interactive hologram display system according to an embodiment of the present invention.

The gesture recognition sensor 100 is a sensor for gesture recognition, and may be configured as a sensor including various depth cameras, such as Azure Kinect.

The hand gesture interaction processing apparatus 200 may be configured to receive depth image data from a sensor, generate hologram data based on hand gesture recognition, and transmit the generated hologram data in real time, a CGH generator 210, It may include a hologram data storage unit 220 , an interaction control unit 230 , and a real-time hologram data transmission unit 240 .

The CGH generator 210 is configured to generate a hologram through a computer generated hologram (CGH), and the generated hologram may be transferred to a file and stored in the hologram data storage 220 . The hologram data storage unit 220 may be composed of one or more memories of various types, and may be stored in a cloud server as necessary.

The interaction control unit 230 is configured to perform hand gesture recognition, and may receive 3D motion data including a depth image including a user's hand from the gesture recognition sensor 100 .

Here, the interaction control unit 230 may include a program or program module that may be executed by one or more processors. The program or program modules included in the interaction control unit 230 may be configured in the form of an operating system, an application program, or a program, and are physically stored on various types of widely used storage devices. can be Such programs or program modules may include one or more routines, subroutines, programs, objects, components, instructions, data structures, and specific tasks ( task) or may include, but are not limited to, various forms for executing specific data types.

The interaction controller 230 performs pre-processing of the depth image, and the pre-processing of the depth image may include background subtraction processing for removing depth information of a background and structures other than the user. Here, the background removal process may include a process of removing the background and the structure by using only depth information in which depth information between the first frame and subsequent frames exceeds a predetermined threshold value.

In addition, the interaction control unit 230 may acquire joint information of the entire hand through CrossInfoNet, a deep learning model that has previously learned a data set from the image after preprocessing. In addition, the interaction control unit 230 sets a bounding box based on the center of the hand based on joint information of the entire hand, and gesture recognition is performed when the movement of the finger crosses the bounding box. can

In addition, the interaction control unit 230 is configured to recognize the upper, lower, left, and right gestures based on whether the index or middle finger moves away from the center of the hand by more than a predetermined standard, and also between the thumb and the tip of the index finger It may be configured to recognize an enlargement or reduction gesture based on whether or not the value of the distance accumulated in the array changes by more than a predetermined reference using an array that continuously stores the distance of .

The real-time hologram data transmitter 240 may include a communicator configured to transmit hologram data, which is transmitted and controlled by the interaction controller 230 , to the hologram display device 300 . When the gesture result defined and recognized in the above process is input to the server using UDP/IP communication, etc., the server can transmit the corresponding hologram to the transmitter.

The hologram display device 300 may be configured to provide a hologram display based on hologram data received from the hand gesture interaction processing device. The transmitted hologram may read a signal received through the hologram display device 300 and display a hologram display based on a hand gesture interaction.

Through such a configuration, a user can observe a new hologram display through a gesture operation performed in real time, and by interacting with a hand located on the hologram display device 300, the 3D hologram display and the user's hand can interact. .

3 is an exemplary diagram for explaining a pre-processing of a depth image according to an embodiment of the present invention.

A method of using only depth information in which a difference in depth information between a first frame and subsequent frames exceeds a threshold value when an image is turned on will be described with reference to FIG. 3 . For example, when the camera is turned on, a frame with only a background or structure other than the user is received as the first frame, and in subsequent frames, the user enters the screen, and the depth difference value from the first frame is continuously calculated, The background and surrounding structures are removed using only the depth information of the image having the depth difference value exceeding the reference threshold, and the image may be an image in which only the user and the user's hand depth information can be used.

4 is an exemplary diagram illustrating a background removal result according to an embodiment of the present invention.

For example, the bottom part of the holographic display using the gesture recognition sensor 100 such as Azure Kinect is closer than the user's hand. Additionally, multiple cameras for eye tracking can be attached to holographic displays to trigger depth information, which can interfere with accurate hand detection. Therefore, since the depth information of the remaining parts except for the moving person must be removed, the depth information of the background and the structure can be deleted using the background removal process in the pre-processing process, and the left figure of FIG. 4 is a screen before the background removal process, and FIG. 4 is the screen after the background removal process.

5 is a conceptual diagram for explaining the structure of a deep learning model CrossInfoNet for finding joint information of a hand using only depth information according to an embodiment of the present invention.

According to an example of the present invention, a CrossInfoNet deep learning model was used to detect a user's hand. CrossInfoNet is a deep learning model that detects a hand using depth information. Unlike other existing models that detect the entire hand image and find joint information, CrossInfoNet detects the entire hand once and then re-apply each palm and finger. detection method is used.

Referring to FIG. 5 , first, the entire hand is detected to obtain approximate finger and palm joint information, and the acquired palm and finger joint information is again found in each other two branches elaborately, and again The elaborate joint information found is transmitted to each other's branches. In other words, the information on precisely finding the palm is transferred to the branch where the finger is precisely found, and the information precisely finding the finger is transferred to the branch where the palm is precisely found. In the branch in which the palm information is found precisely, there are the approximate palm joint information received through a skip connection, the elaborately found palm joint information, and the elaborate finger information shared. When this information is subtracted, rough palm information and elaborate palm information disappear, and only sophisticated finger information remains.

In this way, the joint information of the finger is acquired, and in the branch where the finger is found precisely, the subtracting process is performed using the approximate finger joint information received through skip connection, the precisely found finger joint information, and the shared elaborate palm information. can The result is sophisticated palm information.

Finger and palm information obtained through this process is continuously shared with each learning process, and as a result, high-accuracy finger and palm joint information is obtained. By combining the last finger and palm joint information, joint information of the entire hand can be acquired.

6 is an exemplary view for showing the ROI setting result, hand joint information, and the processing result of the bounding box according to an embodiment of the present invention.

Although the background and structures have been erased with the background subtraction process described above, some noise remains, so you can configure an ROI (Region of Interest) to find a hand only within the ROI. In the holographic display environment, the location where the camera and the user make a gesture is always constant, so the ROI can be set statically.

In addition, after detecting a hand, a bounding box, which is a bounding box, is set based on the center of the hand, and this bounding box is drawn based on the threshold used when recognizing a gesture, and the finger If it is passed, the corresponding gesture can be output.

Referring to the right drawing of FIG. 6 , a bounding box drawn centered on the user's hand can be set using the joint information found in the ROI, and this bounding box operates as a reference threshold, so that the gesture crosses the bounding box. It can be configured to output a gesture only by operating . In an embodiment, joint information of 14 hands including the center of the palm can be obtained by learning through the NYU hand dataset, and the hand and the bounding box detected along with the joint point can be checked as shown in FIG. 7 .

7 is an exemplary diagram for explaining a gesture recognition operation for up, down, left, right, enlargement, reduction, and continuous rotation (left and right) according to an embodiment of the present invention.

As shown in FIG. 6 , each gesture can be defined using the relative positional relationship of 14 joint information obtained as a result of hand detection, the dot product of each finger, and the distance between the fingers.

Gesture types include up, down, left, right, enlargement, reduction, and continuous rotation left and right. For up, down, left, and right gestures, when the relative positions of the upper, lower, left, and right of the middle finger with respect to the center of the hand exceed the reference threshold, and the dot product of the vector between the ring finger and the palm of each little finger is negative. A corresponding gesture may be output.

The zoom-in and zoom-out gestures can use a heap-shaped array that continuously stores the distance between the tips of the thumb and index finger. When the value of the distance accumulated in the array tends to increase to exceed the reference threshold value, an enlargement gesture may be output, and when the distance decreases to exceed the reference threshold value, a reduction gesture may be output.

In the continuous rotation left and right gesture, the value of the dot product of the vector with the palm for all five fingers exceeds the standard threshold, and unlike the basic left and right rotation, the ring finger and the little finger also have a relative position with the center of the palm. When the left and right reference thresholds are exceeded, the gesture may be output.

In this case, since the basic left and right gestures overlap with the continuous rotation left and right gestures, it is possible to distinguish them through the dot product of the vectors of the fingers and the palms, respectively. The dot product can use the dot product of each finger and the vector between the finger and the hand center. Here, the value of the dot product is normalized from -1 to 1. When the finger is bent, the value of the dot product becomes a negative value close to -1 due to the property of the dot product of the vector, and when the finger is bent, the value of the dot product of the vector becomes close to 1. You can use this to distinguish gestures by whether each of the five fingers is bent or extended.

For example, the hand gesture may be a total of eight basic rotation up, down, left, right, enlargement, reduction, and continuous rotation left and right. First, the basic rotation up, down, left, right can be divided into an operation of swiping (swipe) with the index and middle fingers straight. Intuitively, when swiping in up, down, left, and right directions, respectively, a gesture such as a swipe direction may be output. Second, the zoom-in/reduction gesture may be performed using the thumb and index finger. When the thumb and index finger are gathered and then spread apart, the zoom-in gesture and the zoom-out gesture may be output when the two fingers are spread apart and then gathered. Finally, the continuous rotation left and right may be recognized as a motion of swiping left and right with both palms extended.

8 is an exemplary view showing the state of the hologram in the basic rotation up, down, left, right, and basic state according to an embodiment of the present invention.

8 shows the results for a basic rotation gesture. In the default state, an up gesture rotates the holographic image upward, and a down gesture rotates it downward. Similarly, if you take a left and right gesture, you can see how it rotates to the left and right, respectively.

9 is an exemplary diagram illustrating a basic state and a state of a hologram that has been enlarged and reduced according to an embodiment of the present invention.

9 shows the results of the zoom-in and zoom-out gestures. In the default state, it can be seen that the hologram image becomes larger when the zoom-in gesture is taken, and the hologram image becomes smaller when the zoom-out gesture is taken.

10A and 10B are result tables showing experimental results according to an embodiment of the present invention.

An experiment was conducted with 10 people for a total of 8 types of gestures defined according to the present invention. All subjects recognized the gesture operation method and the delay of 1 second, and conducted 10 experiments per gesture. All subjects performed the experiment in the same test environment, and the experiment environment was conducted with the hand distance from Azure Kinect fixed at 35-50 cm, and the camera was viewed from the front. In addition, a total of 80 tests for 8 gestures were performed for each experimenter, that is, 100 experiments were performed for each gesture.

In the table shown in FIG. 10A, TP, FP, FN, Precision, Recall, and F1 scores for each gesture 100 times are shown. In the Up gesture, FN results were obtained twice, and in the zoom-in and zoom-out gestures, FP results were obtained once each. And in the continuously rotating Right gesture, 7 FN results were obtained. Using these results, precision, recall, and F1 score were calculated. All gestures except zoom-in and zoom-out got a Precision value of 100. The zoom-in and zoom-out gestures obtained a precision value of 99, respectively. All gestures except for the above and continuous rotation right gestures obtained a Recall value of 100. Up gesture got a Recall value of 98 and continuous rotation right gesture got a Recall value of 93. As for the F1 score calculated using precision and recall values, 98.98 for up gesture, 99.49 for zoom in/out gesture, 96.37 for continuous rotation right gesture, and 100 for the remaining gestures.

The various embodiments described herein may be implemented by hardware, middleware, microcode, software, and/or combinations thereof. For example, various embodiments may include one or more application specific semiconductors (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs). ), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions presented herein, or a combination thereof.

Also, for example, the various embodiments may be embodied in or encoded on a computer-readable medium comprising instructions. The instructions embodied in or encoded on a computer-readable medium may cause a programmable processor or other processor to perform a method, eg, when the instructions are executed. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium may be any available medium that can be accessed by a computer. For example, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage medium, magnetic disk storage medium or other magnetic storage device or desired program code, instructions or data accessible by a computer. may include any other medium that can be used to transport or store in the form of structures.

Such hardware, software, firmware, etc. may be implemented in the same device or in separate devices to support the various operations and functions described herein. Additionally, components, units, modules, components, etc. described as “parts” in the present invention may be implemented together or individually as separate but interoperable logic devices. Depictions of different features of modules, units, etc. are intended to emphasize different functional embodiments, and do not necessarily imply that they must be realized by separate hardware or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware or software components or integrated within common or separate hardware or software components.

Although acts are shown in the figures in a particular order, it should not be understood that these acts need to be performed in the particular order shown, or sequential order, or all shown acts need to be performed to achieve a desired result. . In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the division of various components in the above-described embodiments should not be construed as requiring such division in all embodiments, and that the described components will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there can be

Although the present invention has been described with reference to the embodiment shown in the drawings, which is merely exemplary, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: gesture recognition sensor
200: hand gesture interaction device
210: CGH generator
220: hologram data storage unit
230: interaction control
240: real-time hologram data transmitter
300: hologram display device

Claims (20)

An interactive hologram display system based on hand gesture recognition, comprising:
sensors for gesture recognition;
a hand gesture interaction processing device configured to receive depth image data from the sensor, generate hologram data based on hand gesture recognition, and transmit the generated hologram data in real time; and
A holographic display device configured to provide a holographic display based on holographic data received from the hand gesture interaction processing device
including,
The hand gesture interaction processing apparatus includes an interaction control unit configured to perform hand gesture recognition, wherein the interaction control unit is configured to receive a depth image from the sensor,
The interaction control unit sets a bounding box based on the center of the hand based on joint information of the entire hand, and is configured to perform gesture recognition when the movement of a finger crosses the bounding box,
When the upper, lower, left, and right relative positions of the middle finger with respect to the center of the hand exceed the reference threshold, and the dot product of the vector between the ring finger and the palm of each little finger is negative, the upper, lower, and left , which is configured to recognize a right gesture,
The interaction control unit is configured to recognize continuous rotation left and right gestures when the value of the dot product of the vector with the palm for all five fingers exceeds a reference threshold value,
The interaction control unit uses a heap-shaped array that continuously stores the distance between the tip of the thumb and the index finger, and recognizes an enlargement or reduction gesture based on whether the value of the distance accumulated in the array changes by more than a predetermined reference which is configured to do, an interactive hologram display system. delete The method of claim 1, wherein the interaction control unit performs pre-processing of the depth image, and the pre-processing of the depth image includes background subtraction processing for removing depth information of a background and structures other than a user. That is, an interactive hologram display system. The interactive hologram display system according to claim 3, wherein the background removal process removes the background and the structure by using only depth information in which depth information between a first frame and subsequent frames exceeds a predetermined threshold value. The interactive hologram display system according to claim 3, wherein the pre-processing of the depth image includes an ROI (region of interest) setting process, and is configured to detect a hand only within the set ROI. The interaction hologram display system according to claim 3, wherein the interaction control unit acquires joint information of the entire hand through a deep learning model that has previously learned a data set from the pre-processed image. delete delete delete The interaction hologram display system according to claim 1, wherein the hand gesture interaction processing device further comprises a real-time hologram data transmitter configured to transmit the hologram data to the hologram display device. A method for providing an interactive hologram display based on hand gesture recognition, the method comprising:
acquiring depth image data from a gesture recognition sensor;
Receiving depth image data from the gesture recognition sensor in a hand gesture interaction processing device, generating hologram data based on hand gesture recognition, and transmitting the generated hologram data in real time; and
providing a hologram display in a hologram display device based on the hologram data received from the hand gesture interaction processing device
including,
setting a bounding box based on the center of the hand based on joint information of the entire hand in the hand gesture interaction processing device, and performing gesture recognition when the movement of a finger exceeds the bounding box;
In the hand gesture interaction processing device, when the relative positions of the upper, lower, left, and right of the middle finger with respect to the center of the hand exceed the reference threshold, and the dot product of the vector of the palm of the ring finger and the palm of each little finger is negative, the upper, Recognizing down, left, and right gestures;
recognizing continuous rotation left and right gestures when a value of a vector dot product with a palm for all five fingers in the hand gesture interaction processing device exceeds a reference threshold value; and
Using a heap-shaped array that continuously stores the distance between the tip of the thumb and the index finger in the hand gesture interaction processing device, enlarged or reduced based on whether the value of the distance accumulated in the array changes by more than a predetermined reference Steps to recognize gestures
Interactive hologram display providing method further comprising. The method of claim 11 , further comprising: performing pre-processing of a depth image in the hand gesture interaction processing apparatus, wherein the pre-processing of the depth image includes removing depth information of backgrounds and structures other than the user A method for providing an interactive hologram display, comprising a background subtraction process for The method according to claim 12, wherein the background removal process removes the background and the structure by using only depth information in which depth information between a previous frame and subsequent frames exceeds a predetermined threshold value. The method of claim 13 , wherein the pre-processing of the depth image further comprises an ROI (region of interest) setting processing step, and is configured to detect a hand only within the set ROI. The method of claim 12 , further comprising: acquiring joint information of the entire hand through a deep learning model that has previously learned a data set from the pre-processed image in the hand gesture interaction processing device. delete delete delete The method of claim 11, wherein the hand gesture interaction processing device transmits hologram data to the hologram display device through a real-time hologram data transmitter. A computer-readable recording medium storing a program for implementing the method according to any one of claims 11 to 15 and 19.

Images