KR102268337B1 - Augmented Reality-based performance video viewing system and performance image providing method using it - Google Patents

Abstract

The present invention relates to, and according to the present invention, an AR-based performance video viewing system according to an embodiment of the present invention is a system for viewing a performance as a performance image including a real performance image and an augmented reality (AR) performance. In the audience terminal that can shoot a performance; AR-based performance including an AR-based viewing server that transmits a performance image to the audience terminal when a performance image is received from the audience terminal, and an LED protocol board installed in the performance hall to output regular signals and specific signals during performances A video viewing system may be provided.
In addition, in a method of providing a performance image including an actual performance image and an augmented reality (AR) performance to the audience using an AR-based performance image viewing system, (a) receiving a performance image photographed from the audience terminal receiving; (b) recognizing a regular signal from the received performance image to recognize the performance progress and the performance venue; (c) recognizing one or more LED objects in the performance image; (d) generating effect setting information by recognizing a specific signal in the actual performance image, setting which LED object to display the effect of AR performance according to the specific signal, and (e) AR according to the effect setting information It is possible to provide a method for providing a performance image using an AR-based performance image viewing system, comprising the step of transmitting a performance to the audience terminal and providing the performance image by implementing the AR performance on the actual performance image.

Description

AR-based performance video viewing system and performance image providing method using same {Augmented Reality-based performance video viewing system and performance image providing method using it}

The present invention relates to an AR-based performance video viewing system and a method for providing a performance video using the same, by fusing augmented reality (AR) performance with an offline LED robot performance to increase immersion and create dynamics, and by using an LED protocol board It relates to an AR-based performance video viewing system that effectively synchronizes an offline LED robot performance and an AR performance, and a method for providing a performance video using the same.

Various robot performances are being operated mainly by schools, but most are provided through the National Exhibition Center, and there are only a few performances of about 4 times a day.

At this time, the content is also like a robot dance, and the content update rate is low due to a simple performance performed in the past.

On the other hand, augmented reality and virtual reality are technologies that realize a reality that does not actually exist and allow humans to perceive it and have been growing rapidly since 2010. Leading companies are focusing on securing technological competitiveness and promoting demand for ecosystem construction. trying to expand.

However, the current augmented reality market has no remarkable results except for some markets such as games.

In addition, AR performances that give stage effects by using augmented reality in fields such as performances and performances have been developed, but the AR technology used in conventional AR performances implements virtual performance after checking input values through image sensing. There was a problem in that the quality of the AR performance deteriorated because there was a delay that was delayed compared to the performance, and more data was collected than necessary due to the increased sensitivity of image sensing.

Therefore, the synchronization between the performance and the AR performance is effectively performed, and it is necessary to develop a technology that can increase the immersion and the dynamics of the performance by applying the augmented reality.

In order to solve the above problems, the present invention combines augmented reality (AR) performance with offline LED robot performances to increase immersion and dynamics, and use an LED protocol board to create a relationship between offline LED robot performance and AR performance. An object of the present invention is to provide an AR-based performance video viewing system that synchronizes effectively and a method for providing a performance video using the same.

In order to solve the above problems, the AR-based performance image viewing system according to an embodiment of the present invention is a system for viewing a performance as a performance image including a real performance image and an augmented reality (AR) performance, a visitor terminal capable of photographing; AR-based performance including an AR-based viewing server that transmits a performance image to the audience terminal when a performance image is received from the audience terminal, and an LED protocol board installed in the performance hall to output regular signals and specific signals during performances A video viewing system may be provided.

Here, the AR-based viewing server includes a database for storing a plurality of AR performances; a performance recognition unit receiving the actual performance image from the audience terminal, and recognizing a signal from the actual performance image at all times to recognize the performance progress and the performance venue; an object recognition unit for recognizing one or more LED objects in the real image of the performance hall recognized; When a specific signal is recognized in the real image of the performance, a setting unit that generates effect setting information by setting which LED object to display the effect of AR performance among one or more LED objects recognized according to the specific signal and AR according to the effect setting information It may include an image output unit for transmitting a performance to the audience terminal and providing the performance image by implementing the AR performance on the actual image of the performance.

In addition, the AR-based viewing server may further include a control unit that transmits AR performance setting information to the LED protocol board when AR performance setting information is received according to a performance from the manager terminal.

In addition, the AR-based viewing server may further include a mission unit that provides mission information to the visitor terminal, and transmits mission achievement information to the visitor terminal when a function corresponding to the mission information is performed from the visitor terminal.

In addition, the object recognition unit converts the actual performance image into a gray image, performs black-and-white processing on the actual performance image converted into a gray image using a threshold, and canny edge detector (Canny) from the black-and-white processed performance image It is characterized in that it recognizes the LED object by detecting the outline using Edge Detector).

In addition, when the object recognition unit detects an outline and recognizes an LED object, it removes noise through blurring using a Gaussian mask from the black-and-white-processed performance real image, and uses a Sobel mask to remove the edge , calculates the gradient size and direction of the detected edge, sharpens the edge by removing low pixel values, and detects the outline by selecting and connecting important edges.

In addition, the LED protocol board may include a constant signal unit for outputting a constant signal and a specific signal unit for outputting a specific signal according to an AR performance set according to a performance.

In addition, the AR-based performance video viewing system can set the AR performance and transmit the AR performance setting information to the AR-based viewing server, receive the AR performance and store it in the database, and receive the performance start information to the AR-based viewing server It may further include the manager terminal to transmit to.

In addition, the method for providing a performance image using an AR-based performance image viewing system according to an embodiment of the present invention provides a performance image including a real performance image and an augmented reality (AR) performance to the audience using the AR-based performance image viewing system. A method of providing, comprising the steps of: (a) receiving an actual image of a performance photographed from a visitor terminal; (b) recognizing a regular signal from the received performance image to recognize the performance progress and the performance venue; (c) recognizing one or more LED objects in the performance image; (d) generating effect setting information by recognizing a specific signal in the actual performance image, setting which LED object to display the effect of AR performance according to the specific signal, and (e) AR according to the effect setting information It is possible to provide a method for providing a performance image using an AR-based performance image viewing system, comprising the step of transmitting a performance to the audience terminal and providing the performance image by implementing the AR performance on the actual performance image.

In addition, the step (c), converting the actual performance image into a gray image; It may include the step of black-and-white processing of the actual performance image converted into a gray image using a threshold, and the step of detecting an outline using a Canny Edge Detector in the black-and-white-processed actual performance image.

In addition, the step of detecting the outline using the Canny Edge Detector (Canny Edge Detector), removing the noise through the blurring (blurring) using a Gaussian mask from the black-and-white-processed real image; detecting an edge using a Sobel mask; calculating a gradient magnitude and direction of the detected edge; The method may include sharpening an edge by removing a low pixel value, and detecting an outline by selecting and connecting important edges.

As described above, the AR-based performance video viewing system and the performance video providing method using the same according to an embodiment of the present invention can produce increased immersion and dynamics by fusing augmented reality (AR) performance with offline LED robot performances provided by the science center. can

In addition, by using the LED protocol board, the synchronization between the offline LED robot performance and the AR performance can be effectively achieved to provide a high-quality performance image.

This can increase the satisfaction and demand of visitors.

It is also expected to paint a new blueprint for the future for customers, whose main target is students.

In addition, it is expected that it can be applied to various exhibitions and performances.

1 is an example of using an AR-based performance video viewing system according to an embodiment of the present invention.
2 is a block diagram illustrating the configuration of an AR-based performance video viewing system according to an embodiment of the present invention.
3 is a block diagram illustrating the configuration of an AR-based viewing server of an AR-based performance video viewing system according to an embodiment of the present invention.
4 is a front view showing an LED protocol board of an AR-based performance video viewing system according to an embodiment of the present invention.
5 is a flowchart sequentially illustrating a method for providing a performance video using an AR-based performance video viewing system according to an embodiment of the present invention.
6 is a flowchart sequentially illustrating steps S300 of FIG. 5;
7 is a flowchart sequentially illustrating steps S330 of FIG. 6;

Hereinafter, the description of the present invention with reference to the drawings is not limited to specific embodiments, and various modifications may be made and various embodiments may be provided. In addition, it should be understood that the contents described below include all conversions, equivalents, and substitutes included in the spirit and scope of the present invention.

In the following description, terms such as first and second are terms used to describe various components, meanings are not limited thereto, and are used only for the purpose of distinguishing one component from other components.

Like reference numbers used throughout this specification refer to like elements.

As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In addition, terms such as "comprises", "comprising" or "have" described below are intended to designate the existence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. It should be construed as not precluding the possibility of addition or existence of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

The AR-based performance video viewing system of the present invention is provided in a mobile communication terminal including a smart phone that communicates through a wireless communication network using an application (or mobile app) and a computer environment that communicates through a wired communication network. Hereinafter, for convenience of description, a mobile communication terminal will be mainly described.

The present invention relates to an AR-based performance video viewing system and a method for providing a performance video using the same, and it is intended to enhance immersion and create dynamics by fusion of augmented reality (AR) performance with offline LED robot performances provided in science halls, etc.

In addition, by using the LED protocol board, it is possible to provide high-quality performance video by effectively synchronizing the offline LED robot performance and AR performance.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings 1 to 7 .

1 is an example of using an AR-based performance video viewing system according to an embodiment of the present invention, FIG. 2 is a block diagram illustrating the configuration of an AR-based performance video viewing system according to an embodiment of the present invention, and FIG. It is a block diagram showing the configuration of an AR-based viewing server of an AR-based performance video viewing system according to an embodiment of the present invention, and FIG. 4 is a front view showing an LED protocol board of the AR-based performance video viewing system according to an embodiment of the present invention. to be.

Referring to FIG. 1 , the AR-based performance video viewing system according to an embodiment of the present invention implements an augmented reality (AR) performance when a performance held in a performance hall T is photographed by a visitor terminal 10, so that the viewer It is a system that allows viewing through the terminal 10 as a performance image including an actual performance image and an augmented reality (AR) performance.

Referring to FIG. 2 , the AR-based performance video viewing system of the present invention may include a visitor terminal 10 , an AR-based viewing server 20 , an LED protocol board 30 , and a manager terminal 40 .

The visitor terminal 10 refers to a mobile terminal possessed by an unspecified number of people, including spectators who visit a science hall, an experience hall, a museum, a performance hall, and the like to watch a performance. In this case, the visitor terminal 10 may include a tablet, a personal digital assistant (PDA), etc. in addition to the mobile terminal.

In addition, the visitor terminal 10 may include an application (or mobile app) provided from the AR-based viewing server 20 . That is, since the application (or mobile app) provided from the AR-based viewing server 20 is installed, the AR-based performance video viewing system can be used.

In order to receive the performance image, the audience terminal 10 can obtain a performance image by shooting a performance through an application provided from the AR-based viewing server 20, and the obtained performance image is transferred to the AR-based viewing server 20 can be sent to

In addition, the visitor terminal 10 may receive and output a performance image from the AR-based viewing server 20 and provide it to the visitor.

In addition, the visitor terminal 10 may record a performance image when a recording request is input from a visitor, and end recording of the performance image when an end of recording is input. In this case, the manager terminal 40 may transmit the recorded performance image to the AR-based viewing server 20 when the recording is finished.

The AR-based viewing server 20 may generate a performance image and transmit it to the audience terminal 10 when an actual performance image for a performance is received from the audience terminal 10 .

To this end, the AR-based viewing server 20 includes a database 21 , a performance recognition unit 22 , an object recognition unit 23 , a setting unit 24 , an image output unit 25 and a control unit 26 as shown in FIG. 3 . ) may be included.

The database 21 may store a plurality of AR performances received from the manager terminal 40 .

Here, the AR performance may be designed according to the story performed offline and may be woven into a number of effects.

In addition, the database 21 may receive and store the recorded performance image from the visitor terminal 10 .

Also, the database 21 may store information about a specific signal. That is, information on the meaning of the signal may be stored according to a recognized specific signal.

Accordingly, the setting unit 24 may recognize a specific signal and generate effect setting information accordingly.

The performance recognition unit 22 may receive a performance image from the audience terminal 10 , and recognize a signal from the actual performance image at all times to recognize the performance progress and the performance venue.

That is, the regular signal is output from the LED protocol board 30 at the same time as the performance starts and can be photographed on the visitor terminal 10. In this case, when the regular signal appears in the real image of the performance, the performance recognition unit 22 recognizes it and You can recognize that the show is going on.

As a result, it is possible to identify the implementation time of the AR performance, so that the synchronization (synchronization) between the offline performance and AR performance can be effectively matched.

In addition, the performance recognition unit 22 may recognize the performance hall based on the constant signal output from the LED protocol board 30 installed in the performance hall, thereby aligning the overall position between the offline performance and the AR performance. In other words, it is possible to figure out where the AR performance should be implemented as a whole in the actual performance image.

In addition, the LED object recognition unit 23 can effectively recognize the LED object by first understanding the range where the LED object is located, that is, the location where the AR performance should be implemented.

The object recognition unit 23 may recognize one or more LED objects (R) in the real image of the performance hall recognized.

Here, the LED object (R) is preferably an LED robot that performs various performances such as dancing in the performance hall (T), but is not limited thereto, and may be variously applied using LEDs in addition to the LED robot.

Specifically, the object recognition unit 23 converts an actual performance image into a gray image, performs black-and-white processing on the actual performance image converted into a gray image using a threshold, and canny edge from the black-and-white processed performance image The LED object (R) can be recognized by detecting the outline using a Canny Edge Detector.

More specifically, the object recognition unit 23 may convert the actual performance image into a gray image.

Next, the object recognition unit 23 may perform black-and-white processing on the actual performance image converted into a gray image using a threshold. This is to separate the background other than the LED object (R) such as the LED object (R) and the performance hall (T) from the actual performance image by processing in black and white.

At this time, the object recognition unit 23 changes the pixel value to 255 if the pixel value is equal to or greater than the threshold value in the actual performance image converted to the gray image, and to 0 if it is less than the threshold value, so that the actual performance image is processed in black and white can

Next, the object recognition unit 23 may detect an outline from the black-and-white-processed performance image using a Canny Edge Detector. Accordingly, the LED object R can be finally recognized.

In this case, the object recognition unit 23 may detect an outline through the process of removing noise, detecting an edge, calculating a gradient size and direction, sharpening an edge, and selecting and connecting an important edge.

In more detail, the object recognizer 23 may remove noise through blurring using a Gaussian mask in the black-and-white-processed performance real image. This is to increase the accuracy of the recognition of the LED object (R).

The blurring using a Gaussian mask removes noise by filtering using a Gaussian normal distribution with a mean of 0.

Here, the Gaussian normal distribution has a higher value toward the center and a lower value toward the outer part.

Through this process, the noise of the actual performance image can be removed and the details can be simplified.

After the noise is removed, the object recognizer 23 may detect an edge from the noise-removed performance image using a Sobel mask.

At this time, the object recognition unit 23 detects an edge in the vertical direction to obtain a vertical Sobel mask, and a horizontal Sobel mask by detecting an edge in the horizontal direction, and convolutions with the actual performance image, respectively. Edges can be detected by calculating the grades of two images (values).

After detecting the edge, the object recognition unit 23 may calculate the gradient size and direction of the detected edge.

After calculating the gradient size and direction, the object recognition unit 23 may sharpen the edge by removing low pixel values through non-maximum suppression (NMS). This is a necessary process to find a sharper edge because the detected edge is a blurred edge.

Non-maximum suppression (NMS) is a method of comparing pixel values in 8 neighboring directions based on the central pixel, leaving the central pixel as it is if it is the largest, and removing it if it is not.

The object recognizer 23 may make an edge thin by removing a low pixel value through non-maximum suppression (NMS) to sharpen the edge.

After edge sharpening, the object recognition unit 23 may detect an outline by selecting and connecting important edges among the edges.

At this time, since the object recognition unit 23 may detect a part other than an edge as an edge due to noise or color variation that has not been removed in the real image of the performance, a real edge (hereinafter, “important edge”) among the detected edges ”) is selected only.

To do this, the object recognition unit 23 may describe an important edge with two threshold values.

Here, the two thresholds may include a high threshold and a low threshold.

Accordingly, the object recognition unit 23 removes a strong edge if the edge has a value higher than the high threshold, a weak edge if the edge has a value lower than the high threshold and higher than the low threshold, and a weak edge if the edge is lower than the low threshold, A strong edge and a weak edge can be selected.

Next, the object recognition unit 23 may select important edges by keeping the selected strong edges as they are, and by discriminating and selecting only weak edges related to the strong edges.

An outline can be detected by connecting the selected important edges in this way.

Since the detected outline indicates the LED object R, the LED object R can be detected.

When a specific signal is recognized in the actual performance image, the setting unit 24 may generate effect setting information by setting which LED object to display the effect of the AR performance among one or more LED objects recognized according to the specific signal.

The specific signal recognized here may include, but is not limited to, effect information of AR performance and effect implementation location information.

For example, if the effect implementation location information of a specific signal is information implementing an effect on the 3rd LED object (R) among the recognized 1st, 2nd, and 3rd LED objects (R), the 3rd LED object (R) It is possible to create effect setting information that allows the effect to be implemented in the .

The image output unit 25 may provide the performance image to the audience by transmitting the AR performance to the audience terminal 10 according to the effect setting information and implementing the AR performance in the performance real image.

That is, the image output unit 25 may output the AR performance on the screen of the visitor terminal 10 according to the effect setting information, so that the AR performance can be implemented in a location and sync with the offline performance.

Accordingly, it is possible to provide a performance image with improved immersion.

When AR performance setting information is received according to a performance from the manager terminal 40 , the control unit 26 may transmit AR performance setting information to the LED protocol board 30 .

Accordingly, the LED protocol board 30 may output a specific signal according to the AR performance set by the performance.

In addition, the control unit 26 may receive the recorded performance image from the visitor terminal 10 and store it in the database 21 .

In addition, the control unit 26 receives the performance start information from the manager terminal 40, transmits it to the LED protocol board 30, so that the LED protocol board 30 operates to output a signal at all times, and sets the AR performance It is possible to output a specific signal according to the information.

In addition, the AR-based viewing server 20 may further include a mission unit (not shown).

The mission unit (not shown) may provide mission information to the visitor terminal 10 , and when a function corresponding to the mission information is performed from the visitor terminal 10 , the mission achievement information may be transmitted to the visitor terminal 10 .

That is, the mission unit (not shown) may provide the mission information to the visitor terminal 10 by generating the corresponding performance viewing as mission information according to the performance scheduled to be performed offline.

In addition, the mission unit (not shown) may detect that the mission has been performed when the actual performance image of the performance is received from the audience terminal 10 , generate mission achievement information, and transmit it to the audience terminal 10 .

In this case, the level of interest may be increased by providing mission information and mission achievement information in the form of bingo, but is not limited thereto.

As such, it is possible to improve the interest and viewing rate of visitors through the mission unit (not shown).

The LED protocol board 30 may be installed in the performance hall T to output a regular signal and a specific signal during a performance.

This LED protocol board 30 is the location and sync between the offline performance and the AR performance when the performance is filmed from the audience terminal 10 and the AR-based viewing server 20 wants to provide a performance image to the audience terminal 10 . can be effectively matched.

To this end, the LED protocol board 30 is preferably installed on both sides of the upper end of the performance hall T, as shown in FIG. 1 , but is not limited thereto.

In addition, as shown in FIG. 4 , the LED protocol board 30 may include a constant signal unit 31 and a specific signal unit 32 to output a constant signal and a specific signal, but the shape is not limited thereto.

The regular signal unit 31 is composed of an LED, and when the performance starts and the performance start information is received from the AR-based viewing server 20, it can output a constant signal.

The regular signal unit 31 may continue to output a regular signal until the performance ends.

Accordingly, the AR-based viewing server 20 can recognize the start of the performance, and even if the audience shoots the performance with the audience terminal 10 from the middle of the performance, the AR-based viewing server 20 can recognize that the performance is in progress. , even from the middle of the performance, the audience can receive the performance image using the AR-based performance image viewing system.

Such a constant signal unit 31 may be composed of an orange LED, and four may be arranged in a grid, but is not limited thereto.

The specific signal unit 32 is composed of an LED, and may output a specific signal according to the AR performance set according to the performance.

That is, the specific signal unit 32 receives the AR performance setting information received from the AR-based viewing server 20, determines the AR performance set through the AR performance setting information, and outputs a specific signal according to the set AR performance. have.

This specific signal unit 32 may be composed of a green LED, and 4 may be arranged in a grid, but is not limited thereto.

For example, the specific signal unit 32 may output a specific signal by outputting one or more of the four LEDs arranged in a grid.

Accordingly, the AR-based viewing server 20 can recognize a specific signal and implement the AR performance in the actual video according to the specific signal, and if the viewer uses the AR-based performance video viewing system even from the middle of the performance, it will fit the performance in progress. A synchronized performance image may be provided to the visitor terminal 10 .

The manager terminal 40 means a mobile terminal in which a manager, a person in charge, a director, etc. possesses performances such as a science hall, an experience hall, a museum, and a performance hall. In this case, the manager terminal 40 may include a PC, a tablet, a personal digital assistant (PDA), etc. in addition to the mobile terminal.

In addition, the manager terminal 40 may include an application (or mobile app) provided from the AR-based viewing server 20 . That is, since the application (or mobile app) provided from the AR-based viewing server 20 is installed, the AR-based performance video viewing system can be managed and used.

The manager terminal 40 may set the AR performance and transmit AR performance setting information to the controller 26 of the AR-based viewing server 20 .

In addition, the manager terminal 40 may receive the performance start information and transmit it to the controller 26 of the AR-based viewing server 20 .

In addition, the manager terminal 40 may receive the AR performance and transmit it to the AR-based viewing server 20 to be stored in the database 21 .

Above, a method for providing a performance video using the AR-based performance video viewing system according to an embodiment of the present invention will be described based on the configuration described above. At this time, since it has been described in detail in the description of the system, it will be briefly described.

5 is a flowchart sequentially illustrating a method for providing a performance image using an AR-based performance image viewing system according to an embodiment of the present invention, FIG. 6 is a flowchart sequentially illustrating steps S300 of FIG. 5, and FIG. It is a flowchart sequentially showing steps S330.

Referring to FIG. 5 , the method for providing a performance video using the AR-based performance video viewing system in an embodiment of the present invention includes the step of receiving an actual performance image photographed from the audience terminal (S100), and always in the received performance image Recognizing a signal to recognize a performance progress and a venue (S200), recognizing one or more LED objects in a performance image (S300), recognizing a specific signal in a performance image, and effect of AR performance according to a specific signal Step of generating effect setting information by setting which LED object to display (S400) and transmitting the AR performance to the audience terminal according to the effect setting information and providing a performance image by implementing the AR performance on the actual image of the performance (S500) ) may be included.

First, in step S100 , the AR-based viewing server 20 may receive an actual performance image obtained by photographing a performance from the audience terminal 10 .

In step S200, the AR-based viewing server 20 receives the performance image, and recognizes a regular signal from the received performance image to recognize the performance progress and the performance venue.

In step S300, the AR-based viewing server 20 may recognize one or more LED objects in the real image of the performance hall recognized.

Specifically, referring to FIG. 6 , step S300 is a step of converting an actual performance image into a gray image (S310), a step of black-and-white processing of the actual performance image converted into a gray image using a threshold (S320), and It may include a step (S330) of detecting an outline using a Canny Edge Detector in the black-and-white-processed actual image of the performance.

In step S310, the actual performance image may be converted into a gray image.

In step S320, the actual performance image converted to the gray image in step S310 may be processed in black and white using a threshold.

That is, in step S320, the AR-based viewing server 20 changes the pixel value to 255 if the pixel value is equal to or greater than the threshold value in the actual performance image converted to the gray image and to 0 if it is smaller than the threshold value to convert the actual performance image into black and white. can be processed

In step S330, an outline can be detected using a Canny Edge Detector in the black-and-white-processed performance image in step S320.

More specifically, step S330 is a step of removing noise through blurring using a Gaussian mask in the black-and-white-processed real image as shown in FIG. 7 ( S331 ), an edge using a Sobel mask Detecting (S332), calculating the gradient size and direction of the detected edge (S333), sharpening the edge by removing low pixel values (S334), and selecting and connecting important edges to detect the outline It may include a step (S335).

First, in step S331, noise may be removed from the black-and-white-processed performance image through blurring using a Gaussian mask.

In step S332, an edge may be detected by using a Sobel mask in the real image of the performance from which the noise has been removed.

That is, in step S332, a vertical Sobel mask is detected by detecting an edge in the vertical direction from the black-and-white processed performance image, and a horizontal Sobel mask is obtained by detecting an edge in the horizontal direction, and convolution with the actual performance image, respectively. ), the edge can be detected by obtaining the grade of the two images (values).

In step S333, the gradient magnitude and direction of the edge detected in step S332 may be calculated.

In step S334, the edge may be sharpened by removing a low pixel value through non-maximum suppression (NMS) after step S333.

That is, step S334 compares pixel values in 8 neighboring directions with respect to the center pixel in the performance image in which the edge is detected, and if the center pixel is the largest, it is left as it is, otherwise it is removed by repeating the process to sharpen the edge. can

In step S335, an outline may be detected by selecting and connecting important edges among the edges sharpened in step S334.

That is, step S335 removes a strong edge if the edge has a value higher than the high threshold among the multiple edges, a weak edge if the edge has a value lower than the high threshold and higher than the low threshold, and removes the edge if the edge is lower than the low threshold, A strong edge and a weak edge can be selected.

Then, it is possible to select important edges by keeping the selected strong edges as they are, and by distinguishing only the weak edges related to the strong edges.

Then, the LED object R can be recognized by connecting the selected important edges to detect the exterior line.

Step S400 may generate effect setting information by recognizing a specific signal in the actual performance image, and setting which LED object (R) to display the effect of the AR performance according to the specific signal.

In step S500, the AR performance may be transmitted to the audience terminal 10 according to the effect setting information, and the performance image may be provided by implementing the AR performance in the actual performance image. In this way, by implementing the AR performance in accordance with the actual performance image in the manager terminal 40 , it is possible to provide a performance image that is precisely in sync between the offline performance and the AR performance.

As described above, the AR-based performance video viewing system and the performance video providing method using the same according to an embodiment of the present invention can increase immersion and dynamics by fusing augmented reality (AR) performance with an offline LED robot performance provided by a science museum. You can create a ham.

In addition, by using the LED protocol board, the synchronization between the offline LED robot performance and the AR performance can be effectively achieved to provide a high-quality performance image.

This can increase the satisfaction and demand of visitors.

It is also expected to paint a new blueprint for the future for customers, whose main target is students.

In addition, it is expected that it can be applied to various exhibitions and performances.

The embodiment of the present invention described above is not implemented only through an apparatus and/or method, but may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention, a recording medium in which the program is recorded, etc. Also, such an implementation can be easily implemented by those skilled in the art to which the present invention pertains from the description of the above-described embodiments.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improved forms of the present invention are also provided by those skilled in the art using the basic concept of the present invention as defined in the following claims. is within the scope of the right.

10: visitor terminal
20: AR-based viewing server
21: Database
22: Performance Recognition Department
23: object recognition unit
24: setting unit
25: video output unit
26: control unit
30: LED protocol board
40: Manager
T: concert hall
R: LED object

Claims (5)

In a system for viewing an LED robot performance as a performance image including an actual performance image and an augmented reality (AR) performance,
a visitor terminal capable of filming a performance;
When an actual performance image for a performance is received from the audience terminal, an AR-based viewing server that transmits the performance image to the audience terminal; and
It includes an LED protocol board that is installed on both sides of the upper part of the concert hall and outputs regular signals and specific signals during performances.
The LED protocol board,
When the performance starts and the performance start information is received from the AR-based viewing server, a regular signal unit that outputs a signal at all times, and
AR-based performance video viewing system including a specific signal unit that outputs a specific signal according to the AR performance set according to the performance.
According to claim 1,
The AR-based viewing server,
a database storing a plurality of AR performances;
a performance recognition unit receiving the actual performance image from the audience terminal, and recognizing a signal from the actual performance image at all times to recognize the performance progress and the performance venue;
an object recognition unit for recognizing one or more LED objects in the real image of the performance hall recognized;
When a specific signal is recognized in the actual performance image, a setting unit for generating effect setting information by setting which LED object to display the effect of the AR performance among one or more LED objects recognized according to the specific signal; and
AR-based performance video viewing system including an image output unit for transmitting an AR performance to the audience terminal according to the effect setting information and providing the performance image by implementing the AR performance on the actual performance image.
3. The method of claim 2,
The object recognition unit,
Converting the actual image of the performance into a gray image,
In the actual performance image converted to gray image, if the pixel value is greater than or equal to the threshold, the pixel value is changed to 255, and if it is smaller than the pixel value, the pixel value is changed to 0,
AR-based performance video viewing system, characterized in that the LED object is recognized by detecting the outline using Canny Edge Detector in the black-and-white processed performance image.


delete In the method of providing a performance image including an actual performance image and an augmented reality (AR) performance to the audience using an AR-based performance image viewing system,
(a) receiving a real image of a performance photographed by an LED robot performance from a visitor terminal;
(b) recognizing a regular signal from the received performance image to recognize the performance progress and the performance venue;
(c) recognizing one or more LED objects in the performance image;
(d) generating effect setting information by recognizing a specific signal in the performance image, and setting which LED object to display the effect of the AR performance according to the specific signal; and
(e) transmitting the AR performance to the visitor terminal according to the effect setting information and providing the performance image by implementing the AR performance in the performance real image,
The step (c) is,
converting an actual performance image into a gray image;
Black-and-white processing of the actual performance image converted into a gray image using a threshold value
A method for providing performance video using an AR-based performance video viewing system, characterized in that it includes the step of detecting an exterior line using a Kenny edge detector in the black-and-white-processed performance image

Images