CN103051934A - Intelligent television human-machine interaction method, device and system - Google Patents

Abstract

The present invention provides a smart TV human-computer interaction device, comprising: a remote controller module, adapted to receive user operations, and generate corresponding control signals; a server module, adapted to receive and analyze the control signals to obtain instructions; an area of interest analysis module, adapted to intercept and analyze the current TV picture according to the instruction of analyzing the current picture, so as to obtain the potential area and feed it back to the server module; and an output module, adapted to output the image of the potential area and the selected sensory area Region of interest image. And provide a smart TV human-computer interaction system based on the device, which also includes a TV module, which is suitable for receiving instructions from the output module, operating according to the instructions, and receiving and displaying data from the output module. Correspondingly, a smart TV human-computer interaction method based on the above-mentioned smart TV human-computer interaction device is also provided. The user can conveniently click on the target of interest in the TV screen by using the remote controller.

Description

Smart TV human-computer interaction method, device and system

technical field

The invention relates to the field of smart TVs, in particular to a smart TV human-computer interaction method, device and system.

Background technique

With the development of society, TV will gradually develop into an open service carrying platform and become a home intelligent entertainment terminal. Smart TV will have application platform advantages that traditional TV manufacturers do not have. Smart TV will realize various application services such as network search, IP TV, video on demand (VOD), digital music, network news, and network video calling. TV is becoming the third information access terminal after computers and mobile phones, and users can access the information they need at any time; TV will also become a smart device, enabling cross-platform search between TV, the Internet and programs; Smart TV It will also be an "entertainment center" where users can search for TV channels, record TV shows, and be able to play satellite and cable TV programming, as well as Internet video. The arrival of smart TVs conforms to the trend of high-definition, networked and intelligent TV sets.

In addition to these "networking" and "intelligence", as the basic functions of TV, viewers are more interested in TV programs, such as TV dramas, character interviews, sports events, tourism and other TV programs. Sometimes, viewers are often interested in a certain target in the TV picture, such as a certain actor, athlete, a certain landmark building, etc., and want to know more about the details of the target, they need to carry out an investigation on a certain target in the picture. Easy to click. Different from the computer, to realize these functions on the TV, you can only rely on a remote control. How to use the remote control to conveniently click on the target of interest in the TV screen for subsequent information processing or network search, it is necessary to design a convenient and fast human-computer interaction method and system.

Contents of the invention

The technical problem to be solved by the present invention is to provide a smart TV human-computer interaction method, device and system, so that the user can conveniently click on the target of interest in the TV screen by using a remote control.

According to one aspect of the present invention, a smart TV human-computer interaction device is provided, including: a remote controller module, adapted to receive user operations, and generate corresponding control signals; the control signals include: analyzing the control signal of the current picture and selecting the control signal; the server module is adapted to receive and analyze the control signal to obtain instructions; the instructions include: an instruction for analyzing the current picture corresponding to the control signal for analyzing the current picture, and a selected interest corresponding to the selection control signal Region instructions; the region of interest analysis module is adapted to intercept the current TV image and analyze it according to the instruction of the analysis of the current image, so as to obtain the potential region and feed it back to the server module; the server module is also adapted to receive the Draw the feedback data on the original image to become a potential region image; the server module is also adapted to receive an instruction for selecting a region of interest, and draw the selected feedback data on the original image to become a selected region of interest image; and an output module adapted to output the latent region image and the selected ROI image.

Optionally, the smart TV human-computer interaction device further includes: an information retrieval module, adapted to further perform target analysis, identification and information retrieval on the selected ROI according to the instruction of the selected ROI.

According to another aspect of the present invention, a smart TV human-computer interaction system is provided, including: the above-mentioned smart TV human-computer interaction device; and a TV module adapted to receive instructions from the output module and operate according to the instructions.

Optionally, the TV module receives and displays data from the output module.

According to another aspect of the present invention, there is provided a smart TV human-computer interaction method based on the smart TV human-computer interaction device, including: step 1, receiving user operations, and generating a control signal; the control signal includes: analyzing the current picture Control signal; Step 2, analyze this signal to obtain instruction; Described instruction comprises: the instruction of analyzing current picture corresponding to the control signal of analyzing current picture; Step 3, according to the instruction of described analyzing current picture, to current TV picture Perform analysis to obtain the potential area, draw the potential area on the original image to become a potential area image, and output the potential area image; step 4, receive user operation and generate a control signal; the control signal includes: select the control signal; step 5 , analyzing the signal to obtain instructions; the instructions include: a selected region of interest instruction corresponding to the selection control signal; step 6, drawing the selected feedback data on the original image based on the selected region of interest instruction, becomes the selected ROI image, and outputs the selected ROI image.

Optionally, the smart TV human-computer interaction method further includes: Step 7: Use the image information covered by the selected ROI to further perform target analysis, identification and information retrieval on the selected ROI.

Optionally, in step 1, when the user presses a certain function key of the remote control module, a control signal for analyzing the current picture can be generated; in step 4, based on the potential area, when the user presses a certain function key of the remote control module Number keys or letter keys can generate selection control signals.

Optionally, Step 3 further includes: intercepting the current TV picture, analyzing the potential area of the current picture to obtain the potential area; labeling the potential area according to the position; feeding back the position and label information to the server module.

Optionally, the latent region analysis includes face detection.

Optionally, the potential area is represented by a rectangle; the location information includes the coordinates of the upper left corner of the rectangle and the coordinates of the lower right corner of the rectangle; and the label information includes a digital label.

Compared with the prior art, the present invention has the advantages of:

(1) By automatically analyzing the current screen, some potential areas are provided to the audience, and the audience can easily select the target area by using a common remote control. This solution is simple to use and conforms to the current usage habits of the audience. Convenient and effective human-computer interaction;

(2) After the area of interest is determined, follow-up processing can be performed according to actual needs, such as identifying the target, and performing operations such as searching for relevant information on the Internet according to the identification result.

Description of drawings

Fig. 1 is a structural block diagram of an intelligent TV human-computer interaction device provided in an embodiment of the present invention;

Fig. 2 is a structural block diagram of the intelligent TV human-computer interaction system provided in one embodiment of the present invention;

Fig. 3 is the working process flowchart of the example of the smart TV human-computer interaction system in Fig. 2;

Fig. 4 is a schematic diagram of the analysis results of the region of interest analysis module in one embodiment of the present invention;

Fig. 5 is a flowchart of a human-computer interaction method for a smart TV provided in an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

According to an embodiment of the present invention, a smart TV human-computer interaction device is provided. As shown in Figure 1, the device includes:

The remote control module 101 is adapted to receive user operations and generate corresponding control signals. For example, the user presses a certain function key of the remote control module to generate a control signal for analyzing the current picture; A number key or letter key can generate a number or letter selection control signal. In this embodiment, the remote control module 101 may be a Changhong TV remote control or a Skyworth TV remote control. In other embodiments of the present invention, the user can also use the remote control module 101 with the mouse positioning function to directly move the mouse to the inside of the region of interest, and press the OK key to send a digital selection signal.

The server module 103 is adapted to receive and analyze the signal sent by the remote control module 101 to obtain instructions (the instruction format may be a binary code format, and different codes represent different instructions). If the instruction contained in the signal is an instruction to analyze the current screen or an instruction to select a certain area (such as a number key), proceed as follows:

(1) If it is an instruction to analyze the current picture, the server module 103 will invoke the ROI analysis module 104 to receive the analysis result (i.e. feedback data) of the ROI analysis module 104, and the server module 103 will draw the received feedback data on the become a latent area image on the original image, and send the latent area image to the output module 105; and

(2) If it is an instruction to select an area in the current screen, that is, an instruction to select an area of interest (for example, a number key), the server module 103 will compare the received instruction with the feedback data, and then compare The data of the selected region is drawn on the original image to become a selected ROI image, and the ROI image is sent to the output module 105 .

The region of interest analysis module 104 is suitable for analyzing the current TV picture, specifically including:

(1) Intercept the current TV picture, and analyze the region of interest on the current picture. The image area of interest can be completed by automatically analyzing image features (including but not limited to edge features, texture features, etc.), or through target detection and recognition algorithms in pattern recognition for specific types of target detection (such as face detection, etc.), Other technologies can also be used to automatically detect potential regions, and the specific method used is not limited here.

Take face detection as an example below to illustrate the detection process of the ROI analysis module 104 to the face target (the training and detection module of the face detection disclosed in opencv can be utilized): first the faces in the TV program are marked, As the positive samples of the face classifier, some pictures without faces are collected as negative samples, and the training module of face detection is called to generate the face detection classifier. On the picture to be detected, call the face detection module, scan the entire image, use the trained face classifier to detect the face target in the image, and give the finally detected face area. The specific technical details and operation process are currently well-known technologies in the academic and industrial circles, and will not be described in detail here. Other object training and detection methods are similar to this. Potential targets can also be detected using other techniques, and the specific methods used are not limited here.

The regions of interest obtained by these analyzes may be areas such as faces, which can be represented by rectangle or circle parameters, and their position information can be represented by rectangle, circle or ellipse information, such as the coordinates of the upper left corner of the rectangle, the length of the rectangle wide information. In other embodiments of the present invention, the aforementioned regions may be represented by other available shape parameters.

(2) Label the analyzed potential areas in the current frame by position; and feed back the position and label information to the server module 103 . For example, the label can start from 1, and can be number keys such as 1, 2, 3, etc. In other embodiments of the present invention, the above-mentioned numbering methods can be implemented in other available ways, such as letter numbers, Chinese character numbers, and the like.

The analysis results of the region of interest analysis module 104 can be expressed in the following data formats:

{m, (1, (x11, y11), (x12, y12)), (2, (x21, y21), (x22, y22)), ...}

Among them, m represents the m potential regions obtained by the analysis, and (1, (x11, y11), (x12, y12)) represents the rectangular representation of the first region of interest, expressed by the coordinates of the upper left corner of the rectangle and the coordinates of the lower right corner of the rectangle.

For example, as shown in Figure 4, the analysis results can be expressed as:

{2, (1, (100, 50), (200, 200)), (2, (250, 70), (320, 180))}.

The server module 103 and the region of interest analysis module 104 mentioned above can be respectively implemented by a microprocessor chip, or the functions of the server module 103 and the region of interest analysis module 104 can be completed on one microprocessor chip.

The output module 105 receives and outputs the data sent by the server module 103 .

For example, as shown in Figure 4, the instruction received by the server module 103 is that the remote controller presses the number key 1, that is, the first area of interest is selected, then the server module deletes other interest 2, and only keeps interest 1, Draw the graph represented by the data {1, (1, (100, 50), (200, 200))} onto the original image, and then send it to the output module 105, and the output module 105 sends the data to the TV for display .

Further, the smart TV human-computer interaction device may also include: an information retrieval module 106 .

When the instruction received by the server module 103 is to select a certain area (such as a number key) in the current screen, the server module 103 also calls the information retrieval module 106 .

The information retrieval module 106 is adapted to receive the viewer's region of interest data information forwarded by the server module, such as {1, (1, (100, 50), (200, 200))}, and intercept the region of interest in the current TV screen The covered image information uses the data stored in the server module to further analyze and identify the target of the regional content, including but not limited to, actors, athletes, interview guests, landmark buildings, etc. Taking human faces as an example, the server module can store the facial images of characters appearing in the TV series in advance, and tag these facial images with names to build a human face library. Compare the image information of the sensing area with the image information in the face database to determine the identity of the character in the area of interest. The target is searched on the server side or on the network according to the recognition result, and the final search result is obtained and fed back to the server module 103 . Information retrieval methods include but are not limited to image matching, target matching, keyword retrieval, etc., and can utilize actor information, guest information, etc. of a specific TV program. The results of information retrieval can be information such as images, videos, or texts.

The server module 103 receives the analysis results and retrieval results from the ROI analysis module 104 and the information retrieval module 106 and sends them to the output module 105 . The output module 105 can send the data to the TV for display. In other embodiments of the present invention, the expressions may also be sent to other devices to provide instructions (including but not limited to graphics, colors, and sounds) to users.

In combination with the above devices, an embodiment of the present invention provides a smart TV human-computer interaction system. As shown in FIG. 2 , the system includes the above-mentioned smart TV human-computer interaction device and a TV module 107 . The TV module 107 is suitable for receiving specific instructions sent by the output module 105 (including commands related to traditional remote controllers, such as changing channels, adjusting volume, etc.) and displaying images, and operating according to the instructions. The operation and processing process All are known technologies in the current academic and industrial circles, and will not be repeated here.

As shown in Figure 3, the working process of the system is as follows:

S101. The button of the remote controller sends a signal for analyzing the current picture; that is, when the viewer is interested in a certain area in the current picture and wants to know more about the target, he needs to press a certain function button on the remote controller;

S102. The server module analyzes the signal, and calls the ROI analysis module;

S103, the region of interest analysis module feeds back the analysis results to the server module; the region of interest analysis module first intercepts the current picture, automatically analyzes the current picture, uses an algorithm to obtain the potential regions of the user, and labels them in order, and finally lists the potential regions of interest The position information and label information are fed back to the server module;

S104. The server module draws the numbered analysis result data to the original image, and sends it to the output module;

S105. The remote controller presses the number keys to select a certain area of interest; the user uses the number keys to click on the potential area obtained from the automatic analysis displayed on the screen;

S106. The server module parses the signal, and invokes the information retrieval module;

S107. The information retrieval module retrieves the target according to the region of interest selected by the user and feeds the retrieval result back to the server module; the retrieval result can be image, video or text information, webpage information, etc.;

S108. The server module sends the retrieval result to the output module.

Corresponding to the above-mentioned device, in one embodiment of the present invention, a smart TV human-computer interaction method is provided, which mainly includes two steps of automatically generating a potential area according to screen content and determining an area of interest. As shown in Figure 5, the method specifically includes:

S201. Receive a user operation and generate a control signal; the control signal includes: analyzing the control signal of the current screen; the process is the same as that of the above-mentioned remote control module 101, and will not be described again;

S202. Analyze the signal to obtain an instruction; the instruction includes: an instruction for analyzing the current image corresponding to the control signal for analyzing the current image;

S203. According to the instruction for analyzing the current picture, analyze the current TV picture to obtain the potential area, draw the potential area on the original image to become a potential area image, and output the potential area image; the process is the same as that of the server module 103, The working process of the region of interest analysis module 104 is the same and will not be repeated;

S204. Receive a user operation and generate a control signal; the control signal includes: a selection control signal;

S205. Analyze the signal to obtain an instruction; the instruction includes: an instruction for selecting a region of interest corresponding to the selection control signal;

S206. Draw the selected feedback data on the original image based on the instruction of selecting the region of interest to become the image of the selected region of interest, and output the image of the selected region of interest. This process is the same as the description of the server module 103 above, and will not be repeated here.

Further, the above method also includes after step S206:

S207. Using the image information covered by the selected ROI, further perform target analysis and recognition on the selected ROI. This process is the same as the description of the server module 103 and the information retrieval module 106 above, and will not be repeated here.

Using the above method and system, if the viewer is interested in some parts of the current TV screen and wants to know more about relevant information, he needs to press a certain button on the remote control, and the remote control sends a wireless signal to the TV, and the TV resolves This command intercepts the current image, uses the content analysis module to analyze the current image, and obtains some areas that the user may be interested in through a certain algorithm, and labels them, and displays them in the current image at the same time, and the user uses the remote control Number keys to select the marked target frame. The result is fed back to the TV, and the information retrieval module performs background search or network search according to the selected target, and presents the result for the viewer to learn more information.

It should be noted and understood that various modifications and improvements can be made to the invention described in detail above without departing from the spirit and scope of the invention as claimed in the appended claims. Accordingly, the scope of the claimed technical solution is not limited by any particular exemplary teaching given.

Claims (10)

1. A smart TV human-computer interaction device, comprising: The remote control module is adapted to receive user operations and generate corresponding control signals; the control signals include: analyzing the control signals of the current screen and selecting the control signals; The server module is adapted to receive and analyze the control signal to obtain instructions; the instructions include: an instruction for analyzing the current picture corresponding to the control signal for analyzing the current picture, and an instruction for selecting the region of interest corresponding to the selection control signal ; The region of interest analysis module is adapted to intercept and analyze the current TV picture according to the instruction of analyzing the current picture, so as to obtain the potential area and feed it back to the server module; the server module is also suitable for receiving the feedback data drawing on the original image to become a potential region image; the server module is further adapted to receive an instruction for selecting a region of interest, and draw the selected feedback data on the original image to become a selected region of interest image; and An output module adapted to output the latent region image and the selected ROI image. 2. The smart TV human-computer interaction device according to claim 1, further comprising: The information retrieval module is adapted to further perform target analysis, identification and information retrieval on the selected ROI according to the instruction of selecting ROI. 3. A smart TV human-computer interaction system, comprising: The smart TV human-computer interaction device according to any one of claims 1-2; and The television module is suitable for receiving instructions from the output module and operating according to the instructions. 4. The smart TV human-computer interaction system according to claim 3, wherein the TV module receives and displays data from the output module. 5. A smart TV human-computer interaction method, comprising: Step 1, receiving user operation and generating a control signal; the control signal includes: analyzing the control signal of the current screen; Step 2, analyzing the signal to obtain an instruction; the instruction includes: an instruction for analyzing the current image corresponding to the control signal for analyzing the current image; Step 3. According to the instruction of analyzing the current picture, analyze the current TV picture to obtain the potential area, draw the potential area on the original image to become a potential area image, and output the potential area image; Step 4, receiving user operation and generating a control signal; the control signal includes: selection control signal; Step 5, analyzing the signal to obtain an instruction; the instruction includes: an instruction for selecting a region of interest corresponding to the selection control signal; Step 6: Drawing the selected feedback data on the original image based on the instruction of selecting the region of interest to become the image of the selected region of interest, and outputting the image of the selected region of interest. 6. The intelligent TV human-computer interaction method according to claim 5, further comprising: Step 7: Use the image information covered by the selected ROI to further perform target analysis, recognition and information retrieval on the selected ROI. 7. The intelligent TV human-computer interaction method according to claim 5, wherein, In step 1, when the user presses a certain function key of the remote controller module, a control signal for analyzing the current screen can be generated; In step four, based on the potential area, when the user presses a number key or letter key of the remote controller module, a selection control signal may be generated. 8. The intelligent TV human-computer interaction method according to claim 5, wherein, step 3 also includes: Capture the current TV picture, and analyze the potential area of the current picture to obtain the potential area; Label potential regions by location; Feed back location and label information to the server module. 9. The smart TV human-computer interaction method according to claim 8, wherein the potential region analysis includes face detection. 10. The intelligent TV human-computer interaction method according to claim 8, wherein the potential area is represented by a rectangle; the location information includes coordinates of the upper left corner of the rectangle and the coordinates of the lower right corner of the rectangle; and the label information includes a digital label.

Images