KR102744896B1 - Display apparatus and controlling method thereof - Google Patents

Abstract

A display device is disclosed. The display device includes a processor that controls the display to display a UI screen including a display and a plurality of text objects, controls the display to display a preset number together with a text object different from a predetermined language among the plurality of text objects, and performs an operation related to a text object corresponding to the displayed number when the recognition result of a voice spoken by a user includes the displayed number.

Description

{DISPLAY APPARATUS AND CONTROLLING METHOD THEREOF}

The present disclosure relates to a display device and a control method thereof, and more particularly, to a display device providing voice recognition control for content composed of various languages and a control method thereof.

Thanks to the development of electronic technology, various types of display devices have been developed and distributed. In particular, various electronic devices such as TVs, mobile phones, PCs, laptop PCs, PDAs, etc. have been widely used in most households.

Meanwhile, technology using voice recognition has been developed recently to control display devices more conveniently and intuitively.

Conventionally, display devices controlled by the user's voice perform voice recognition using a voice recognition engine. However, since different voice recognition engines exist for each language, it was necessary to determine in advance which voice recognition engine to use for voice recognition. Therefore, the system language of the display device was usually determined as the language to be used for voice recognition.

However, for example, if the language used in the hyperlink text displayed on the display device is English and the system language of the display device is Korean, there is a problem in that even if the user speaks the voice corresponding to the hyperlink text, the voice is converted into Korean text through the Korean speech recognition engine, so the hyperlink text cannot be selected.

In this way, in the past, there were limitations in controlling the display device by voice when the system language and the language actually displayed on the display device were different.

The present disclosure is made in accordance with the aforementioned necessity, and an object of the present disclosure is to provide a display device that provides voice recognition control for contents composed of various languages, and a control method thereof.

In order to achieve the above-described purpose, according to one embodiment of the present disclosure, a display device includes a processor configured to control the display to display a UI screen including a display and a plurality of text objects, to control the display to display a preset number together with a text object having a different language from a predetermined language among the plurality of text objects, and to perform an operation related to a text object corresponding to the displayed number when a recognition result of a voice spoken by a user includes the displayed number.

In this case, the processor may set the language set in the setting menu of the display device to the predetermined language. Or, the most frequently used language in the plurality of text objects may be set to the predetermined language.

Meanwhile, the UI screen is a web page, and the processor can set a language corresponding to language information of the web page to the predetermined language.

Meanwhile, the processor may determine that a text object composed of two or more languages among the plurality of text objects is a text object different from the predetermined language if the inclusion ratio of the predetermined language is less than a preset ratio.

Meanwhile, the processor can control the display to display the preset number adjacent to a text object corresponding to the preset number.

Meanwhile, the display device according to the present disclosure further includes a communication unit that performs communication with an external device, and the processor can control the display to display the preset number while a signal corresponding to selection of a specific button of the external device is received.

In this case, the external device includes a microphone, the communication unit receives a voice signal corresponding to a voice input through the microphone of the external device, and the processor can perform an operation related to a text object corresponding to the displayed number if the recognition result for the received voice signal includes the displayed number.

In this case, if the recognition result for the received voice signal includes text corresponding to any one of the plurality of text objects, the processor can perform an operation related to the corresponding text object.

Meanwhile, the action related to the text object may be an action of displaying a web page of a URL address corresponding to the text object or an action of executing an application program corresponding to the text object.

Meanwhile, the plurality of text objects are included in the execution screen of the first application, and if the processor determines that there is no object corresponding to the recognition result of a voice spoken by the user while the execution screen of the first application is displayed on the execution screen of the first application, the processor can execute a second application different from the first application and perform an operation corresponding to the recognition result of the voice.

In this case, the second application is an application that provides search results for a search word, and if the processor determines that an object corresponding to a recognition result of a voice spoken by a user while the execution screen of the first application is displayed is not present on the execution screen of the first application, the processor can execute the second application to provide a search result using a text corresponding to the recognition result of the voice as a search word.

Meanwhile, the display device according to the present disclosure further includes a communication unit that communicates with a server that performs voice recognition for a plurality of different languages, and the processor controls the communication unit to provide a voice signal corresponding to a voice spoken by the user and information on the predetermined language to the server, and if the voice recognition result received from the server includes the displayed number, an operation related to a text object corresponding to the displayed number can be performed.

In this case, if the voice recognition result received from the server includes text corresponding to any one of the plurality of text objects, the processor can perform an operation related to the corresponding text object.

Meanwhile, a control method of a display device according to an embodiment of the present disclosure includes a step of displaying a plurality of text objects, a step of displaying a preset number together with a text object that is different from a preset language among the plurality of text objects, and a step of performing an operation related to a text object corresponding to the displayed number if the recognition result of a voice spoken by a user includes the displayed number.

In this case, the control method of the display device according to the present disclosure may further include a step of setting the most frequently used language in the plurality of text objects to the predetermined language.

Meanwhile, the plurality of text objects are included in a web page, and the control method of the display device according to the present disclosure may further include a step of setting a language corresponding to language information of the web page to the predetermined language.

Meanwhile, the control method of the display device according to the present disclosure may further include a step of determining that a text object composed of two or more languages among the plurality of text objects is a text object different from the predetermined language if the inclusion ratio of the predetermined language is less than a preset ratio.

Meanwhile, the step of displaying the preset number together may display the preset number adjacent to a text object corresponding to the preset number.

Meanwhile, the step of displaying the preset number together can display the preset number while a signal corresponding to selection of a specific button of the external device is received from the external device.

Meanwhile, the step of performing an action related to the text object may display a web page of a URL address corresponding to the text object or execute an application program corresponding to the text object.

Meanwhile, the plurality of text objects are included in the execution screen of the first application, and the control method of the display device according to the present disclosure may further include a step of executing a second application different from the first application and performing an operation corresponding to the recognition result of the voice when it is determined that an object corresponding to the recognition result of a voice spoken by the user while the execution screen of the first application is displayed is not present in the execution screen of the first application.

Meanwhile, the control method of the display device according to the present disclosure further includes a step of providing a voice signal corresponding to the voice spoken by the user and information on the predetermined language to a server that performs voice recognition for a plurality of different languages, and the step of performing an operation related to the text object may perform an operation related to a text object corresponding to the displayed number if the voice recognition result received from the server includes the displayed number.

Meanwhile, in a computer-readable recording medium storing a program for executing a method for controlling a display device according to an embodiment of the present disclosure, the method for controlling the display device includes the steps of controlling the display device to display a plurality of text objects, the step of controlling the display device to display a preset number together with respect to a text object that is different from a predetermined language among the plurality of text objects, and the step of performing an operation related to a text object corresponding to the displayed number when the recognition result of a voice spoken by a user includes the displayed number.

Figures 1 and 2 are drawings for explaining a voice command input method in a display device according to various embodiments of the present disclosure.
FIG. 3 is a drawing for explaining a voice recognition system according to one embodiment of the present disclosure;
FIG. 4 is a block diagram for explaining the configuration of a display device according to one embodiment of the present disclosure.
FIGS. 5 to 7 are drawings for explaining a numerical display method for object selection according to various embodiments of the present disclosure.
FIGS. 8 and 9 are drawings for explaining a voice search method according to various embodiments of the present disclosure.
FIG. 10 is a block diagram for explaining the configuration of a display device according to another embodiment of the present disclosure, and
FIG. 11 is a flowchart for explaining a method for controlling a display device according to an embodiment of the present disclosure.

Before describing the present disclosure in detail, the description method of the specification and drawings will be described.

First, the terms used in this specification and claims are general terms selected in consideration of functions in various embodiments of the present disclosure. However, these terms may vary depending on the intention of a person skilled in the art, legal or technical interpretation, emergence of new technologies, etc. In addition, some terms are terms arbitrarily selected by the applicant. These terms may be interpreted as defined in this specification, and if there is no specific definition of a term, it may be interpreted based on the overall content of this specification and common technical knowledge in the art.

In addition, the same reference numbers or symbols described in each drawing attached to this specification represent parts or components that perform substantially the same function. For the convenience of explanation and understanding, the same reference numbers or symbols are used in different embodiments to describe them. That is, even if components having the same reference numbers are all depicted in multiple drawings, the multiple drawings do not mean one embodiment.

In addition, terms including ordinal numbers such as "first", "second", etc. may be used in this specification and claims to distinguish between components. These ordinals are used to distinguish between identical or similar components, and the meaning of the terms should not be interpreted in a limited manner due to the use of these ordinals. For example, components associated with these ordinals should not be restricted in terms of the order of use or arrangement, etc., by their numbers. If necessary, each ordinal number may be used interchangeably.

In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, it should be understood that terms such as "comprise" or "consist of" are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

In the embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are terms used to refer to components that perform at least one function or operation, and these components may be implemented as hardware or software, or as a combination of hardware and software. In addition, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except in cases where each of them needs to be implemented as individual specific hardware.

In addition, in the embodiments of the present disclosure, when it is said that a part is connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, when it is said that a part includes a certain component, it means that it can further include other components, rather than excluding other components, unless there is a specific description to the contrary.

Hereinafter, the present disclosure will be described in detail using the attached drawings.

FIG. 1 is a drawing for explaining a display device according to one embodiment of the present disclosure controlled by voice recognition.

Referring to FIG. 1, the display device (100) may be a TV as shown in FIG. 1, but this is only an example, and may be implemented as any device having a display function, such as a smartphone, desktop PC, laptop, smart watch, navigation, or refrigerator.

The display device (100) can perform an operation based on the recognition result of the voice spoken by the user. For example, if the user says "Change to channel 7," the program of channel 7 can be displayed, and if the user says "Turn off the power," the power can be turned off. In addition, the display device (100) can also operate as if conversing with the user. For example, if the user says "What is the name of the currently broadcasting program?", the display device can output the message "The title of the program you inquired about is ○○○" as a voice or text, and if the user says "How is the weather today?", the display device can output the message "Please tell me the region you want" as a voice or text, and if the user says "Seoul," the display device can output the message "The temperature in Seoul is ○○" as a voice or text.

As illustrated in FIG. 1, the display device (100) can receive a user's voice through a microphone connected to the display device (100) or included in the display device (100). Alternatively, the display device (100) can also receive a voice signal corresponding to a voice input through a microphone of the external device from the external device. This will be described with reference to FIG. 2.

FIG. 2 is a drawing for explaining a display system according to one embodiment of the present disclosure.

Referring to FIG. 2, the display system includes a display device (100) and an external device (200).

The display device (100) is a device that operates according to the voice recognition results as described in Fig. 1.

In Fig. 2, an example is shown in which an external device (200) is implemented as a remote control, but it may also be implemented as an electronic device such as a smartphone, tablet PC, or smart watch.

The external device (200) is a device including a microphone, and can transmit a voice signal corresponding to a voice input through the microphone to the display device (100). For example, the external device (200) can transmit a voice signal to the display device (100) using a wireless communication method such as infrared (IR), RF, Bluetooth, or Wi-Fi.

The microphone of the external device (200) may be activated only when there is a preset event to save power. For example, the microphone is activated while the microphone button (210) of the external device (200) is pressed, and the microphone is deactivated when the microphone button (210) is released. In other words, voice input can be received only while the microphone button (210) is pressed.

Voice recognition for voice input through the microphone of the display device (100) or the microphone of the external device (200) can be performed through an external server. Fig. 3 is a drawing for explaining an embodiment related to this.

Referring to FIG. 3, the voice recognition system (2000) includes a display device (100) and a server (300).

The display device (100) is a device that operates according to the voice recognition results as described in Fig. 1. As described above, the display device (100) can transmit a voice signal corresponding to a voice input through the microphone of the display device (100) or the microphone of the external device (200) to the server (300).

The display device (100) can transmit information (hereinafter, “language information”) indicating the language in which the voice signal should be recognized, along with the voice signal, to the server (300). Even for the same voice signal, different voice recognition results may be obtained depending on which language voice recognition engine is used.

The server (300) can perform speech recognition for a plurality of different languages. The server (300) can include a plurality of speech recognition engines corresponding to each of the multiple languages. For example, the server (300) can include a Korean speech recognition engine, an English speech recognition engine, a Japanese speech recognition engine, etc. When the server (300) receives a speech signal and language information from the display device (100), the server (300) can perform speech recognition using a speech recognition engine corresponding to the language information for the speech signal.

And the server (300) transmits the result of voice recognition to the display device (100), and the display device (100) can perform an operation corresponding to the result of voice recognition received from the server (300).

For example, if the text included in the voice recognition result received from the server (300) matches a text object displayed on the display device (100), the display device (100) can perform an operation related to the corresponding text object. For example, if there is a text object matching the text included in the voice recognition result on a web page, the display device (100) can display a web page of a URL address corresponding to the corresponding text object. However, this is only an example, and UI objects provided by various applications of the display device (100) can be selected by voice recognition and the corresponding operation can be performed.

Meanwhile, in Fig. 3, it is illustrated that there is only one server (300), but there may be multiple servers corresponding to each of multiple languages. For example, there may be separate servers responsible for Korean speech recognition and English speech recognition.

Meanwhile, in the above-described example, voice recognition is described as being performed in a server (300) that is a separate device from the display device (100), but according to another example, the display device (100) can also perform the function of the server (300). In other words, the above-described display device (100) and server (300) can also be implemented as a single product.

FIG. 4 is a block diagram for explaining the configuration of a display device (100) according to one embodiment of the present disclosure.

The display device (100) includes a display (110) and a processor (120).

The display (110) may be implemented as, for example, an LCD (Liquid Crystal Display), and in some cases, may be implemented as a CRT (cathode-ray tube), a PDP (plasma display panel), an OLED (organic light emitting diodes), a TOLED (transparent OLED), etc. In addition, the display (110) may be implemented as a touch screen capable of detecting a user's touch operation.

The processor (120) is configured to control the overall operation of the display device (100).

For example, the processor (120) may include a CPU, a RAM, a ROM, and a system bus. Here, the ROM is a configuration in which a set of commands for system booting is stored, and the CPU copies the operating system stored in the storage of the display device (100) to the RAM according to the commands stored in the ROM, and executes the O/S to boot the system. When booting is complete, the CPU may copy various applications stored in the storage to the RAM and execute them to perform various operations. In the above description, the processor (120) includes only one CPU, but it may be implemented with multiple CPUs (or DSPs, SoCs, etc.) when implemented.

When a user command for selecting an object displayed on the display (110) is input, the processor (120) can perform an operation associated with the object selected by the user command. Here, the object can be any object that can be selected, for example, a hyperlink or an icon. The operation associated with the selected object can be, for example, an operation for displaying a page, document, image, etc. connected to the hyperlink, an operation for executing a program corresponding to an icon, etc.

A user command for selecting an object may be, for example, a command input through various input devices (e.g., mouse, keyboard, touchpad, etc.) connected to the display device (100), or a voice command corresponding to a voice spoken by the user.

Although not shown in FIG. 4, the display device (100) may further include a voice receiving unit for receiving voice input. The voice receiving unit may include a microphone to directly receive a voice spoken by a user and generate a voice signal, or may receive an electric voice signal from an external device (200). In the latter case, the voice receiving unit may be implemented as a communication unit for performing wired or wireless communication with the external device (200). Such a voice receiving unit may not be included in the display device (100) depending on the case. For example, a voice signal corresponding to a voice input through a microphone of the external device (200) may be transmitted to the server (300) via a device other than the display device (100) or may be directly transmitted from the external device (200) to the server (300), and the display device (100) may be implemented in a form that receives only a voice recognition result from the server (300).

The processor (120) can control the display (110) to display numbers together with text objects that are different from a predetermined language among the text objects displayed on the display (110).

Here, the predetermined language means the language that is the basis of voice recognition (the language of the voice recognition engine to be used for voice recognition), and can be set manually by the user, or can be set automatically. In the case of manually setting the language, for example, the language set as the language used (or system language) in the setting menu provided by the display device (100) can be set as the language that is the basis of voice recognition.

In one embodiment of automatically setting a language as the basis for speech recognition, the processor (120) can identify the language most frequently used in text objects currently displayed on the display (110) and automatically set that language as the language as the basis for speech recognition.

Specifically, the processor (120) can analyze the type of characters (e.g., Korean or alphabet) included in each of the plurality of text objects currently displayed on the display (110) and set the language corresponding to the type of characters most commonly used in the plurality of text objects as the language that serves as the basis for speech recognition.

According to another embodiment, if the objects currently displayed on the display (110) are objects of a web page, the processor (120) may set a language corresponding to the language information of the web page as the language that serves as the basis for speech recognition. The language information of the web page can be checked, for example, in the lang attribute of HTML (e.g., <html lang="en">).

Once the language that is the basis of voice recognition is set, the processor (120) can control the display (110) to display an arbitrary number together with a text object that is different from the language that is the basis of voice recognition. The user can select a text object by saying an arbitrary number displayed on the display (110). In addition, since an image object cannot be selected by voice, the processor (120) can control the display (110) to display an arbitrary number together with an image object as well.

The processor (120) can determine that a text object composed of only a language other than the language that is the basis of speech recognition is a text object different from the language to be used for speech recognition. In addition, the processor (120) can determine that a text object composed of two or more languages is a text object different from the language to be used for speech recognition if the inclusion ratio of the language that is the basis of speech recognition is less than a preset ratio. This will be described in more detail with reference to FIG. 5.

Figure 5 illustrates a specific screen displayed on the display (110).

Referring to FIG. 5, a UI screen including a plurality of text objects (51 to 59) is displayed on a display (110). It is assumed that a language that is the basis of voice recognition is set to English. The processor (120) can control the display (110) so that arbitrary numbers (① to ⑥) are displayed together with text objects (51 to 56) composed of a language other than English. The numbers (① to ⑥) can be displayed at positions adjacent to the corresponding text objects (51 to 56). In addition, for text objects (51, 58) composed of English, specific icons (57a, 58a) are displayed together around them, thereby informing the user that the text objects (51, 58) can be selected by speaking the text included in the text objects (51, 58). The icon (57a, 58a) may be expressed as “T” as shown in Fig. 5, but is not limited thereto and may be expressed in various forms, such as “Text”.

For a text object (59) composed of two or more languages, the processor (120) can check whether the inclusion ratio of English is less than a preset ratio (e.g., 50%) and, if so, control the display (110) to display numbers together. The text object (59) illustrated in FIG. 5 is composed of Korean and English, and since the inclusion ratio of English exceeds the preset ratio (e.g., 50%), numbers are not displayed together. Instead, an icon (59a) that notifies that the text object can be selected by speaking the text included in the text object can be displayed adjacent to the text object (59).

Meanwhile, in FIG. 5, the number is illustrated as having a shape like "①", but there is no limitation on the shape of the number. For example, it may be in a shape including "1" in a square instead of a circle, or it may simply be displayed as "1". According to another embodiment of the present disclosure, it may be displayed as a word of a language that is the basis of speech recognition. If the language that is the basis of speech recognition is English, it may be displayed as "one", and if the language that is the basis of speech recognition is Spanish, it may be displayed as "uno".

Meanwhile, although not shown in FIG. 5, a phrase that encourages the user to say a number, such as “You can select an object corresponding to the number you said,” may additionally be displayed on the display (110) along with the number.

According to another embodiment of the present disclosure, the processor (120) may determine that a text object composed of two or more languages is a text object whose language is different from the language that is the basis of speech recognition if the language of the first word is different from the language to be used for speech recognition. The present embodiment will be described with reference to FIG. 6.

Figure 6 illustrates a specific screen displayed on the display (110).

Referring to FIG. 6, a UI screen including a plurality of text objects (61 to 63) is displayed on the display (110). Let us assume that the language to be used for voice recognition is set to Korean. The processor (120) can determine that the text object (61) composed of two or more languages is a text object different from the language that is the basis of voice recognition because the language of the first word "AAA" is English, not Korean, which is the language that is the basis of voice recognition. Accordingly, the processor (120) can control the display (110) so that a number (①) is displayed together with the text object (61).

According to an embodiment described with reference to FIG. 6, even if a text object composed of two or more languages includes a language that is the basis of speech recognition in a preset ratio or more, if the first word is different from the language that is the basis of speech recognition, a number is displayed. Conversely, even if a text object composed of two or more languages includes a language that is the basis of speech recognition in a ratio less than the preset ratio, if the first word is the same as the language that is the basis of speech recognition, a number is not displayed. This is because a user is likely to say the word that exists at the very front of a text object in order to select the text object.

Meanwhile, according to another embodiment of the present disclosure, since image objects cannot be selected by voice, numbers can also be displayed on image objects. This embodiment will be described below with reference to FIG. 7.

Figure 7 illustrates a specific screen displayed on the display (110).

Referring to FIG. 7, a first image object (71), a second image object (72), a third image object (74), a first text object (73), and a second text object (75) are displayed on a display (110). The processor (120) can control the display (110) to display a number (①) together with the first image object (71).

Meanwhile, according to another embodiment of the present disclosure, if a plurality of objects displayed on the display (110) are objects having URL links, and if, as a result of comparing the URL links of the plurality of objects, there are objects having the same URL link, if none of the objects can be selected by voice recognition, the processor (120) can control the display (110) to display a number only for one of the objects, and if any one of the objects can be selected by voice recognition, the processor can control the display (110) to not display a number.

To explain more specifically, if a plurality of objects (i.e., text objects or image objects that are different from the language that is the basis of voice recognition) that cannot be selected by voice recognition are displayed on the display (110), and if they have links with the same URL address, a number may be displayed on only one of the objects. Referring to FIG. 7, since the second image object (72) is an object that cannot be selected by voice, and the text object (73) is composed of English, a language different from Korean, the language that is the basis of voice recognition, neither the second image object (72) nor the first text object (73) can be selected by voice, but since the second image object (72) and the first text object (73) are linked to the same URL address when selected, a number (②) may be displayed only on the second image object (72), which is one of the second image object (72) and the first text object (73). Alternatively, a number may be displayed on the text object (73) instead of the second image object (72). This is to minimize the number of numbers displayed on the display (110).

In order to minimize the number of numbers displayed on the display (110), according to another embodiment of the present disclosure, if a plurality of objects having the same URL address are displayed on the display (110), and any one of them is a text object identical to the language that is the basis of voice recognition, no numbers are displayed for any of them. Referring to FIG. 7, the processor (120) compares the URL address of the third image object (74) with the ULR address of the second text object (75) and determines that they are the same, and if the second text object (75) is determined to be a text object identical to Korean, which is the language that is the basis of voice recognition, the processor (120) controls the display (110) not to display numbers on the third image object (74).

If the recognition result of the voice spoken by the user includes a specific text displayed on the display (110), the processor (120) can perform an operation related to a text object corresponding to the text. Referring to FIG. 5, if the user says “Voice recognition,” the processor (120) can control the display (110) to display a page of a URL address corresponding to the text object (59).

Meanwhile, according to one embodiment of the present disclosure, if the recognition result of the voice spoken by the user includes a text that is commonly included in two or more text objects among the text objects displayed on the display (110), the processor (120) may display a number in each of the corresponding text objects, and when the user speaks the displayed number, an operation related to the text object corresponding to the number may be performed. Referring to FIG. 5, if the recognition result of the voice spoken by the user includes “Speech recognition,” the processor (120) searches for a text object that includes “Speech recognition” among the text objects displayed on the screen. If a plurality of text objects (57, 58) are searched, the processor (120) may control the display (110) to display an arbitrary number next to the text objects (57, 58). For example, the number ⑦ may be displayed next to the text object (57), the number ⑧ may be displayed next to the text object (58), and the user may select the text object (57) by speaking the number “7.” If the recognition result of the voice spoken by the user includes a number displayed on the display (110), the processor (120) can perform an operation related to a text object or image object corresponding to the included number. Referring to FIG. 6, if the user says “one,” the processor (120) can control the display (110) to display a page of a URL address corresponding to the text object (61).

The voice spoken by the user may be input through the microphone of the display device (100) or may be input through the microphone of the external device (200). In the latter case, the display device (100) may include a communication unit for communicating with the external device (200) including the microphone, and the communication unit may receive a voice signal corresponding to the voice input through the microphone of the external device (200). If the recognition result for the voice signal received from the external device (200) through the communication unit includes a number displayed on the display (110), the processor (120) may perform an operation related to a text object corresponding to the number. Referring to FIG. 6, when the user says “one” into the microphone of the external device (200), the external device (200) transmits a voice signal to the display device (200), and the processor (120) may control the display (110) to display a page of a URL address corresponding to the text object (61) based on the voice recognition result for the received voice signal.

Meanwhile, the numbers displayed in response to the text or image object may be displayed only for a certain period of time. According to one embodiment, the processor (120) may control the display (110) to display the numbers while a signal corresponding to the selection of a specific button is received from the external device (200). That is, the numbers may be displayed only while the user presses a specific button of the external device (200). Here, the specific button may be, for example, the microphone button (210) of the external device (200) described in FIG. 2.

According to another embodiment, the processor (120) may display numbers when a voice input through the microphone of the display device (100) includes a preset keyword (e.g., “Hi TV”), and remove the displayed numbers when a preset period of time has elapsed without a voice being input through the microphone of the display device (100).

Meanwhile, although the above-described embodiments have described that numbers are displayed, numbers do not necessarily have to be displayed, and any words (meaningful or meaningless words) that a user can see and read may be used. For example, instead of 1, 2, 3, a, b, c, etc. may be displayed. According to another embodiment of the present disclosure, if there is a search window on a web page displayed on the display (110), the user can easily perform a search by uttering a word he or she wants to search for and a specific keyword that executes the search function. For example, if there is a search window on a web page displayed on the display (110), the search result for "○○○" can be displayed on the display (110) by simply saying "Search ○○○" or "Search ○○○."

To this end, the processor (120) can detect a search word input window on a web page displayed on the display (110). Specifically, the processor (120) can search for an object that can be input among the constituent objects of the web page displayed on the display (110). An input tag on HTML is an object that can be input. An input tag has various attributes, and among them, type attributes clearly define the input characteristics. If the type is "search", the object clearly corresponds to a search word input window.

However, in the case of objects whose type is "text", it is not possible to immediately determine whether it is a search input window. This is because general input objects also have a text type, so it is not possible to distinguish whether the object is a search input window or a general input window. Therefore, in this case, a separate process is required to determine whether it is a search input window.

For objects whose type is "text", information about additional attributes of the object is referenced to determine whether it is a search input box. If the title or aria-label contains the keyword "search", the object can be determined to be a search input box.

And the processor (120) determines whether a specific keyword is included in the recognition result of the voice spoken by the user. Here, the specific keyword may be "search", "find", etc. If it is determined that the specific keyword is included, the processor (120) checks the location of the specific keyword in order to more accurately determine the user's intention. If at least one word exists before or after the specific keyword, it is highly likely that the user's intention is to search for the at least one word. If the voice recognition result only includes a specific keyword such as "search" or "find", it is highly likely that the user's intention is not to search.

This process of judging the user's intention can be performed on the display device (100), and can also be performed on the server (300) and the result provided to the display device (100).

When the user's search intent is determined, the processor (120) selects the remaining words excluding the specific keyword as search words, and performs a search by entering the selected search words into the search word input window detected according to the above-described method. For example, as shown in FIG. 8, when a web page including a search word input window (810) is displayed on the display (110), the processor (120) detects the search word input window (810), and when the user speaks "puppy search," the processor (120) selects "puppy" as a search word from the voice recognition results for the spoken voice, enters the search word into the detected search word input window (810), and performs a search.

Meanwhile, the operation of detecting a search word input window in a web page displayed on the display (110) may be performed after it is determined that a specific keyword is included in the voice recognition result, or may be performed in advance.

Fig. 9 is a drawing for explaining another example of a search word input method. Fig. 9 is for explaining a search execution method when there are multiple search word input windows in one web page.

Referring to FIG. 9, a case where there are two search windows in a web page is illustrated. The first search word input window (910) is for news search, and the second search word input window (920) is for stock search. The processor (120) performs a search using the search word input window displayed at the time when the user utters a voice including a search word, based on information about the arrangement position of objects and information about the layout of the current screen. For example, when the first search word input window (910) is displayed on the display (110), if the user utters a voice including a search word and a specific keyword, the processor (120) inputs the search word into the first search word input window (910), and when the second search word input window (920) is displayed on the display (110), if the user utters a voice including the search word and a specific keyword, the processor (120) can input the search word into the second search word input window (920). That is, if there are multiple search input boxes in one web page, a search can be performed using the search input box currently displayed on the screen.

Voice control is performed based on the screen shown on the display (110). That is, basically, a function according to a voice command is performed using an application corresponding to the screen being displayed on the display (110). However, if the input voice command does not match an object included in the currently displayed screen or is different from the function of the application currently displaying the screen, another application may be executed to perform the function according to the voice command.

For example, if the currently running application is a web browsing application and the voice spoken by the user does not match an object within the web page displayed by the web browsing application, the processor (120) may execute another preset application to perform a search function corresponding to the voice spoken by the user. Here, the other preset application may be an application that provides a search function, for example, an application that provides search results for text corresponding to the voice using the search engine of Google™, an application that provides search results for VOD content corresponding to text corresponding to the voice, etc. Meanwhile, before such another application is executed, the processor (120) may display a UI for obtaining the user's consent, such as "There are no results matching ○○○ on the current screen. Would you like to search for ○○○ on the Internet?", and after the user's consent is entered in the UI, an Internet search application, etc. may be executed to provide the search results.

The display device (100) may include a voice processing unit that processes the voice recognition result received from the server (300) and an application unit that executes an application installed on the display device (100). The voice processing unit provides the voice recognition result received from the server (300) to the application unit. If the recognition result is provided while the first application of the application unit is executed and the screen of the first application is displayed on the display (110), the first application may perform the operation described above based on the voice recognition result provided from the voice processing unit. For example, operations such as searching for a text or image object corresponding to a number included in the voice recognition result, searching for a text object corresponding to a word included in the voice recognition result, and executing a search after entering a keyword in a search window when “search” is included in the voice recognition result may be performed. If the first application does not have an operation to perform using the voice recognition result provided from the voice processing unit, that is, for example, if there is no text or image object corresponding to the voice recognition result or there is no search window, the first application may notify the voice processing unit of this, and the voice processing unit may control the application unit to execute a second application that can perform an operation related to the voice recognition result. For example, the second application is an application that provides search results for a specific search term. The application unit can execute the second application to provide search results using text included in the voice recognition results as a search term.

Fig. 10 is a block diagram illustrating the configuration when the display device (100) is implemented as a TV. In explaining Fig. 10, description of the configuration that overlaps with the configuration described in Fig. 4 is omitted.

Referring to FIG. 10, the display device (100) may be implemented as, for example, an analog TV, a digital TV, a 3D-TV, a smart TV, an LED TV, an OLED TV, a plasma TV, a monitor, a curved TV having a screen with a fixed curvature, a flexible TV having a screen with a fixed curvature, a bent TV having a screen with a fixed curvature, and/or a curvature-variable TV capable of changing the curvature of the current screen by a received user input, but is not limited thereto.

The display device (100) includes a display (110), a processor (120), a tuner (130), a communication unit (140), a microphone (150), an input/output unit (160), an audio output unit (170), and a storage unit (180).

The tuner (130) can select only the frequency of the channel to be received by the display device (100) from among many radio wave components by amplifying, mixing, resonating, etc. the broadcast signal received wired or wirelessly. The broadcast signal can include video, audio, and additional data (e.g., EPG (Electronic Program Guide).

The tuner (130) can receive video, audio and data in a frequency band corresponding to a channel number corresponding to a user input.

The tuner (130) can receive broadcast signals from various sources, such as terrestrial broadcasting, cable broadcasting, or satellite broadcasting. The tuner (130) can also receive broadcast signals from various sources, such as analog broadcasting or digital broadcasting.

The tuner (130) may be implemented as an all-in-one with the display device (100) or as a separate device (e.g., a set-top box, a tuner connected to an input/output unit (160)) having a tuner unit electrically connected to the display device (100).

The communication unit (140) is a component that performs communication with various types of external devices according to various types of communication methods. The communication unit (140) can be connected to an external device through a local area network (LAN) or the Internet, and can be connected to an external device by a wireless communication method (for example, wireless communication such as Z-wave, 4LoWPAN, RFID, LTE D2D, BLE, GPRS, Weightless, Edge Zigbee, ANT+, NFC, IrDA, DECT, WLAN, Bluetooth, WiFi, Wi-Fi Direct, GSM, UMTS, LTE, WiBRO, etc.). The communication unit (140) includes various communication chips such as a WiFi chip (141), a Bluetooth chip (142), an NFC chip (143), and a wireless communication chip (144). The WiFi chip (141), the Bluetooth chip (142), and the NFC chip (143) perform communication in a WiFi method, a Bluetooth method, and an NFC method, respectively. The wireless communication chip (174) refers to a chip that performs communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evoloution), etc. In addition, the communication unit (140) includes an optical receiving unit (145) that can receive a control signal (e.g., IR pulse) from an external device (200).

The processor (120) can transmit a voice signal and language information (information on a language that is the basis of voice recognition) to the server (300) through the communication unit (140), and when the server (300) transmits the result of voice recognition performed on the voice signal using a voice recognition engine of a language corresponding to the language information, the result of the voice recognition can be received through the communication unit (140).

The microphone (150) can receive a voice spoken by a user and generate a voice signal corresponding to the received voice. The microphone (150) can be implemented as an integral part of the display device (100) or can be separated. The separated microphone (150) can be electrically connected to the display device (100).

If the display device (100) does not have a microphone, the display device (100) can receive a voice signal corresponding to a voice input through the microphone of the external device (200) from the external device (200) through the communication unit (140). The communication unit (140) can receive a voice signal from the external device (200) through a communication method such as Wi-Fi or Bluetooth.

The input/output unit (160) is configured to be connected to an external device. The input/output unit (160) may include at least one of an HDMI input port (High-Definition Multimedia Interface port, 161), a component input jack (162), and a USB port (163). In addition to those illustrated, the input/output unit (180) may include at least one of ports such as RGB, DVI, HDMI, DP, and Thunderbolt.

The audio output unit (170) is a component for outputting audio, and may output audio included in a broadcast signal received through a tuner (130), audio input through a communication unit (140), an input/output unit (160), or audio included in an audio file stored in a storage unit (180). The audio output unit (170) may include a speaker (171) and a headphone output terminal (172).

The storage unit (180) may include various application programs, data, and software modules for driving and controlling the display device (100) under the control of the processor (120). For example, the storage unit (180) may include a web parsing module for parsing web content data received through the Internet, a JavaScript module, a graphic processing module, a voice recognition result processing module, and an input processing module.

When voice recognition is performed by the display device (100) itself rather than by an external server (300), a voice recognition module including various voice recognition engines suitable for various languages may be stored in the storage unit (180).

The storage unit (180) can store data for configuring various UI screens provided on the display (110). In addition, the storage unit (180) can store data for generating control signals corresponding to various user interactions.

The storage unit (180) may be implemented as a nonvolatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), etc. Meanwhile, the storage unit (180) may be implemented as not only a storage medium within the display device (100), but also an external storage medium, for example, a micro SD card, a USB memory, or a web server via a network.

The processor (120) controls the overall operation of the display device (100) and the signal flow between the internal components of the display device (100), and performs the function of processing data.

The processor (120) includes a RAM (121), a ROM (122), a CPU (123), and a bus (124). The RAM (121), the ROM (122), the CPU (123), etc. may be connected to each other through the bus (124). The processor (120) may be implemented as a SoC (System On Chip).

The CPU (123) accesses the storage (180) and performs booting using the O/S stored in the storage (180). Then, it performs various operations using various programs, contents, data, etc. stored in the storage (180).

A set of commands for system booting, etc. are stored in ROM (122). When a turn-on command is input and power is supplied, CPU (123) copies the O/S stored in storage (180) to RAM (121) according to the command stored in ROM (122) and executes the O/S to boot the system. When booting is complete, CPU (123) copies various application programs stored in storage (180) to RAM (121) and executes the application programs copied to RAM (121) to perform various operations.

The processor (120) can perform various operations using modules stored in the storage unit (180). For example, the processor (120) can parse and process web content data received through the Internet and display the overall layout of the content and each object on the display (110).

When the voice recognition function is activated, the processor (120) analyzes objects of web content to find objects that can be controlled by voice, performs preprocessing on information such as the location of the object, actions related to the object, and whether text is included in the object, and stores the preprocessing results in the storage unit (180).

And the processor (120) can control the display (110) so that objects controllable (selectable) by voice are identifiable and displayed based on the preprocessed object information. For example, the processor (120) can control the display (110) so that the colors of objects controllable by voice are displayed differently from other objects.

And the processor (120) can recognize the voice input by the microphone (150) as text using a voice recognition engine. In this case, the processor (120) uses a voice recognition engine of a predetermined language (a language set as the basis of voice recognition). Alternatively, the processor (120) can send a voice signal and information about the language that is the basis of voice recognition to the server (300) and receive text as a voice recognition result from the server (300).

And the processor (120) can search for an object corresponding to the voice recognition result among the preprocessed objects and display that the object has been selected at the location of the searched object. For example, the processor (120) can control the display (110) to highlight the object selected by voice. And the processor (120) can perform an operation related to the object corresponding to the voice recognition result based on the preprocessed object information and output the result through the display (110) or the audio output unit (170).

FIG. 11 is a flowchart for explaining a control method of a display device (100) according to an embodiment of the present disclosure. The flowchart illustrated in FIG. 11 may be composed of operations processed in the display device (100) described in this specification. Therefore, even if the content is omitted below, the content described regarding the display device (100) may also be applied to the flowchart illustrated in FIG. 11.

Referring to Fig. 11, first, a UI screen including multiple text objects is displayed on a display device (100) (S1110).

And for text objects that are different from a predetermined language among multiple text objects displayed on the display device, a preset number is displayed together (S1120). Here, the preset language means a language that is determined in advance as a basis for voice recognition. The language that will be the basis for voice recognition may be a language set as a default language, set by a user's manual setting, or automatically set based on a language that constitutes objects displayed on the display device (100). In the case of automatic setting, for example, the language that constitutes the object may be confirmed by applying OCR (Optical character recognition) to objects displayed on the display device (100).

And if the recognition result of the voice spoken by the user includes a displayed number, an action related to a text object corresponding to the displayed number is performed (S1130).

The recognition result of the voice spoken by the user can be obtained by the display device's own voice recognition, or can be received by requesting voice recognition from an external server that performs voice recognition for multiple different languages. In the latter case, the display device (100) provides the external server with a voice signal corresponding to the voice spoken by the user and information about the language set as the basis of the voice recognition, and if the voice recognition result received from the external server includes a displayed number, an operation related to a text object corresponding to the displayed number can be performed.

For example, if the text object is a hyperlink text within a web page, the display action of the web page corresponding to the URL address of the text object can be performed, and if the text object is an icon for executing an application, the application can be executed.

Meanwhile, the UI screen including the plurality of text objects may be an execution screen of the first application. The execution screen of the first application may be any screen provided by the first application. If it is determined that an object corresponding to the recognition result of a voice spoken by the user while the execution screen of the first application is displayed is not present on the execution screen of the first application, the display device may execute a second application different from the first application to perform an operation corresponding to the recognition result of the voice. Here, the first application may be a web browsing application, and the second application may be an application that performs a search from various sources, such as the Internet, data stored in the display device, VOD content, channel information (e.g., EPG), etc. For example, if there is no object corresponding to voice recognition on the currently displayed web page, the display device may execute another application to provide a search result corresponding to voice recognition (e.g., Google search result, VOD search result, channel search result, etc.).

According to the various embodiments described above, voice control for objects composed of various languages is possible, and voice search can also be made easier.

Meanwhile, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In terms of hardware implementation, the embodiments described in the present disclosure may be implemented using at least one of ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions. In some cases, the embodiments described in the present specification may be implemented in the processor (120) itself. In terms of software implementation, embodiments such as the procedures and functions described in the present specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described in the present specification.

Meanwhile, computer instructions for performing processing operations in the display device (100) according to the various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in the non-transitory computer-readable medium are executed by a processor of a specific device, they cause the specific device to perform processing operations in the display device (100) according to the various embodiments described above.

A non-transitory computer-readable medium is not a medium that stores data for a short period of time, such as a register, cache, or memory, but a medium that permanently stores data and can be read by a device. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray disks, USBs, memory cards, and ROMs.

Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be individually understood from the technical idea or prospect of the present disclosure.

100: Display device
110: Display
120: Processor

Claims (30)

In display devices,
display; and
Controlling the display to display a first screen including a plurality of objects;
Receives a voice signal corresponding to a voice input received using a microphone of the display device or an external device,
Processing a speech signal to obtain a speech recognition result including text,
If the above text corresponds to a single object among the multiple objects, an action related to the single object is performed,
a processor for controlling the display to display at least two icons if the text corresponds to at least two objects among the plurality of objects;
Each of the at least two icons above represents a number and is displayed adjacent to a corresponding object among the at least two objects,
The above numbers are expressed as words of the basic language set for speech recognition, a display device. In the first paragraph,
The above display device further includes a communication unit;
The above processor,
Controlling the communication unit to transmit the above voice signal to the server,
A display device that receives the voice recognition result from the server using the above communication unit. In the first paragraph,
The above processor,
A display device identifying at least one object containing the above text. In the third paragraph,
The above processor,
If the identified object is one, perform an action related to the identified object,
A display device that controls the display to display two icons adjacent to each of the two identified objects, if there are two objects identified above. In the first paragraph,
The above processor,
A display device that controls the display to display the at least two icons based on information about the positions of the at least two objects. In the first paragraph,
The input voice is a first voice input, the voice signal is a first voice signal, the voice recognition result is a first voice recognition result, the text is a first text, and the single object is a first single object.
The above processor,
Receive a second voice signal corresponding to a second voice input received using the microphone of the display device or the external device,
Processing the second speech signal to obtain a second speech recognition result including a second text,
A display device that performs an action related to the second single object when the second text corresponds to a second single object among the at least two objects. In the first paragraph,
The above processor,
A display device that sets the language set as the language of use in the settings menu provided by the above display device as the language that serves as the basis for voice recognition. In the first paragraph,
The single object and the at least two objects are selectable based on the speech signal,
A display device, wherein each of said single objects and said at least two objects include text objects included in a preset language. In the first paragraph,
The above processor,
A display device, controlling the display to display a phrase instructing the user to provide additional voice input including the number, together with at least two icons. In the first paragraph,
The above processor,
A display device that controls the display to display search results based on the text included in the voice recognition result when it is determined that the plurality of objects do not correspond to the voice recognition result. In a method for controlling a display device,
A step of displaying a first screen including a plurality of objects;
A step of receiving a voice signal corresponding to a voice input received using a microphone of the display device or an external device;
A step of processing a speech signal to obtain a speech recognition result including text;
If the above text corresponds to a single object among the plurality of objects, a step of performing an action related to the single object; and
If the above text corresponds to at least two objects among the plurality of objects, a step of displaying at least two icons is included;
Each of the at least two icons above represents a number and is displayed adjacent to a corresponding object among the at least two objects,
A control method in which the above numbers are expressed as words of a base language set for speech recognition. In Article 11,
The above control method is,
a step of transmitting the above voice signal to a server; and
A control method further comprising: a step of receiving the voice recognition result from the server. In Article 11,
The above control method is,
A control method further comprising: identifying at least one object comprising said text; In Article 13,
The steps for performing actions related to the above single object are:
If the identified object is one, perform an action related to the identified object,
The step of displaying at least two icons above is:
A control method for displaying two icons adjacent to each of the two identified objects when there are two objects identified above. In Article 11,
The step of displaying at least two icons above is:
A control method for displaying at least two icons based on information about the locations of at least two objects. In Article 11,
The input voice is a first voice input, the voice signal is a first voice signal, the voice recognition result is a first voice recognition result, the text is a first text, and the single object is a first single object.
The above control method is,
A step of receiving a second voice signal corresponding to a second voice input received using the microphone of the display device or the external device;
A step of processing the second speech signal to obtain a second speech recognition result including a second text; and
A control method further comprising: a step of performing an action related to the second single object if the second text corresponds to a second single object among the at least two objects. In Article 11,
The above control method is,
A control method further comprising: a step of setting a language set as a language used in a setting menu provided by the display device as a language that serves as the basis for voice recognition. In Article 11,
The single object and the at least two objects are selectable based on the speech signal,
A control method, wherein each of said single objects and said at least two objects include a text object included in a preset language. In Article 11,
The above control method is,
A control method further comprising: displaying a phrase instructing the user to provide an additional voice input including the number, together with at least two icons. In Article 11,
The above control method is,
A control method further comprising: a step of displaying a search result based on the text included in the voice recognition result, if it is determined that the plurality of objects do not correspond to the voice recognition result. A non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of a display device, cause the display device to perform an operation, the operation comprising:
A step of displaying a first screen including a plurality of objects;
A step of receiving a voice signal corresponding to a voice input received using a microphone of the display device or an external device;
A step of processing a speech signal to obtain a speech recognition result including text;
If the above text corresponds to a single object among the plurality of objects, a step of performing an action related to the single object; and
If the above text corresponds to at least two objects among the plurality of objects, a step of displaying at least two icons is included;
Each of the at least two icons above represents a number and is displayed adjacent to a corresponding object among the at least two objects,
A computer-readable recording medium in which the above numbers are expressed as words of a basic language set for voice recognition. In Article 21,
The above actions are,
a step of transmitting the above voice signal to a server; and
A computer-readable recording medium further comprising: a step of receiving the voice recognition result from the server. In Article 21,
The above actions are,
A computer-readable recording medium further comprising: a step of identifying at least one object comprising the text; In Article 23,
The steps for performing actions related to the above single object are:
If the identified object is one, perform an action related to the identified object,
The step of displaying at least two icons above is:
A computer-readable recording medium, which displays two icons adjacent to each of the two identified objects, if there are two objects identified above. In Article 21,
The step of displaying at least two icons above is:
A computer-readable recording medium that displays at least two icons based on information about the locations of at least two objects. In Article 21,
The input voice is a first voice input, the voice signal is a first voice signal, the voice recognition result is a first voice recognition result, the text is a first text, and the single object is a first single object.
The above actions are,
A step of receiving a second voice signal corresponding to a second voice input received using the microphone of the display device or the external device;
A step of processing the second speech signal to obtain a second speech recognition result including a second text; and
A computer-readable recording medium further comprising: a step of performing an action related to the second single object if the second text corresponds to a second single object among the at least two objects; In Article 21,
The above actions are,
A computer-readable recording medium further comprising: a step of setting a language set as a language used in a setting menu provided by the display device as a language that serves as the basis for voice recognition; In Article 21,
The single object and the at least two objects are selectable based on the speech signal,
A computer-readable storage medium, wherein each of said single objects and said at least two objects include text objects contained in a preset language. In Article 21,
The above actions are,
A computer-readable medium further comprising: a step of displaying a phrase instructing the user to provide additional voice input including the number, together with at least two icons mentioned above; In Article 21,
The above actions are,
A computer-readable recording medium further comprising: a step of displaying a search result based on the text included in the voice recognition result, if it is determined that the plurality of objects do not correspond to the voice recognition result.






.