KR102895054B1 - Electronic device and operating method for processing a voice input based on a gesture - Google Patents

Abstract

A method is provided for obtaining a voice input and a gesture corresponding to the voice input, obtaining an image including the surrounding environment of an electronic device based on a direction indicated by the gesture, extracting additional information related to the voice input from the image, obtaining request information regarding an operation requested to the electronic device by the voice input based on the additional information, and processing the voice input based on the gesture in an electronic device that processes the voice input based on the request information.

Description

Electronic device and operating method for processing a voice input based on a gesture

The present disclosure relates to an electronic device for processing voice input based on gestures and a method of operating the same.

Users can request various actions from their electronic devices via voice commands. For example, users can perform voice commands in a conversational manner with people around them, rather than with the electronic device.

In everyday conversation, even if the user does not explicitly indicate what he or she wants to request in the conversation, the other party to the conversation can clearly recognize the request included in the user's conversation by taking into account the user's gestures and the surrounding context.

Therefore, when a user gives a voice command in the manner of a normal conversation, there is a need for a method to perform an action according to the user's voice command in accordance with the user's intention, even if the object requested in the voice command is not clearly indicated.

The object of the present disclosure is to provide an electronic device for processing voice input based on gestures and an operating method thereof, in order to solve the aforementioned problem.

In addition, the present invention provides a computer-readable recording medium containing a program for executing the above method on a computer. The technical problems to be solved are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, a first aspect of the present disclosure may provide a method for processing a voice input based on a gesture in an electronic device, the method comprising: acquiring a voice input and a gesture corresponding to the voice input; acquiring an image including a surrounding environment of the electronic device based on a direction indicated by the gesture; extracting additional information related to the voice input from the image; acquiring request information regarding an operation requested to the electronic device by the voice input based on the additional information; and processing the voice input based on the request information.

In addition, a second aspect of the present disclosure provides an electronic device for processing a voice input based on a gesture, the electronic device including: an A/V input unit; an output unit for outputting a result of processing the voice input; and at least one processor for controlling the A/V input unit to obtain a voice input, obtain a gesture corresponding to the voice input, control the A/V input unit to obtain an image including a surrounding environment of the electronic device based on a direction indicated by the gesture, extract additional information related to the voice input from the image, obtain request information regarding an operation requested to the electronic device by the voice input based on the additional information, and process the voice input based on the request information.

In addition, a third aspect of the present disclosure can provide a recording medium storing a program for performing the method of the first aspect.

According to one embodiment, even if the information contained in the voice input is unclear, an action that matches the user's intention can be performed by clearly interpreting the voice input based on the user's gesture and the surrounding environment.

FIG. 1 is a drawing showing an example of receiving voice input according to one embodiment.
FIG. 2 is a drawing showing an example of capturing an image of the surrounding environment according to the direction indicated by a gesture according to one embodiment.
Figure 3 is a drawing showing an example of capturing an image of the surrounding environment according to one embodiment.
FIG. 4 is a block diagram for explaining the internal configuration of an electronic device (1000) according to one embodiment.
FIG. 5 is a block diagram for explaining the internal configuration of an electronic device (1000) according to one embodiment.
Figure 6 is a flowchart illustrating a method for processing voice input based on gestures according to one embodiment.
Figure 7 illustrates an example of obtaining request information from an image based on a gesture according to one embodiment.
FIG. 8 is a diagram illustrating an example of obtaining an image including a surrounding environment based on a gesture according to one embodiment.
FIG. 9 is a diagram illustrating an example of processing voice input based on gestures according to one embodiment.
Fig. 10 is a drawing showing an example of a gesture according to one embodiment.
FIG. 11 is a diagram illustrating an example of obtaining request information corresponding to voice input from an image according to one embodiment.
Figure 12 is a diagram showing an example of processing voice input based on gestures.
FIG. 13 is a diagram showing an example of processing voice input by acquiring an image based on a gesture according to one embodiment.
FIG. 14 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.
FIG. 15 is a diagram showing an example of obtaining information on the surrounding environment by processing voice input based on gestures according to one embodiment.
FIG. 16 is a diagram illustrating an example of collecting information and performing an action based on the collected information by processing voice input based on a gesture according to an embodiment.
FIG. 17 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.
FIG. 18 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.
FIG. 19 is a diagram showing an example of providing information requested by a user by processing voice input based on a gesture according to one embodiment.
FIG. 20 is a diagram showing an example of providing information requested by a user by processing voice input based on a gesture according to one embodiment.
FIG. 21 is a diagram illustrating an example of providing information requested by a user by processing voice input based on a gesture according to one embodiment.
FIG. 22 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.
FIG. 23 is a drawing showing an example of performing an operation of capturing an image by processing a voice input based on a gesture according to one embodiment.
FIG. 24 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.
FIG. 25 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.
FIG. 26 is a diagram showing an example of providing information requested by a user by processing voice input based on a gesture according to one embodiment.
FIG. 27 is a diagram showing an example of controlling an external device by processing voice input based on a gesture according to one embodiment.

Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement them. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity of description, and similar reference numerals have been used throughout the specification to indicate similar elements.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where the parts are "directly connected" but also the cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise stated.

The artificial intelligence-related functions according to the present disclosure are operated via a processor and memory. The processor may be comprised of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a Digital Signal Processor (DSP), a graphics-only processor such as a GPU or a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operating rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

The present invention will be described in detail with reference to the attached drawings below.

FIG. 1 is a drawing showing an example of receiving voice input according to one embodiment.

Referring to FIG. 1, the electronic device (1000) can perform an operation corresponding to the voice input upon receiving the voice input.

An electronic device (1000) according to one embodiment is an example of a device that provides a response in response to a user's voice input, and may be an electronic device equipped with a voice control function and capable of performing an operation according to a user's input.

An electronic device (1000) according to one embodiment may be implemented in various forms. For example, the electronic device (1000) described in this specification may include, but is not limited to, a digital camera, a smart phone, a laptop computer, a tablet PC, an e-book terminal, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a vehicle, and the like. The electronic device (1000) described in this specification may be a wearable device that can be worn by a user. A wearable device may include, but is not limited to, at least one of an accessory type device (e.g., a watch, a ring, an wristband, an ankleband, a necklace, glasses, a contact lens), a head-mounted device (HMD), a fabric or clothing-integrated device (e.g., an electronic garment), a body-attached device (e.g., a skin pad), or a bio-implantable device (e.g., an implantable circuit). Hereinafter, for the convenience of explanation, an example in which the electronic device (1000) is an artificial intelligence speaker device or a smart phone will be described.

According to one embodiment, the electronic device (1000) may be a device that receives a user's voice input through a conversational interface and performs an action in response to the user's voice input. The conversational interface may be an interface for the user that receives a voice input from the user and provides a response to the voice input from the user. According to one embodiment, the conversational interface may include, but is not limited to, a virtual assistant, an artificial intelligence assistant, etc. The virtual assistant or artificial intelligence assistant may be a software agent that processes a task requested by the user and provides a service specialized for the user.

An electronic device (1000) according to an embodiment may capture an image including a user's gesture (121, 122, 123) and an image of a surrounding environment captured based on a direction indicated by the user's gesture when a user's voice input is received. The gesture (121, 122, 123) according to an embodiment and the surrounding environment existing in the direction indicated by the gesture (121, 122, 123) may be captured together in a single image, or may be each captured separately and captured in multiple images.

Gestures (121, 122, 123) according to an embodiment may refer to various types of movements by which a user applies input to an electronic device (1000). Gestures according to an embodiment may include various types of movements of the user, such as movements of various parts of the user's hands, feet, head, etc., the user's gaze, and the user's posture.

According to one embodiment, the captured images including gestures (121, 122, 123) can be used to acquire gestures performed by the user. Therefore, the gestures (121, 122, 123) according to one embodiment are not limited to being acquired through captured images, and can be acquired using various methods. For example, the gestures (121, 122, 123) according to one embodiment can be acquired by various means through which gestures can be input, such as a remote control, an acceleration sensor, a tilt sensor, a gyro sensor, a 3-axis magnetic sensor, a stylus pen, etc.

An image of the surrounding environment, including the surrounding environment of an electronic device (1000) according to one embodiment, may be captured based on the direction indicated by a gesture obtained from the electronic device (1000). For example, the image of the surrounding environment may be captured by capturing a space existing in the direction indicated by the gesture by the electronic device (1000).

According to one embodiment, the electronic device (1000) can obtain requested information, which is information requested to the electronic device (1000) by voice input, by extracting additional information related to the voice input from the surrounding environment image. Therefore, according to one embodiment, based on the image captured based on the direction indicated by the gesture, the information requested to the electronic device (1000) by voice input is clearly determined, and thus an operation in accordance with the user's intention can be performed.

Additional information according to an embodiment may include information related to a voice input among various types of information that can be extracted from the surrounding environment image. For example, if the voice input includes a question asking about the status of "this", the additional information may include information about "this" extracted from the surrounding environment image. The additional information according to an embodiment may be information that can be used to clearly identify "this", and may include, for example, information about the location where "this" exists (e.g., on table number 1 in the center of the living room, or coordinate values indicating the location), information about the area of the surrounding environment image corresponding to "this", etc. Accordingly, the electronic device (1000) according to an embodiment may generate request information corresponding to the voice input by clearly identifying "this", which is the target of an operation by the voice input, based on the additional information.

In generating request information according to an embodiment, since the target of the operation (e.g., "this") is clearly determined, the operation to be performed by the electronic device (1000) in response to the voice input can also be determined based on the characteristics of the target. For example, if the target of the operation is a home appliance, the operation can be determined to determine the on/off status of the home appliance. Additionally, if the target of the operation is a person, the operation can be determined to search for the health status of the person. Without being limited to the above examples, various operations can be determined based on the target of the operation.

Accordingly, the request information according to one embodiment may include specific and clear information about the operation to be performed in the electronic device (1000) and its target, determined based on additional information, unlike voice input that includes ambiguous information.

An electronic device (1000) according to one embodiment can perform an operation based on request information. The request information according to one embodiment can include information regarding an operation determined based on additional information of an image. Therefore, according to one embodiment, even if the user's request included in the voice input is unclear, the operation to be performed in response to the voice input can be clearly determined based on the gesture and the image captured according to the gesture.

FIG. 2 is a drawing showing an example of capturing an image of the surrounding environment according to the direction indicated by a gesture according to one embodiment.

Referring to FIG. 2, an image of the surrounding environment captured according to the direction indicated by a gesture according to an embodiment can be obtained by capturing the image so that the space in the direction indicated by the user's gesture is included among the space the user is looking at.

In one embodiment, the space that the user is looking at can be inferred based on information about the angle (210) of the user's gaze and information about the posture (220) of the user uttering the voice input.

Information about the angle of gaze (210) according to one embodiment may represent the angle (e.g., 15 degrees) of the direction in which the user's gaze is directed relative to the horizontal plane. For example, information about the angle of gaze (210) may be obtained based on various information obtained from the user's movements, such as eye movements or the direction in which the front of the user's face is directed. Information about the angle of gaze (210) according to one embodiment may be obtained based on an image of the user's appearance captured for the purpose of obtaining a gesture.

Additionally, information regarding a posture (220) according to one embodiment may include skeleton information representing the user's posture. For example, the skeleton information may be obtained by extracting the user's joint movements based on an image of the user's appearance captured to acquire a gesture.

According to one embodiment, the work space the user is looking at can be inferred based on information about the angle of gaze (210) and information about the user's posture (220). Not limited to the above-described example, the work space the user is looking at can be inferred through various methods.

Additionally, information about the direction (230) of a gesture according to one embodiment may include information indicating the direction in which the user's fingertip points based on an image captured by the user to obtain the gesture.

An electronic device (1000) according to one embodiment can capture an image of the surrounding environment based on information about various movements of the user related to gestures, such as information about the angle of gaze (210), information about posture (220), and information about the direction of the gesture (230). For example, based on information about various movements of the user, an image of the surrounding environment including a space from which information necessary for performing an action in accordance with the user's intention can be extracted can be captured.

Figure 3 is a drawing showing an example of capturing an image of the surrounding environment according to one embodiment.

Referring to FIG. 3, the electronic devices (1000-1, 1000-2, 1000-3) can obtain an image including the surrounding environment of the electronic devices (1000-1, 1000-2, 1000-3) based on the direction indicated by the gesture. The electronic devices (1000-1, 1000-2, 1000-3) according to one embodiment are devices corresponding to the electronic device (1000) of FIG. 1, and are devices that can be implemented as smart glasses, a smart phone or tablet, a driving robot, etc., respectively.

Referring to 310, the electronic device (1000-1) can capture an image including the space of 311 based on the direction indicated by the user's gesture. According to one embodiment, the electronic device (1000-1) is a smart glass that is worn on the user's head and can capture an image from a first-person perspective based on the user's gaze.

An electronic device (1000-1) according to one embodiment can obtain an image including the surrounding environment of the electronic device (1000-1) by capturing an image including a space 311 corresponding to the direction indicated by a user's gesture among several spaces divided by dotted lines, which are spaces that can be captured by the electronic device (1000-1).

For example, among the spaces that can be captured by the electronic device (1000-1), an image including the surrounding environment of the electronic device (1000-1) can be obtained by capturing an enlarged image of the space of 311 according to the direction indicated by the user's gesture.

In addition, as another example, among the multiple cameras provided in the electronic device (1000-1), an image including the surrounding environment of the electronic device (1000-1) can be acquired by a camera capable of capturing the space of 311 according to the direction indicated by the user's gesture. Without being limited to the above-described example, the electronic device (1000-1) can acquire an image including the space of 311 according to the direction indicated by the user's gesture through various methods.

Referring to 320, the electronic device (1000-2) can capture an image including the space of 321 based on the direction indicated by the user's gesture. According to one embodiment, the electronic device (1000-2) is a tablet or a smartphone, and can capture an image from a first-person perspective based on the user's viewpoint.

According to one embodiment, by capturing an image including a space of 321 according to the direction indicated by the user's gesture among the spaces that can be captured by the electronic device (1000-2), an image including the surrounding environment of the electronic device (1000-2) can be obtained.

For example, the electronic device (1000-2) can capture an image of the surrounding environment so that an object (342) existing in the direction indicated by the gesture (341) is included, depending on the direction of the gesture (341), as in the example illustrated in 340.

According to one embodiment, depending on the movement that the user makes to the electronic device (1000-2), an image of a portion of the captured image, set according to the direction indicated by the gesture, can be acquired as an image including the surrounding environment.

According to one embodiment, a user's gesture may be acquired by a camera or various sensors provided in the electronic device (1000-2), but is not limited thereto, and may also be acquired by a camera or sensor of an external device (not shown).

Referring to 330, the electronic device (1000-3) can capture an image including a space 331, which is a space indicated by a user's gesture, based on the direction indicated by the user's gesture. According to one embodiment, the electronic device (1000-3) is a robot vacuum cleaner or a driving robot, and can capture an image from a third-person perspective, which is an external perspective.

An electronic device (1000-3) according to one embodiment can acquire information regarding the direction indicated by a user's gesture from the captured image by capturing the user's appearance. Information regarding the direction indicated by a gesture according to one embodiment can be acquired based on various user movements extracted from the image, such as the user's gaze, the user's posture, and the direction indicated by the user's finger.

Accordingly, the electronic device (1000-2) according to one embodiment can obtain an image including the surrounding environment of the electronic device (1000-2) by capturing an image including a space of 331 according to the direction indicated by the user's gesture among the spaces that can be captured by the electronic device (1000-2).

FIG. 4 is a block diagram for explaining the internal configuration of an electronic device (1000) according to one embodiment.

FIG. 5 is a block diagram for explaining the internal configuration of an electronic device (1000) according to one embodiment.

Referring to FIG. 4, the electronic device (1000) may include a processor (1300), an output unit (1200), and an A/V input unit (1600). However, not all of the components illustrated in FIG. 4 are essential components of the electronic device (1000). The electronic device (1000) may be implemented with more components than the components illustrated in FIG. 4, or may be implemented with fewer components than the components illustrated in FIG. 4.

For example, as illustrated in FIG. 5, the electronic device (1000) according to one embodiment may further include a user input unit (1100), a sensing unit (1400), a communication unit (1500), an output unit (1200), and a memory (1700) in addition to a processor (1300), an output unit (1200), and an A/V input unit (1600).

The user input unit (1100) refers to a means for a user to input data for controlling an electronic device (1000). For example, the user input unit (1100) may include, but is not limited to, a key pad, a dome switch, a touch pad (contact electrostatic capacitance type, pressure resistive film type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.), a jog wheel, a jog switch, etc.

According to one embodiment, the user input unit (1100) can receive various types of user input for processing voice input based on gestures. For example, the user input unit (1100) can receive a user's touch input or key input for processing voice input based on gestures. Without being limited to the above-described examples, the user input unit (1100) can receive various types of user input.

The output unit (1200) can output an audio signal, a video signal, or a vibration signal, and the output unit (1200) can include a display unit (1210), an audio output unit (1220), and a vibration motor (1230).

The display unit (1210) displays and outputs information processed in the electronic device (1000). According to one embodiment, the display unit (1210) can display the result of processing voice input based on gestures.

Meanwhile, when the display unit (1210) and the touchpad form a layer structure to form a touch screen, the display unit (1210) can be used as an input device in addition to an output device. The display unit (1210) can include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display. In addition, depending on the implementation form of the electronic device (1000), the electronic device (1000) may include two or more display units (1210).

The audio output unit (1220) outputs audio data received from the communication unit (1500) or stored in the memory (1700). According to one embodiment, the audio output unit (1220) can output the result of processing voice input based on gestures.

The vibration motor (1230) can output a vibration signal. Furthermore, the vibration motor (1230) can also output a vibration signal when a touch is input to the touchscreen. According to one embodiment, the vibration motor (1230) can output the result of processing a voice input based on a gesture.

The processor (1300) typically controls the overall operation of the electronic device (1000). For example, the processor (1300) can control the user input unit (1100), output unit (1200), sensing unit (1400), communication unit (1500), A/V input unit (1600), etc., by executing programs stored in the memory (1700).

The electronic device (1000) may include at least one processor (1300). For example, the electronic device (1000) may include various types of processors, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an NPU (Neural Processing Unit).

The processor (1300) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor (1300) from the memory (1700) or received via the communication unit (1500) and provided to the processor (1300). For example, the processor (1300) may be configured to execute instructions according to program code stored in a storage device such as a memory.

According to an embodiment, a processor (1300) may acquire a voice input and a gesture corresponding to the voice input, and may acquire an image including the surrounding environment of the electronic device (1000) based on a direction indicated by the gesture. In addition, the processor (1300) may extract additional information related to the voice input from the image. According to an embodiment, the additional information may include information that can be extracted from the image for processing the voice input, for example, information for clearly identifying an object (e.g., an object, a certain range of an area) on which an operation corresponding to the voice input is performed.

According to an embodiment, the processor (1300) may obtain request information regarding an operation requested to the electronic device (1000) by a voice input based on additional information. The request information according to an embodiment may include clear information for performing the operation as it is obtained based on the additional information, which is information extracted from an image, even if the voice input is inaccurate. The request information according to an embodiment may include clear information corresponding to the voice input, in which inaccurate information included in the voice input is replaced based on the additional information extracted from the image.

According to one embodiment, a processor (1300) may perform an operation for processing voice input based on request information.

The sensing unit (1400) can detect the status of the electronic device (1000) or the status around the electronic device (1000) and transmit the detected information to the processor (1300).

The sensing unit (1400) may include at least one of a geomagnetic sensor (1410), an acceleration sensor (1420), a temperature/humidity sensor (1430), an infrared sensor (1440), a gyroscope sensor (1450), a position sensor (e.g., GPS) (1460), a pressure sensor (1470), a proximity sensor (1480), and an RGB sensor (illuminance sensor) (1490), but is not limited thereto.

A sensing unit (1400) according to one embodiment can be used to obtain a gesture corresponding to a voice input based on information detected by various sensors.

The communication unit (1500) may include one or more components that enable the electronic device (1000) to communicate with a server (2000) or an external device (not shown). For example, the communication unit (1500) may include a short-range communication unit (1510), a mobile communication unit (1520), and a broadcast receiving unit (1530).

The short-range wireless communication unit (1510) may include, but is not limited to, a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, a UWB (ultra wideband) communication unit, an Ant+ communication unit, etc.

The mobile communication unit (1520) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signals may include various types of data, such as voice call signals, video call signals, or text/multimedia message transmission and reception.

The broadcast receiver (1530) receives broadcast signals and/or broadcast-related information from an external source via a broadcast channel. The broadcast channel may include a satellite channel or a terrestrial channel. Depending on the implementation example, the electronic device (1000) may not include the broadcast receiver (1530).

According to one embodiment, the communication unit (1500) can transmit and receive various data necessary for processing voice input based on gestures. For example, when a gesture corresponding to a voice input is input by an external device (not shown), the communication unit (1500) can receive the gesture input from the external device (not shown).

The A/V (Audio/Video) input unit (1600) is for inputting audio signals or video signals, and may include a camera (1610) and a microphone (1620), etc. The camera (1610) can obtain image frames, such as still images or moving images, through an image sensor in video call mode or shooting mode. Images captured through the image sensor can be processed through a processor (1300) or a separate image processing unit (not shown).

In one embodiment, a camera (1610) can capture an image including a gesture corresponding to a voice input. In one embodiment, a gesture corresponding to a voice input can be acquired based on the image captured by the camera (1610).

Additionally, the camera (1610) according to one embodiment can capture an image including the surrounding environment of the electronic device (1000) based on the direction indicated by the gesture.

According to one embodiment, a microphone (1620) can receive a user's voice input. According to one embodiment, when a user's voice input is detected by the microphone (1620), at least one image including a gesture corresponding to the voice input and an image including the surrounding environment of the electronic device (1000) based on the direction indicated by the gesture can be captured by the camera (1610). The microphone (1620) receives an external acoustic signal and processes it into electrical voice data.

At least one of a voice input, an image including a gesture, and an image including the surrounding environment of the electronic device (1000) according to an embodiment may be acquired by the A/V input unit (1600) of the electronic device (1000), but is not limited thereto, and may be acquired by an external device (not shown) and received through the communication unit (1500).

The memory (1700) can store a program for processing and controlling the processor (1300), and can also store data input to or output from the electronic device (1000).

According to one embodiment, the memory (1700) may store data necessary for processing voice input based on gestures. For example, the memory (1700) may store a pre-trained artificial intelligence model to extract additional information related to voice input from an image and obtain request information based on the additional information.

An artificial intelligence model for extracting additional information according to an embodiment can be used to obtain additional information by identifying an object or a certain range of areas from an image and obtaining location information (e.g., coordinate values) for the identified object or the certain range of areas.

The memory (1700) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

Programs stored in memory (1700) can be classified into multiple modules according to their functions, for example, a UI module (1710), a touch screen module (1720), a notification module (1730), etc.

The UI module (1710) can provide specialized UIs, GUIs, etc. that are linked to the electronic device (1000) for each application. The touch screen module (1720) can detect a user's touch gesture on the touch screen and transmit information about the touch gesture to the processor (1300). According to some embodiments, the touch screen module (1720) can recognize and analyze a touch code. The touch screen module (1720) can also be configured as separate hardware including a controller.

To detect touch or proximity touch on a touchscreen, various sensors may be installed inside or near the touchscreen. One example of a sensor for detecting touch on a touchscreen is a tactile sensor. A tactile sensor is a sensor that detects the contact of a specific object at or beyond the level of human touch. A tactile sensor can detect various information, such as the roughness of the contact surface, the hardness of the contact object, and the temperature of the contact point.

User touch gestures may include tap, touch and hold, double tap, drag, pan, flick, drag and drop, and swipe.

The notification module (1730) can generate a signal to notify the occurrence of an event in the electronic device (1000).

Figure 6 is a flowchart illustrating a method for processing voice input based on gestures according to one embodiment.

Referring to FIG. 6, in step 610, an electronic device (1000) according to an embodiment may acquire a voice input and a gesture corresponding to the voice input. The gesture according to an embodiment may be acquired based on a movement by a user uttering the voice input.

Gestures according to one embodiment may be acquired based on various forms of movement, such as the user's gaze, posture, etc., as well as the movements of various body parts of the user. The electronic device (1000) according to one embodiment may perform an action in accordance with the user's intention by acquiring additional information using gestures, even when information included in a voice input is unclear.

According to one embodiment, the electronic device (1000) can acquire a gesture by capturing an image including the user's gesture in order to acquire a gesture corresponding to the voice input when a voice input is detected. Without being limited to the above-described example, the electronic device (1000) can acquire the user's gesture using various methods other than video capturing.

In step 620, the electronic device (1000) according to one embodiment may acquire an image including the surrounding environment based on the direction of the gesture. The electronic device (1000) according to one embodiment may identify a space related to the voice input based on the direction indicated by the gesture, and capture an image including the space related to the voice input.

In step 630, the electronic device (1000) according to one embodiment may extract additional information related to the voice input from the image captured in step 620. The electronic device (1000) according to one embodiment may perform voice recognition on the voice input, thereby obtaining at least one text corresponding to the voice input, and may determine a text representing ambiguous information among the obtained texts. The additional information according to one embodiment may include various types of information that may be used to clarify ambiguous information included in the voice input.

For example, if the object on which an action is performed is ambiguous, such as “this” or “over there,” among the information included in the voice input, the electronic device (1000) can determine an object or area judged to be the object on which an action is performed from the image, and extract at least one of identification information and location information corresponding to the determined object or area as additional information.

In step 640, the electronic device (1000) according to one embodiment may obtain request information in which ambiguous information is replaced with clear information based on the additional information extracted in step 630. For example, among the text corresponding to the voice input, request information may be obtained in which "over there" is replaced with "(location information corresponding to over there)", which is information included in the additional information. The request information according to one embodiment may include information regarding an operation requested to the electronic device (1000) via the voice input.

Accordingly, at step 650, the electronic device (1000) can process the voice input based on request information that includes clear information. In one embodiment, the request information may include information in which ambiguous information is replaced with clear information based on a gesture indicating the user's intention. Accordingly, the electronic device (1000) according to one embodiment can process the voice input by performing an action in accordance with the user's intention based on the request information.

Figure 7 illustrates an example of obtaining request information from an image based on a gesture according to one embodiment.

Referring to FIG. 7, request information corresponding to voice input can be obtained based on an image acquired according to a user's gesture (710, 720, 730, 740).

According to one embodiment, depending on the direction of the user's gesture (710), an image corresponding to an area of 711 in the image may be acquired. If the voice input is "What color is it?", since it is not clear what "it" is in the text of the voice input, identification information about "it" may be acquired as additional information based on the image of 711. Since "it" is an expression indicating a relatively close object, request information may be acquired based on the additional information including identification information corresponding to a relatively close "red tomato" among the objects included in the image of 711.

In addition, when the voice input according to one embodiment is "What color is it?", since it is not clear what "that" is in the text of the voice input, identification information about "that" can be obtained as additional information based on the image of 711. Since "that" is an expression indicating a relatively distant object, the request information can be obtained based on the additional information including identification information corresponding to a "yellow tomato" located at a relatively distant location among the objects included in the image of 711.

In one embodiment, depending on the direction of each gesture (720, 730, 740), images corresponding to areas 721, 731, and 741 of the images may be acquired. If the voice input is "How many she holds?", since it is not clear what "she" is in the text of the voice input, depending on the area indicated by each gesture, identification information about "she" in the voice input may be acquired as additional information based on the images 721, 731, or 741. In one embodiment, request information may be acquired based on the additional information including identification information indicating the second, third, and fourth girls from the left, respectively, for "she" in the voice input corresponding to the gestures (720, 730, 740).

FIG. 8 is a diagram illustrating an example of obtaining an image including a surrounding environment based on a gesture according to one embodiment.

Referring to FIG. 8, at 810, 820 and 830, the image can be captured so as to include the space existing in the direction pointed by each gesture.

At 810, voice inputs such as “What is this?”, “Is it a lamp or a plant?”, and “Is it switch on/off?” may be received. In an embodiment, since it is not clear what “this” or “it” is among the information included in the voice input, the electronic device (1000) may obtain identification information of an object (811) that may correspond to “this” or “it” as additional information from an image of the surrounding environment. In an embodiment, based on the additional information, request information is obtained in which “this” or “it” in the voice input is replaced with identification information of the object (811), thereby processing the voice input.

At 820, voice inputs such as “How many people are here?”, “Are these people men?”, and “What are these people doing?” may be received as voice inputs. In an embodiment, since it is not clear what “here” and “these people” are among the information included in the voice input, the electronic device (1000) may obtain, as additional information, identification information indicating an area of the image corresponding to “here” and identification information of objects (821, 822) corresponding to “these people” from an image in which the surrounding environment is captured. In an embodiment, based on the additional information, request information is obtained in which “here” and “these people” of the voice input are replaced with identification information of an image area and identification information of objects (821, 822), respectively, thereby processing the voice input.

At 820, voice inputs such as "Where is he sitting?", "Where is her bag?", etc. may be received as voice inputs. In an embodiment, since it is not clear what "he" and "she" are among the information included in the voice input, the electronic device (1000) may acquire identification information for objects 822 and 821 that may correspond to "he" and "she", respectively, as additional information from a captured image of the surrounding environment. According to an embodiment, based on the additional information, request information is acquired in which "he" and "she" in the voice input are replaced with identification information representing objects 822 and 821, thereby processing the voice input.

At 830, voice inputs such as "Are these people parents?" and "Are they busy?" can be received. In one embodiment, the surrounding environment image can be captured to include objects 841 and 842 depending on the direction of the gesture.

In an embodiment, an electronic device (1000) may acquire identification information for objects 831 and 832 that may correspond to "these, they" from an image of a surrounding environment, as additional information, since it is not clear what "these, they" are among the information included in a voice input. However, depending on the direction of the gesture, the object 843 may be excluded from the acquisition of the additional information. In an embodiment, based on the additional information, the voice input may be processed by acquiring request information in which "these, they" in the voice input is replaced with identification information representing objects 831 and 832.

FIG. 9 is a diagram illustrating an example of processing voice input based on gestures according to one embodiment.

Referring to FIG. 9, at 910, as the user's gesture is input while moving, an image of the surrounding environment can be captured so as to include the space in the direction each of the moving gestures points.

According to one embodiment, when "Clean here" is received as a voice input, an image of the surrounding environment including the 911 space may be captured according to the direction indicated by the gesture corresponding to the voice input. In one embodiment, since it is unclear what "here" is among the information included in the voice input, the electronic device (1000) may acquire location information about the 911 space that may correspond to "here" as additional information from the captured image of the surrounding environment. According to one embodiment, based on the additional information, the voice input may be processed by acquiring request information in which "here" of the voice input is replaced with location information about the 911 space.

In addition, according to one embodiment, when a gesture moves and a voice input of "Here?" is received, an image of the surrounding environment including spaces 912 and 913 may be captured according to the directions indicated by the gesture corresponding to the voice input. In one embodiment, since it is not clear what "Here?" is among the information included in the voice input, the electronic device (1000) may obtain, as additional information, location information for spaces 912 and 913 that may correspond to "Here?" from an image of the surrounding environment. According to one embodiment, based on the additional information, request information is obtained in which "Here?" of the voice input is replaced with location information for spaces 912 and 913, thereby processing the voice input.

In one embodiment, at 920, additional information may be acquired from an image of the surrounding environment based on a user's gesture. When a voice input such as "Volume like that?" is received, an image of the surrounding environment including an object (920) may be captured based on the direction indicated by the gesture.

In an electronic device (1000) according to one embodiment, since it is not clear what “that” is among the information included in the voice input, identification information (e.g., living room TV) regarding an object (920) included in an image of the surrounding environment as a target related to “that” can be obtained as additional information.

In one embodiment, based on the additional information, "that" may be determined to represent information related to an action that can be performed on the object (920). For example, based on an action corresponding to a gesture of moving a finger in a preset arrow direction, the gesture of 920 may be determined to be an action of adjusting the volume level. Accordingly, by acquiring request information in which "that" is replaced with a value representing the volume level corresponding to the gesture, the voice input may be processed.

In one embodiment, at 930, additional information may be acquired from an image of the surrounding environment based on a user's gesture. When a voice input of "Order that pizza?" is received, an image of the surrounding environment including an object (931) may be captured based on the direction indicated by the gesture.

According to an embodiment, an electronic device (1000) may obtain, as additional information, identification information regarding an object (931) included in an image of a surrounding environment as an object related to “that” when it is unclear what “that” is among the information included in a voice input.

In one embodiment, based on the additional information, "that" may be determined to represent information related to an action that can be performed on the object (931). For example, based on an action corresponding to a gesture of moving a hand in a preset arrow direction, the gesture of 930 may be determined to represent pizza. Accordingly, by acquiring request information in which "that" is replaced with "pizza" corresponding to the gesture, the voice input may be processed.

Fig. 10 is a drawing showing an example of a gesture according to one embodiment.

Referring to FIG. 10, at least one gesture may be preset to correspond to a preset text during voice input. For example, according to one embodiment, gestures may be preset with texts as shown in Table 1 below.

prepositional determiner demonstrative pronoun possessive pronoun Quantifier all, all of, half, both, double, twice, quite, rather, such, what, etc. This, these, that, those my, our, your, his, her, its, their much, some, no, any, many, enough, several, lot of, plenty of, little, few, all, both, another, etc.

1010 to 1040 are all gestures for pointing at an object with a hand, and can be preset as gestures corresponding to texts corresponding to prepositional determiners, demonstrative pronouns, and possessive pronouns in Table 1 among the texts included in the voice input. Accordingly, gestures 1010 to 1040 can be used as gestures for pointing at objects that can be expressed as prepositional determiners, demonstrative pronouns, and possessive pronouns.

1050 and 1060 are gestures that can be expressed as the direction of the head and the direction of the gaze, respectively, and, like 1010 to 1040, can be preset as gestures corresponding to texts corresponding to the prepositional determiners, demonstrative pronouns, and possessive pronouns of Table 1. Accordingly, the gestures of 1050 and 1060 can be used as gestures for pointing to objects that can be expressed as prepositional determiners, demonstrative pronouns, and possessive pronouns.

1070 to 1080 are gestures that can be expressed by moving the hand in the direction of the arrow, and can be preset as gestures corresponding to the text corresponding to the quantifiers in Table 1. Accordingly, gestures 1070 to 1080 can be used as gestures to indicate quantities.

FIG. 11 is a diagram illustrating an example of obtaining request information corresponding to voice input from an image according to one embodiment.

Referring to FIG. 11, at 1110, an electronic device (1000) according to an embodiment can obtain request information by replacing at least one text corresponding to a voice input based on information extracted from an image.

An electronic device (1000) according to an embodiment may detect, based on an image, a text portion to be replaced, "child" (1114), from among at least one text corresponding to a voice input, "Where is the child sitting?" For example, among the at least one text, a text indicating an object on which an action corresponding to the voice input is to be performed may be detected as the text portion to be replaced based on the image. Without being limited to the above-described example, the text portion to be replaced may be detected based on the image according to various criteria and methods.

An electronic device (1000) according to an embodiment can detect at least one object (1111, 1112, 1113) in an image. In addition, the electronic device (1000) can determine an object (1111) corresponding to a “child” (1114) based on at least one of the results of analyzing the characteristics of each detected object (1111, 1112, 1113) and the direction indicated by the gesture.

At 1120, the electronic device (1000) can obtain the requested information by replacing "child" (1114) with "he" (1121) having information corresponding to the object (1111), based on the object (1111) corresponding to "child" (1114). According to one embodiment, the electronic device (1000) can replace "child" (1114) with a simpler expression, "he" (1121), to facilitate text processing. Without being limited to the above example, the text corresponding to the object (1111) can be replaced with various expressions based on the object (1111).

Accordingly, the request information according to one embodiment may further include information about an area of an object (1111) corresponding to the text in the image, in addition to at least one text.

An electronic device (1000) according to one embodiment can process a voice input by analyzing features extracted from an image based on request information corresponding to the voice input.

Figure 12 is a diagram showing an example of processing voice input based on gestures.

Referring to FIG. 12, an electronic device (1000) according to an embodiment may perform an operation of processing a voice input based on at least one image (1202) including a surrounding environment, request information (1201) including text corresponding to a voice input, and information about a gesture (1203).

An electronic device (1000) according to an embodiment may perform an operation of processing the voice input using an image including a surrounding environment. However, in order to obtain more information necessary for processing the voice input, the electronic device (1000) according to an embodiment may additionally obtain an image including a surrounding environment of the electronic device (1000) that includes a different scene from the image. Accordingly, the electronic device (1000) according to an embodiment may perform an operation of processing the voice input using not only an image obtained according to a gesture, but also an additionally obtained image including a different scene from the image.

An electronic device (1000) according to one embodiment may determine whether to acquire additional images based on the request information. For example, if it is determined that an object included in the request information is not included in the surrounding environment image, an additional image including the object may be acquired, thereby processing the voice input.

Without being limited to the above-described examples, the electronic device (1000) can process voice input based on at least one image (1202) of the surrounding environment acquired through various methods and request information (1101) corresponding to the voice input.

An electronic device (1000) according to one embodiment can extract a response to the request information (1201) by using VQA (Visual Question Answering) technology based on text features extracted from at least one text of the request information (1201) and image features extracted from an image corresponding to the request information (1201).

For example, according to the VQA technology, the electronic device (1000) can obtain a result value corresponding to the request information (1201) based on text features extracted from the text of the request information (1201) and image features extracted from at least one image (1202). According to an embodiment, the electronic device (1000) can obtain a response to the request information (1201) by fusing features extracted from the text and the image (1202) according to the VQA technology.

Not limited to the above-described example, the electronic device (1000) can obtain response information corresponding to the request information (1201) based on the request information (1201) and the image (1202) according to various methods other than the VQA technology.

According to one embodiment, the request information (1201) input to 1221 may include at least one text. In addition, if the request information (1201) includes non-text information, the information may be converted to text or deleted before being input to 1221.

For example, if the request information (1201) includes information about an area of the surrounding environment image related to the action target, the information may be deleted, and instead of the entire area of the image 1202 in 1211, an image corresponding to an area of the surrounding environment image related to the action target may be input. Accordingly, according to one embodiment, if the request information (1201) includes information other than text, the information may be deleted from the request information (1201), and the information may be reflected and input into the image input to 1211, thereby performing text feature extraction and image feature extraction in 1221 and 1222 according to the VQA technology.

In addition, the request information (1201) may be input to 1221 by including information about the area of the surrounding environment image related to the action target as a voice command in the request information (1201). For example, in the voice command of the request information (1201), the location information of the action target expressed as text is inserted, so that non-text information can be converted into text and then processed.

In addition, according to one embodiment, instead of request information (1201), a user's voice command may be directly input at 1221. For example, in the operation of extracting combined features (1224) and outputting the result (1231) according to the VQA technology below, text features may be corrected based on information obtained from image features, and combined features may be extracted at 1224 based on the image features and the corrected text features.

Among the text of request information (1201) according to one embodiment, text from which text features are to be extracted can be identified. For example, text having the characteristics of a question or command can be determined by a pre-trained artificial intelligence model (e.g., RNN).

In one embodiment, for text having the characteristics of a question or command, response information based on the combined characteristics can be obtained from the request information (1201). Without being limited to the above-described example, response information can be generated in various ways based on the text of the request information (1201).

In steps 1221 to 1224, an electronic device (1000) according to an embodiment can obtain response information for request information (1201) by combining features detected from an image (1202) and text of request information (1201).

In 1221, an electronic device (1000) according to an embodiment can extract text features from at least one text of request information (1201).

In 1211, an electronic device (1000) according to an embodiment can detect an object for processing request information (1201) in an image (1202). The request information (1201) according to an embodiment can include at least one text, and based on the image features of an object area corresponding to each text, combined features can be extracted.

For example, the electronic device (1000) can detect, in accordance with FIG. 11, an area including an object corresponding to at least one text from which the text feature is extracted among the request information (1201) in the image (1202).

Not limited to the above-described example, at least one object may be detected by a pre-trained artificial intelligence model (e.g., Faster R-CNN (Region of Convolutional Neural Network)) to detect an object from an image (1202).

In 1222, an electronic device (1000) according to an embodiment can extract image features from an area of an object detected in 1211. For example, the electronic device (1000) can extract various features from an area of an image including an object by using a pre-learned artificial intelligence model (e.g., RNN) to extract image features from the area of the image.

At 1212, an electronic device (1000) according to an embodiment may obtain a gesture corresponding to an object detected in an image (1202) and 1211. At 1223, according to an embodiment, depending on the type of gesture, weight values to be applied to the request information (1201) and the object when combining features may be determined.

For example, if the gesture is simply a pointing gesture pointing to an object of the image (1202), such as 1010 to 1060 of FIG. 10, the weight for the text feature of 1221 may be determined higher. On the other hand, if the gesture is a gesture that includes information related to an object of the image (1202), such as 1070 to 1090 of FIG. 10, for example, a gesture that includes information related to the action (e.g., quantity, shape information) in the gesture itself, such as gestures of 1070, 1080, and 1090 of FIG. 10, the weight for the image feature acquired in 1222 may be determined higher.

Not limited to the above examples, in one embodiment, weights determined in various ways based on gestures are applied to image features and text features, and then combined features can be extracted.

In 1224, an electronic device (1000) according to an embodiment may extract a combined feature from weighted image features and text features. In an embodiment, the electronic device (1000) may also extract a combined feature from an image feature of an area corresponding to an area indicated by a gesture among the image features based on a gesture.

For example, if the area of the object detected in 1211 is wide, combined features can be extracted based on image features contained in a portion of the object area indicated by the gesture. Not limited to the example described above, combined features can be extracted using various methods from weighted image features and text features.

In 1231, an electronic device (1000) according to an embodiment may output response information for request information (1201) as a result of processing a voice input based on combined characteristics. According to an embodiment, response information, which is a result of processing the voice input according to the combined characteristics, may be output further based on information about the context of the user of the electronic device (1000), for example, information about the user's characteristics, preferences, status, etc.

FIG. 13 is a diagram showing an example of processing voice input by acquiring an image based on a gesture according to one embodiment.

1310 of FIG. 13 illustrates an image captured by an electronic device (1000) of a gesture (1311) and the surrounding environment of the electronic device (1000). According to one embodiment, a voice input may be processed based on an area of 1312 in the image, depending on the direction indicated by the finger of the gesture (1311). For example, a voice input may be processed based on request information obtained based on additional information extracted based on the image area of 1312. In addition, a voice input may be processed by obtaining response information according to VQA technology based on the request information and the image area of 1312.

1320 shows an image captured by an electronic device (1000) of a gesture (1321) and the surrounding environment of the electronic device (1000). According to one embodiment, for a voice input (1323), an object (1322) included in the image may be extracted as an object related to the voice input, depending on the direction indicated by the finger of the gesture (1321). According to one embodiment, additional information may be extracted based on information about the object (1322), and request information corresponding to the voice input (1323) may be obtained based on the additional information. In addition, based on an image area including the request information and the extracted object (1322), response information may be obtained according to VQA technology, thereby processing the voice input (1323).

1330 shows an image captured by an electronic device (1000) of a gesture (1334) and the surrounding environment of the electronic device (1000). According to one embodiment, for a voice input (1331), an object (1333) included in the image can be extracted as an object related to the voice input according to the direction (1335) pointed by the finger of the gesture (1334). In addition, a region of interest (ROI) related to the voice input (1331) can be additionally set in the image according to a user's touch input to the electronic device (1000). Accordingly, the object (1333) can be identified in the image captured by the electronic device (1000) based on not only the direction (1335) of the gesture but also the additionally set ROI region. According to one embodiment, additional information may be extracted based on information about an object (1333), and based on the additional information, request information corresponding to a voice input (1331) may be acquired. In addition, based on an image area including the request information and the extracted object (1333), response information may be acquired according to VQA technology, thereby processing the voice input (1331). A result (1336) according to the response information according to one embodiment may be output from the electronic device (1000) for the object (1333), as in the example illustrated in 1330.

FIG. 14 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.

Referring to FIG. 14, the electronic device (1000) can acquire gestures (1401, 1402) based on the direction of the user's gaze and fingers upon receiving the user's voice input (1404). The electronic device (1000) of FIG. 14 may be a mobile robot capable of guiding customers while moving within a store.

An electronic device (1000) according to an embodiment can acquire an image captured to include a surrounding environment related to a voice input (1404) according to a direction indicated by a user's gesture (1401, 1402). An electronic device (1000) according to an embodiment can process a user's voice input (1404) based on the acquired image.

For example, the electronic device (1000) may acquire an image captured to include the appearances (1403) of guests present in the direction indicated by the user's gestures (1401, 1402) according to the user's voice input (1404). In one embodiment, the electronic device (1000) may acquire an image area including the appearances (1403) of guests as an object area corresponding to "those customers" in the voice input (1404). Accordingly, the electronic device (1000) may acquire request information based on the information acquired from the image area. In one embodiment, the electronic device (1000) may move to the location (1403) of the guests according to the request information and perform an operation for entertaining the guests.

FIG. 15 is a diagram showing an example of obtaining information on the surrounding environment by processing voice input based on gestures according to one embodiment.

Referring to FIG. 15, the electronic device (1000) can acquire gestures (1501, 1502) based on the direction of the user's gaze and fingers upon receiving the user's voice input (1503). The electronic device (1000) of FIG. 15 may be a mobile robot capable of collecting various information while moving within a house.

An electronic device (1000) according to an embodiment can acquire an image captured to include a surrounding environment related to a voice input (1503) according to a direction indicated by a user's gesture (1501, 1502). An electronic device (1000) according to an embodiment can process a user's voice input (1503) based on the acquired image.

For example, the electronic device (1000) may obtain an image captured to include a person (1504) present in the direction indicated by the user's gesture (1501, 1502) according to the user's voice input (1503). According to an embodiment, the electronic device (1000) may obtain an image area including a baby (1504) as an object area corresponding to "son Freddy" in the voice input (1503). For example, when "son Freddy" is uttered in the voice input (1503), the electronic device (1000) may obtain an image including a baby (1504) according to the direction indicated by the user's gesture (1501, 1502). Accordingly, the electronic device (1000) may obtain request information based on the information obtained from the image.

An electronic device (1000) according to an embodiment may perform an operation of collecting information according to a user's voice input (1503) based on requested information. For example, the electronic device (1000) may perform an operation of collecting information extracted from an image including a baby (1504) based on the user's voice input (1503).

FIG. 16 is a diagram illustrating an example of collecting information and performing an action based on the collected information by processing voice input based on a gesture according to an embodiment.

Referring to FIG. 16, the electronic device (1000) can acquire gestures (1602, 1603) based on the direction of the user's gaze and fingers by receiving the user's voice input (1604, 1606). The electronic device (1000) of FIG. 16 may be a mobile robot that can collect various information while moving inside a house.

An electronic device (1000) according to an embodiment can acquire an image captured to include a surrounding environment related to a voice input (1604, 1606) according to a direction indicated by a user's gesture (1602, 1603). An electronic device (1000) according to an embodiment can collect information about the surrounding environment based on the acquired image and according to the user's voice input (1604, 1606).

For example, the electronic device (1000) may acquire images that are captured to include objects (1605, 1607) that exist in the direction indicated by the user's gesture (1602, 1603) based on the user's voice input (1604, 1606). According to an embodiment, the electronic device (1000) may extract additional information based on the images, and, based on the extracted additional information, acquire information about an object (1605, 1607) corresponding to "this" included in the voice input (1604, 1606) as request information. According to an embodiment, the voice input (1604, 1606) may be processed based on the request information.

According to an embodiment, an electronic device (1000) may process a voice input (1604) by performing an operation to pre-store an object (1605) included in an image as a “dangerous” object based on a user’s voice input (1604). In addition, the electronic device (1000) may process a voice input (1606) by performing an operation to pre-store an object (1607) included in an image as a “toy” based on a user’s voice input (1606).

In 1611, the electronic device (1000) can perform an operation based on information about objects (1605, 1607) that have been stored in advance according to the user's voice input (1604, 1606) of 1601. In one embodiment, the electronic device (1000) can perform an operation by detecting objects (1605, 1607) that have been stored in advance in an image of the surrounding environment based on information about the objects (1605, 1607).

According to an embodiment, the electronic device (1000) may perform an action based on the detected object (1605) when an object (1605) stored as a “dangerous” object among the pre-stored objects is detected. For example, when a baby (1612) is detected together with the “dangerous” object (1605), the electronic device (1000) may perform an action to notify the user that the “dangerous” object (1605) and the baby (1612) are nearby. In addition, the electronic device (1000) may additionally perform an action to notify the user whether the baby (1612) is currently in a dangerous state by analyzing the behavior of the baby (1612) from the captured image.

In addition, the electronic device (1000) according to one embodiment may perform an operation based on the detected object (1607) when an object (1607) stored as a “toy” among pre-stored objects is detected. For example, when a baby (1613) is detected together with an object (1607) stored as a “toy,” the electronic device (1000) may perform an operation to notify the user that the object (1607) stored as a “toy” and the baby (1613) are nearby. In addition, the electronic device (1000) may additionally perform an operation to analyze the behavior of the baby (1613) from a captured image and notify the user of information regarding the current status of the baby (1613).

FIG. 17 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.

Referring to FIG. 17, the electronic device (1000) can acquire gestures (1701, 1702) based on the direction of the user's gaze and fingers upon receiving the user's voice input (1703). The electronic device (1000) of FIG. 17 may be a mobile robot capable of collecting various information while moving inside a house.

An electronic device (1000) according to an embodiment can acquire an image that includes a surrounding environment related to a voice input (1703) according to a direction indicated by a user's gesture (1701, 1702). An electronic device (1000) according to an embodiment can capture an image that includes an area (1704) indicated by the user's gesture. An electronic device (1000) according to an embodiment can perform an operation according to a user's voice input (1703) based on the acquired image.

For example, the electronic device (1000) may obtain location information about an area (1704) corresponding to "this" and "here" among voice inputs (1703) as additional information based on an image including an area (1704) indicated by a gesture. Accordingly, the electronic device (1000) may generate request information to include location information corresponding to "this" and "here" based on the additional information, and may perform an operation based on the request information. The electronic device (1000) according to one embodiment may perform an operation on the area (1704) according to the request information.

In addition, the electronic device (1000) according to one embodiment can obtain an image captured to include a surrounding environment related to a voice input (1705) according to a direction indicated by a user's gesture (1707, 1708). The electronic device (1000) according to one embodiment can capture an image to include an area (1706) indicated by the user's gesture. In addition, the electronic device (1000) can obtain location information about an area (1706) corresponding to "this" among the voice inputs (1703) as additional information based on the image including the area (1706) indicated by the gesture. The electronic device (1000) according to one embodiment can perform an operation on the area (1706) according to request information corresponding to the voice input (1705) obtained based on the additional information.

FIG. 18 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.

Referring to FIG. 18, the electronic device (1000) can obtain gestures (1801, 1802) based on the direction of the user's gaze and fingers upon receiving the user's voice input (1803). The electronic device (1000) of FIG. 18 may be a mobile robot capable of collecting various information while moving inside a house. According to an embodiment, the electronic device (1000) can obtain gestures (1801, 1802) corresponding to the voice input (1803) by capturing an image including the gestures (1801, 1802) while moving inside a house.

An electronic device (1000) according to an embodiment may acquire an image captured to include a surrounding environment related to a voice input (1803) according to a direction indicated by a user's gesture (1801, 1802). An electronic device (1000) according to an embodiment may collect information about the surrounding environment based on the acquired image and according to the user's voice input (1803).

For example, the electronic device (1000) may obtain location information about an area corresponding to "this line" among voice inputs (1803) as additional information based on an image including an area (1804) indicated by gestures (1801, 1802). Accordingly, the electronic device (1000) may generate request information to include location information corresponding to "this line" based on the additional information, and perform an operation based on the request information. According to an embodiment, the electronic device (1000) may set a "line" (1804) in an area corresponding to the location information, and perform an operation according to the user's voice input (1804) based on the set line (1804). For example, in response to a voice input (1804) such as "Children can't cross this line (1804)," the electronic device (1000) can determine where children (1805, children) present near the line (1804) are present with respect to the line (1804), and based on the determination result, perform an action of informing the user or children (1805) of information about the area where the children are located with respect to the line (1804).

Additionally, the electronic device (1000) according to one embodiment can set a movement area of an external device (1808) that moves within a house according to a user's voice input. For example, the external device (1808) may be a robot vacuum cleaner that moves while cleaning the floor.

In one embodiment, when an electronic device (1000) receives a voice input such as "the external device (1808) cannot cross this line (1804)," the electronic device (1000) may perform an action according to the voice input based on an image captured including the line (1804). For example, the external device (1808) may move in the direction of an arrow in an area within the line (1804) based on a request received from the electronic device (1000).

FIG. 19 is a diagram showing an example of providing information requested by a user by processing voice input based on a gesture according to one embodiment.

Referring to FIG. 19, an electronic device (1000) may obtain a gesture (1902) based on the direction in which a user's finger is pointed upon receiving a user's voice input (1901). The electronic device (1000) of FIG. 19 may be a portable device, including a smartphone, that a user can hold and use. According to an embodiment, the electronic device (1000) may obtain a gesture (1902) corresponding to a voice input (1901) by capturing an image including the gesture (1902).

In one embodiment, the voice input (1901) may include user input requesting information about food stored in the refrigerator (1905), such as, for example, “What is this?”, “Is it fresh?”, “When I order this?”, “Order this?”, “What can be cooked from this?”, “Observe amount of this?”, “Set temperature to save this?”

An electronic device (1000) according to an embodiment of the present invention may acquire an image captured to include a surrounding environment related to a voice input (1901) according to a direction indicated by a user's gesture (1902). For example, the electronic device (1000) may acquire an image including an area 1904 among images captured by the electronic device (1000) according to the direction indicated by the gesture (1902) as an image of the surrounding environment.

An electronic device (1000) according to an embodiment may obtain identification information about "it" of a voice input (1901) as additional information based on an acquired image of the surrounding environment. An electronic device (1000) according to an embodiment may obtain identification information about "it" by using a pre-learned artificial intelligence model (e.g., CNN) to identify an object corresponding to "it" in the image of the surrounding environment.

An electronic device (1000) according to an embodiment may provide a user with information about "it", which is information requested by a voice input (1901), based on the acquired additional information. An electronic device (1000) according to an embodiment may provide the user with information requested by additionally acquiring information about a refrigerator (1905) (e.g., temperature setting information, stored food information) if necessary, based on the voice input (1901).

FIG. 20 is a diagram showing an example of providing information requested by a user by processing voice input based on a gesture according to one embodiment.

Referring to FIG. 20, the electronic device (1000) can acquire a gesture (2011) according to the user's touch input by receiving the user's voice input (2012). The electronic device (1000) of FIG. 20 may be a portable device including a smartphone that the user can hold and use.

According to one embodiment, an electronic device (1000) can acquire an image area including beef (2013) among images captured by the electronic device (1000) as an image of the surrounding environment based on a point touched by a user's gesture (2011).

An electronic device (1000) according to an embodiment may obtain identification information about "this" in a voice input (2012) as additional information based on an image of the surrounding environment. An electronic device (1000) according to an embodiment may obtain identification information about beef (2013) as identification information about "this" by using a pre-learned artificial intelligence model (e.g., CNN) to identify an object corresponding to "this" in the image of the surrounding environment.

An electronic device (1000) according to an embodiment may provide a user with information regarding "this," which is information requested by a voice input (2012), based on the acquired additional information. For example, based on a voice input (2012) including "What can we cook from this?", information regarding possible dishes using beef (2013) may be provided based on identification information regarding beef (2013).

Additionally, the electronic device (1000) according to one embodiment may, after providing response information for the voice input (2012), additionally receive a voice input (2022) such as “If add this?” Upon receiving the voice input (2022), the electronic device (1000) may acquire a user gesture of touching an area corresponding to the potato (2023).

According to one embodiment, an electronic device (1000) can acquire an image area including a potato (2023) among images captured by the electronic device (1000) as an image of the surrounding environment based on a point touched by a user's gesture.

An electronic device (1000) according to an embodiment may obtain identification information about "this" in a voice input (2022) as additional information based on an image of the surrounding environment. An electronic device (1000) according to an embodiment may obtain identification information about a potato (2023) as identification information about "this" by using a pre-learned artificial intelligence model (e.g., CNN) to identify an object corresponding to "this" in the image of the surrounding environment.

An electronic device (1000) according to an embodiment may provide a user with information regarding "this," which is information requested by a voice input (2022), based on the acquired additional information. For example, based on a voice input (2022) including "If add this?", in addition to beef (2013), information regarding possible dishes using beef (2013) and potatoes (2023) may be provided based on identification information regarding potatoes (2023).

In addition, the electronic device (1000) according to one embodiment may, after providing response information for the voice input (2022), additionally receive a voice input (2032) such as “The resulting dish is compatible with this.” Upon receiving the voice input (2032), the electronic device (1000) may acquire a user gesture of touching an area corresponding to the steamed rice (2033).

According to one embodiment, an electronic device (1000) can acquire an image area including air rice (2033) among images captured by the electronic device (1000) as an image of the surrounding environment based on a point touched by a user's gesture.

An electronic device (1000) according to an embodiment may obtain identification information about "this" of a voice input (2032) as additional information based on an image of the surrounding environment. An electronic device (1000) according to an embodiment may obtain identification information about rice (2033) as identification information about "this" by using a pre-learned artificial intelligence model (e.g., CNN) to identify an object corresponding to "this" in the image of the surrounding environment.

An electronic device (1000) according to an embodiment may provide the user with information regarding “this,” which is information requested by a voice input (2032), based on the acquired additional information. For example, based on a voice input (2032) including “The resulting dish is compatible with this?”, information may be provided regarding whether a dish made with beef (2013) and potatoes (2023) is compatible with plain rice (2032).

FIG. 21 is a diagram illustrating an example of providing information requested by a user by processing voice input based on a gesture according to one embodiment.

Referring to FIG. 21, the electronic device (1000) can obtain a gesture (2112, 2122) based on the direction in which the user's finger is pointed when receiving a user's voice input (2111, 2121). The electronic device (1000) of FIG. 21 may be a portable device including a smartphone that the user can hold and use. According to an embodiment, the electronic device (1000) can obtain a gesture (2112, 2122) corresponding to the voice input (2111, 2121) by capturing an image including the gesture (2112, 2122).

According to an embodiment, an electronic device (1000) may acquire an image captured to include a surrounding environment related to a voice input (2111, 2121) according to a direction indicated by a user's gesture (2112, 2122). For example, the electronic device (1000) may acquire an image including an area 2123 among images captured by the electronic device (1000) as an image of the surrounding environment according to the direction indicated by the gesture (2112, 2122).

An electronic device (1000) according to an embodiment may obtain identification information for "this" in a voice input (2111, 2121) as additional information based on an acquired surrounding environment image. An electronic device (1000) according to an embodiment may obtain identification information for "this" by using a pre-learned artificial intelligence model (e.g., CNN) to identify an object corresponding to "this" in the surrounding environment image.

An electronic device (1000) according to one embodiment can provide the user with information about “this,” which is information requested by voice input (1901), based on the acquired additional information.

For example, the electronic device (1000) may provide information about a component of a water pipe pointed to by a gesture (2112) in response to a voice input (2111) at 2110. Additionally, the electronic device (1000) may provide information about a component of an automobile engine pointed to by a gesture (2122) in response to a voice input (2121) at 2120.

FIG. 22 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.

Referring to FIG. 22, the electronic device (1000) can obtain a gesture (2212, 2222) based on the direction in which the user's finger is pointed, upon receiving a user's voice input (2211, 2221). The electronic device (1000) of FIG. 22 may be a portable device including a smartphone that the user can hold and use. According to an embodiment, the electronic device (1000) can obtain a gesture (2212, 2222) corresponding to the voice input (2211, 2221), respectively, by capturing an image including the gesture (2212, 2222).

According to an embodiment, an electronic device (1000) may acquire an image captured to include a surrounding environment related to a voice input (2211, 2221) according to a direction indicated by a user's gesture (2212, 2222). For example, the electronic device (1000) may acquire an image including a portion of an image captured by the electronic device (1000) as an image of the surrounding environment according to the direction indicated by the gesture (2212, 2222).

An electronic device (1000) according to an embodiment may obtain identification information for "this" in a voice input (2211, 2221) as additional information based on an acquired surrounding environment image. An electronic device (1000) according to an embodiment may obtain identification information for "this" by using a pre-learned artificial intelligence model (e.g., CNN) to identify an object corresponding to "this" in the surrounding environment image.

For example, in 2210 and 2220, the electronic device (1000) can obtain information about the result of recognizing the text indicated by the gesture (2212, 2222) as additional information.

An electronic device (1000) according to one embodiment can perform an action regarding “this” according to a voice input (2211, 2221) based on the acquired additional information.

For example, in 2210, the electronic device (1000) can recognize at least one text indicated by a gesture (2212) based on a voice input (2211) including “copy this text.” The electronic device (1000) can store the recognized at least one text in the electronic device (1000) and perform various operations using the stored text based on a user’s request.

Additionally, at 2220, the electronic device (1000) can recognize at least one text indicated by a gesture (2222) based on a voice input (2221) including “translate this.” The electronic device (1000) can provide a translation result for the at least one recognized text to the user.

FIG. 23 is a drawing showing an example of performing an operation of capturing an image by processing a voice input based on a gesture according to one embodiment.

Referring to FIG. 23, the electronic device (1000) can obtain a gesture (2313, 2322) based on the direction in which the user's finger is pointing when receiving a user's voice input (2311, 2321). The electronic device (1000) of FIG. 23 may be a device such as a smartphone or a digital camera that the user can hold in his hand and use to capture an image. According to an embodiment, the electronic device (1000) can obtain a gesture (2313, 2322) included in the captured image when capturing an image (2312) including a scene that the user wishes to capture.

According to an embodiment, an electronic device (1000) may acquire an image captured to include a surrounding environment related to a voice input (2311, 2321) according to a direction indicated by a user's gesture (2313, 2322). For example, the electronic device (1000) may acquire an image including a part of an area (2314, 2323) among images captured by the electronic device (1000) according to a direction indicated by a gesture (2313, 2322) as an image of the surrounding environment.

An electronic device (1000) according to an embodiment may obtain identification information for "here" of a voice input (2311, 2321) as additional information based on an acquired surrounding environment image. An electronic device (1000) according to an embodiment may obtain identification information for "here" by using a pre-learned artificial intelligence model (e.g., CNN) to identify an object corresponding to "here" in the surrounding environment image.

An electronic device (1000) according to one embodiment can perform an operation related to “here” according to a voice input (2311, 2321) based on the acquired additional information.

For example, in 2310, the electronic device (1000) may identify an object pointed to by a gesture (2313) based on a voice input (2311) including “Focus here.” The electronic device (1000) according to an embodiment may display an image (2312) including the object by adjusting the focus based on the area of the identified object in consideration of the characteristics of the identified object. For example, if the identified object is determined to be an object existing at a distance from the electronic device (1000), the image (2312) may be displayed by adjusting the focus so that the distant object can be optimally captured.

Additionally, in 2321, the electronic device (1000) can identify an object pointed to by a gesture (2323) based on a voice input (2321) including “More light here.” In one embodiment, the electronic device (1000) can display an image of a captured object by adjusting the brightness of an area including the object, taking into account the characteristics of the identified object. For example, the electronic device (1000) can display an image of a captured object by adjusting the brightness so that a portrait can be optimally captured, taking into account that the identified object is a person.

FIG. 24 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.

Referring to FIG. 24, the electronic device (1000) can acquire a gesture (not shown) according to the user's touch input by receiving the user's voice input (2401, 2402). The electronic device (1000) of FIG. 24 may be a driving robot that drives inside a factory and moves products.

According to an embodiment, an electronic device (1000) can acquire an image area including a product (2403, product) among images captured by the electronic device (1000) as an image of the surrounding environment based on a user's gesture acquired together with a voice input (2401, 2402).

For example, the electronic device (1000) can obtain voice inputs (2401, 2402) including “What ID is located there?” and “Transfer this product to there..”, respectively.

An electronic device (1000) according to an embodiment may obtain, as additional information, identification information for "there" and "this" of voice inputs (2401, 2402) based on a surrounding environment image. An electronic device (1000) according to an embodiment may obtain identification information about a product (2403) as identification information for "there" and "this" by using a pre-learned artificial intelligence model (e.g., CNN) to identify objects corresponding to "there" and "this" in the surrounding environment image.

An electronic device (1000) according to an embodiment may perform a requested operation according to a voice input (2401, 2402) based on the acquired additional information. For example, the electronic device (1000) may provide the user with information regarding the ID of the product (2403), or move the product (2403) to a location corresponding to “there.”

FIG. 25 is a drawing showing an example of performing an action by processing a voice input based on a gesture according to one embodiment.

Referring to FIG. 25, the electronic device (1000) can obtain a gesture (2502, 2504) based on the direction in which the user's finger is pointed, upon receiving the user's voice input (2501, 2503). The electronic device (1000) of FIG. 25 may be a vehicle (e.g., car, motorcycle, bicycle, etc.) on which the user is riding.

For example, the electronic device (1000) can receive a voice input (2501) including “Keep distance between this car, Keep this speed, What this?” and a voice input (2503) including “What is the place here? Turn this way.”

An electronic device (1000) according to one embodiment can obtain a voice input (2501, 2503) and corresponding gestures (2502, 2504) by taking a video including gestures (2502, 2504) using a camera device (2505) that takes a picture of a user.

An electronic device (1000) according to one embodiment can obtain an image including a surrounding environment related to a voice input (2501, 2503) according to a direction indicated by a user's gesture (2502, 2504) by using a camera device (2507) that photographs the surrounding environment.

An electronic device (1000) according to an embodiment may obtain identification information about "this car", "this", "here", and "this way" of voice inputs (2501, 2503) based on an acquired surrounding environment image as additional information. An electronic device (1000) according to an embodiment may obtain identification information about "this car", "this", "here", and "this way" by using a pre-learned artificial intelligence model (e.g., CNN) to identify objects corresponding to "this car", "this", "here", and "this way" in an surrounding environment image.

An electronic device (1000) according to one embodiment can perform an operation corresponding to a voice input (2501, 2503) based on the acquired additional information.

For example, the electronic device (1000) may identify a "car," a "speedometer reading," an "external object," etc., existing in the direction indicated by a gesture (2502), based on a voice input (2501), and, based on the identified result, adjust the driving speed of the car or provide information about the "external object." In addition, the electronic device (1000) may identify a "building," a "street," etc., existing in the direction indicated by a gesture (2504), based on a voice input (2503), and, based on the identified result, provide information about the "building" or adjust the driving direction of the car.

FIG. 26 is a diagram showing an example of providing information requested by a user by processing voice input based on a gesture according to one embodiment.

Referring to FIG. 26, the electronic device (1000) can obtain a gesture (2602, 2604) based on the direction in which the user's finger is pointed, upon receiving the user's voice input (2601, 2603). The electronic device (1000) of FIG. 26 may be a wearable glass, which is a device that the user can wear like glasses.

For example, the electronic device (1000) can receive a voice input (2601) including “What is the name of this street?” and a voice input (2603) including “Is it a hotel? What is this building number?”

An electronic device (1000) according to one embodiment can obtain gestures (2602, 2604) corresponding to voice inputs (2601, 2603) by taking a picture including gestures (2602, 2604) using a camera (not shown) provided in the electronic device (1000).

An electronic device (1000) according to one embodiment can obtain an image including a surrounding environment related to a voice input (2601, 2603) according to a direction indicated by a user's gesture (2602, 2604).

An electronic device (1000) according to an embodiment may obtain, as additional information, identification information about "this street", "this", and "this building" of voice inputs (2601, 2603) based on an acquired surrounding environment image. An electronic device (1000) according to an embodiment may obtain identification information about "this street", "this", and "this building" by using a pre-learned artificial intelligence model (e.g., CNN) to identify objects corresponding to "this street", "this", and "this building" in an surrounding environment image.

An electronic device (1000) according to one embodiment can provide the user with information requested in a voice input (2601, 2603) based on the acquired additional information.

For example, the electronic device (1000) may identify the name of a street in the direction indicated by a gesture (2602) based on a voice input (2601) and provide the identified result to the user. In addition, the electronic device (1000) may obtain information regarding the name of a hotel or the number of a building in the direction indicated by a gesture (2604) based on a voice input (2603) and provide the obtained information to the user.

FIG. 27 is a diagram showing an example of controlling an external device by processing voice input based on a gesture according to one embodiment.

Referring to FIG. 27, the electronic device (1000) can obtain a gesture (2702, 2704) based on the direction in which the user's finger is pointed, upon receiving the user's voice input (2701, 2703). The electronic device (1000) of FIG. 27 may be a wearable glass, which is a device that the user can wear like glasses.

For example, the electronic device (1000) may receive a voice input (2701) including “Switch on that lights” and a voice input (2703) including “Put virtual TV on this wall.”

An electronic device (1000) according to one embodiment can obtain gestures (2702, 2704) corresponding to voice inputs (2701, 2703) by taking a picture including gestures (2702, 2704) using a camera (not shown) provided in the electronic device (1000).

An electronic device (1000) according to one embodiment can obtain an image including a surrounding environment related to a voice input (2701, 2703) according to a direction indicated by a user's gesture (2702, 2704).

An electronic device (1000) according to an embodiment may obtain, as additional information, identification information for “that lights” and “this wall” of voice inputs (2701, 2703) based on an acquired surrounding environment image. An electronic device (1000) according to an embodiment may obtain identification information for “that lights” and “this wall” by using a pre-learned artificial intelligence model (e.g., CNN) to identify objects corresponding to “that lights” and “this wall” in the surrounding environment image.

An electronic device (1000) according to one embodiment can provide the user with information requested in a voice input (2701, 2703) based on the acquired additional information.

For example, the electronic device (1000) can identify at least one light in a direction indicated by a gesture (2702) based on a voice input (2701) and control the identified light. In addition, the electronic device (1000) can identify a wall in a direction indicated by a gesture (2704) based on a voice input (2703) and identify a beam projector capable of displaying a virtual TV on the identified wall. Accordingly, the electronic device (1000) can control the identified beam projector so as to display a virtual TV on the identified wall.

According to one embodiment, even if the information contained in the voice input is unclear, an action that matches the user's intention can be performed by clearly interpreting the voice input based on the user's gesture and the surrounding environment.

A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Additionally, in this specification, a “part” may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

Claims (15)

In an electronic device, a method for processing voice input based on gestures,
A step of obtaining a voice input and a gesture corresponding to the voice input;
A step of acquiring an image including the surrounding environment of the electronic device based on the direction indicated by the gesture;
A step of extracting additional information related to the voice input from the above image;
A step of obtaining request information regarding an operation requested to the electronic device by the voice input based on the above-mentioned additional information; and
A step of processing the voice input based on the request information,
The above additional information is:
At least one of identification information and location information regarding at least one object or area related to the voice input extracted from the image,
The above additional information is extracted by identifying at least one object or region related to the voice input in the image,
The at least one object or the area is identified in the image based on the direction pointed by the gesture and the ROI (Region Of Interest) area related to the voice input in the image.
method. delete In the first paragraph, the step of obtaining the request information comprises:
A step of obtaining at least one text corresponding to the above voice input;
A step of replacing at least one of the at least one text with another text based on the above additional information; and
A step of obtaining the requested information based on the at least one text including the replaced text,
The steps to replace the above text with another text are
A method further comprising the step of replacing at least one of said at least one text with another text based on said gesture. delete In the first paragraph, the step of processing the voice input based on the request information comprises:
A step of extracting text features from at least one text corresponding to the above request information;
A step of extracting image features based on an area of an object corresponding to at least one text included in the image;
A step of obtaining a combined feature based on the text feature and the image feature; and
A method comprising the step of processing the voice input based on the combined features. In the fifth paragraph, the step of obtaining the combined features comprises:
A step of determining weights for the text features and the image features according to the type of the gesture;
A step of applying weights corresponding to the text features and the image features, respectively; and
A method comprising a step of obtaining the combined feature based on the text feature and the image feature to which the weights are respectively applied. delete In an electronic device for processing voice input based on gestures,
A/V(Audio/Video) input;
An output unit that outputs the result of processing the above voice input; and
Controlling the above A/V input unit to obtain voice input and obtain a gesture corresponding to the voice input,
By controlling the A/V input unit, an image including the surrounding environment of the electronic device is obtained based on the direction indicated by the gesture,
From the above image, additional information related to the voice input is extracted,
Based on the above additional information, request information regarding an operation requested to the electronic device by the voice input is obtained,
At least one processor for processing the voice input based on the request information,
The above additional information is:
At least one of identification information and location information regarding at least one object or area related to the voice input extracted from the image,
The above additional information is extracted by identifying at least one object or region related to the voice input in the image,
An electronic device wherein at least one object or area is identified in the image based on a direction indicated by the gesture and a region of interest (ROI) associated with the voice input in the image. delete In the 8th paragraph, the at least one processor
Obtaining at least one text corresponding to the above voice input,
Based on the above additional information, replacing at least one of the at least one text with another text,
An electronic device, comprising: obtaining the request information based on the at least one text, including the replaced text; and further based on the gesture, replacing at least one of the at least one text with another text. delete In the 8th paragraph, the at least one processor
Extracting text features from at least one text corresponding to the above request information,
Extracting image features based on the area of an object corresponding to at least one text included in the image,
Based on the above text features and the above image features, a combined feature is obtained,
An electronic device that processes the voice input based on the combined features. In the 12th paragraph, the at least one processor
Determine the weights for the text features and the image features according to the type of the gesture,
Applying weights corresponding to the above text features and the above image features, respectively,
An electronic device that obtains the combined feature based on the text feature and the image feature to which the weights are respectively applied. delete A computer-readable recording medium having recorded thereon a program for implementing the method of any one of claims 1, 3, 5, and 6.