JP7178983B2 - Agent device, agent method and program - Google Patents

Description

The present invention relates to agent devices, agent methods, and programs.

In recent years, instead of manually inputting instructions to the device to be operated by the operator, the operator speaks and recognizes the instructions included in the speech, making it possible to easily perform input operations by voice. A technique for doing so is known (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2001-147134

Here, the operator may give a plurality of instructions by speaking. However, with the conventional technology, it is difficult to determine which instruction should be executed with priority when a plurality of instructions are given by utterance.

Aspects of the present invention have been made in consideration of such circumstances, and aim to provide an agent device, an agent method, and a program capable of executing a plurality of uttered instructions in an appropriate order. one of the purposes.

The agent device, agent method, and program according to the present invention employ the following configurations.
(1) An agent device according to one aspect of the present invention includes an acquisition unit that acquires data representing voice uttered by a user, and recognizes the content of the user's utterance based on the data acquired by the acquisition unit. a speech recognition unit for specifying an instruction included in the utterance content; an output control unit for outputting information responding to the instruction specified by the specifying unit to an information output device including a display unit; a determining unit that determines whether or not the specified multiple instructions can be executed simultaneously when the multiple instructions are specified by the specifying unit; priority for adding a priority to the plurality of instructions included in the utterance content based on relevance to the content displayed by the display unit when it is determined that execution is impossible and a priority addition unit, wherein, when the identification unit identifies a plurality of the instructions, the output control unit adds the instructions to the instructions in descending order of the priority added to the priority addition unit. The corresponding information is output to the information output device.

Aspect (2) is the agent device according to aspect (1) above, wherein the determination unit refers to list information of instructions that cannot be executed simultaneously, and determines whether the specified plurality of instructions are It determines whether or not they can be executed simultaneously.

Aspect (3) is the agent device according to aspect (1) or (2) above, wherein the output control unit is determined by the determination unit to be capable of simultaneously executing a plurality of instructions. In this case, the information output device is caused to output information responding to a plurality of the instructions.

(4) An agent device according to another aspect of the present invention includes an acquisition unit that acquires data representing a voice uttered by a user, and acquires content of the user's utterance based on the data acquired by the acquisition unit. a speech recognition unit for recognition, an identification unit for identifying an instruction included in the utterance content, an output control unit for outputting information responding to the instruction identified by the identification unit to an information output device including a display unit; a priority adding unit that adds priority to the plurality of instructions included in the utterance content based on relevance to the content displayed by the display unit, wherein the output control unit comprises: When a plurality of the instructions are specified by the specifying unit, the information output device is caused to output information corresponding to the instructions in order of the high priority added to the priority adding unit. .

Aspect (5) is the agent device according to any one of aspects (1) to (4) above, wherein the priority adding unit is configured to display the plurality of instructions included in the utterance content on the display unit. or if both are related to the content displayed by the display unit, the priority is added based on the order in which the plurality of instructions are spoken. .

Aspect (6) is the agent device according to any one of aspects (1) to (5) above, wherein the priority adding unit is configured to display the plurality of instructions included in the utterance content on the display unit. If it is not related to the content displayed by the display unit, or if both are related to the content displayed by the display unit, the priority is added based on the conjunction connecting the multiple instructions.

Aspect (7) is the agent device according to any one of aspects (1) to (6) above, wherein the priority adding unit is configured to display the plurality of instructions included in the utterance content on the display unit. If it is not related to the content displayed by the display unit, or if both are related to the content displayed by the display unit, the priority is added based on the words and phrases that indicate the order of the instructions included in the utterance content. It is something to do.

An aspect of (8) is the agent device according to any one of aspects (1) to (7) above, wherein the priority adding unit is configured to display the plurality of instructions included in the utterance content on the display unit. If it is not related to the content displayed by the display unit, or if both are related to the content displayed by the display unit, the priority is added based on the phrase indicating the timing of the instruction included in the utterance content. It is something to do.

(9) An agent method according to another aspect of the present invention is such that a computer acquires data indicating a voice uttered by a user, recognizes the content of the user's utterance based on the acquired data, specifying instructions included in the utterance content, causing an information output device including a display unit to output information responding to the specified instructions, and when a plurality of the instructions are specified, the specified instructions It is determined whether or not a plurality of instructions can be executed simultaneously, and if it is determined that a plurality of instructions cannot be executed simultaneously, the plurality of instructions included in the utterance content are A priority is added based on the relevance to the content displayed by the display unit, and when a plurality of the instructions are specified, the information corresponding to the instruction is displayed in order from the instruction with the highest priority added. is output to the information output device.

(10) A program according to another aspect of the present invention causes a computer to acquire data indicating a voice uttered by a user, recognizes the contents of the user's utterance based on the acquired data, specifying an instruction included in the content, outputting information responding to the specified instruction to an information output device including a display unit, and when a plurality of the specified instructions are specified, the specified instructions are simultaneously output. If it is determined that a plurality of instructions cannot be executed at the same time, the display is performed for the plurality of instructions included in the utterance content. Priorities are added based on the relevance to the content displayed by the unit, and when a plurality of the instructions are specified, the information corresponding to the instructions is displayed in order from the instructions with the highest priority added. This is to be output to an information output device.

According to aspects (1) to (10), a plurality of uttered instructions can be executed in an appropriate order.

According to aspect (2), instructions can be executed in a more appropriate order.

According to aspects (5) to (8), the order of instructions to be executed can be determined with higher accuracy.

It is a figure showing an example of composition of agent system 1 concerning an embodiment. 1 is a diagram showing an example of the configuration of an agent device 100 according to an embodiment; FIG. It is a figure which shows an example in the vehicle interior seen from the driver's seat. It is a figure which shows an example in the vehicle interior which looked at the vehicle M from above. 4 is a diagram showing an example of contents of list information 154. FIG. It is a figure which shows an example of a structure of the server apparatus 200 which concerns on embodiment. It is a figure which shows an example of the content of the reply information 232. FIG. FIG. 10 is a diagram showing an example of a scene in which priority is added to instructions; It is a figure which shows an example of the scene which makes an information output device output information based on the added priority. 4 is a flow chart showing a series of processes of the agent device 100 according to the embodiment; 4 is a flow chart showing an example of the flow of processing of the server device 200 according to the embodiment. FIG. 10 is a diagram showing an example of an agent device 100A according to a modified example;

Embodiments of an agent device, an agent method, and a program according to the present invention will be described below with reference to the drawings.

<Embodiment>
[System configuration]
FIG. 1 is a diagram showing an example of the configuration of an agent system 1 according to an embodiment. The agent system 1 according to the embodiment includes, for example, an agent device 100 mounted on a vehicle M and a server device 200 existing outside the vehicle M. FIG. The vehicle M is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle. The drive source of these vehicles may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electric power generated by a generator connected to the internal combustion engine, or electric power discharged from a secondary battery or a fuel cell.

Agent device 100 and server device 200 are communicably connected via network NW. The network NW includes a LAN (Local Area Network), a WAN (Wide Area Network), and the like. The network NW may include, for example, a network using wireless communication such as Wi-Fi and Bluetooth (registered trademark, hereinafter omitted).

The agent system 1 may be composed of multiple agent devices 100 and multiple server devices 200 . Hereinafter, a case where the agent system 1 includes one agent device 100 and one server device 200 will be described.

The agent device 100 uses the agent function to acquire voices from the occupants of the vehicle M, and transmits the acquired voices to the server device 200 . Also, the agent device 100 interacts with the occupant, provides information such as images and videos, and controls the in-vehicle equipment VE mounted on the vehicle M based on data (hereinafter referred to as agent data) obtained from the server device. or control other devices.

The server device 200 communicates with the agent device 100 mounted on the vehicle M and acquires various data from the agent device 100 . The server device 200 generates agent data suitable as a response to the occupants of the vehicle M based on the acquired data, and provides the generated agent data to the agent device 100 .

[Agent device configuration]
FIG. 2 is a diagram showing an example of the configuration of the agent device 100 according to the embodiment. The agent device 100 according to the embodiment includes, for example, a communication unit 102, a microphone 106, a speaker 108, a display unit 110, a control unit 120, and a storage unit 150. These devices and devices may be connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, and the like. The configuration of the agent device 100 shown in FIG. 2 is merely an example, and part of the configuration may be omitted, or another configuration may be added.

The communication unit 102 includes a communication interface such as a NIC (Network Interface controller). The communication unit 102 communicates with the server device 200 and the like via the network NW.

A microphone 106 is a voice input device that converts voice in the vehicle into an electric signal and picks up the voice. The microphone 106 outputs data of collected sound (hereinafter referred to as sound data) to the control unit 120 . For example, the microphone 106 is installed near the front when the passenger sits on the seat inside the vehicle. For example, the microphone 106 is placed near a mat lamp, steering wheel, instrument panel, or seat. A plurality of microphones 106 may be installed in the vehicle interior.

The speaker 108 is installed, for example, near the seat in the vehicle compartment or near the display unit 110 . Speaker 108 outputs sound based on the information output by control unit 120 .

Display unit 110 includes a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display. Display unit 110 displays an image based on information output from control unit 120 . A combination of the speaker 108 and the display unit 110 is an example of an "information output device."

FIG. 3 is a diagram showing an example of the interior of the vehicle viewed from the driver's seat. Microphones 106A to 106C, speakers 108A to 108C, and display units 110A to 110C are installed in the vehicle interior of the illustrated example. The microphone 106A is provided, for example, on the steering wheel, and mainly picks up the voice uttered by the driver. The microphone 106B is provided, for example, on the instrument panel (dashboard or garnish) IP in front of the front passenger seat, and mainly picks up the voices spoken by the occupant in the front passenger seat. The microphone 106C is installed, for example, near the center of the instrument panel (between the driver's seat and the passenger's seat).

The speaker 108A is installed, for example, under the door on the driver's seat side, the speaker 108B is installed, for example, under the door on the passenger seat side, and the speaker 108C is installed, for example, near the display unit 110C, that is, the instrument It is installed near the center of the panel IP.

The display unit 110A is, for example, a HUD (Head-Up Display) device that displays a virtual image ahead of the driver's line of sight when viewing the outside of the vehicle. The HUD device is, for example, a device that allows an occupant to visually recognize a virtual image by projecting light onto the front windshield of the vehicle M or a transparent member having light transmittance called a combiner. The occupant is mainly the driver, but may be an occupant other than the driver.

The display unit 110B is provided on the instrument panel IP near the front of the driver's seat (the seat closest to the steering wheel), and is installed at a position where the passenger can view it through the gap between the steering wheels or through the steering wheel. The display unit 110B is, for example, an LCD or an organic EL display device. The display unit 110B displays, for example, the speed of the vehicle M, the engine speed, the remaining amount of fuel, the radiator water temperature, the travel distance, and other information images.

The display unit 110C is installed near the center of the instrument panel IP. The display unit 110C is, for example, an LCD, an organic EL display device, or the like, like the display unit 110B. The display unit 110C displays contents such as TV programs and movies.

The vehicle M may be further provided with a microphone and a speaker near the rear seats. FIG. 4 is a diagram showing an example of the interior of the vehicle M viewed from above. In addition to the microphone speakers illustrated in FIG. 3, microphones 106D and 106E and speakers 108D and 108E may be installed in the vehicle interior.

The microphone 106D is provided, for example, in the vicinity of the rear seat ST3 installed behind the passenger seat ST2 (for example, the rear surface of the passenger seat ST2), and mainly picks up the voices spoken by the occupant seated on the rear seat ST3. do. The microphone 106E is provided, for example, in the vicinity of the rear seat ST4 installed behind the driver's seat ST1 (for example, behind the driver's seat ST1), and mainly picks up the voices spoken by the passengers seated in the rear seat ST4. do.

The speaker 108D is installed, for example, under the door on the rear seat ST3 side, and the speaker 108E is installed, for example, under the door on the rear seat ST4 side.

Although the vehicle M illustrated in FIG. 1 has been described as a vehicle having a steering wheel that can be operated by a driver who is a passenger, as illustrated in FIG. 3 or 4, the vehicle M is not limited to this. For example, the vehicle M may be a vehicle without a roof, ie without a passenger compartment (or without a clear division thereof).

Further, in the example of FIG. 3 or FIG. 4, it is assumed that the driver's seat where the driver who operates the vehicle M sits, and the passenger's seat and the rear seats where the other passengers who do not operate the vehicle M sit are in one room. but not limited to this. For example, the vehicle M may be a saddle type motorcycle having a steering handle instead of the steering wheel.

Further, in the example of FIG. 3 or 4, the vehicle M is described as being a vehicle having a steering wheel, but the vehicle M is not limited to this. For example, the vehicle M may be an automatically driven vehicle that is not provided with a driving operation device such as a steering wheel. An autonomously driven vehicle is, for example, one that controls one or both of steering and acceleration/deceleration of the vehicle to execute driving control without depending on the operation of the occupant.

Returning to the description of FIG. 2, the control unit 120 includes, for example, an acquisition unit 121, a speech synthesis unit 122, a communication control unit 123, an identification unit 124, a determination unit 125, a priority addition unit 126, and an output control unit. and a portion 127 . These components are realized by executing programs (software) by processors such as CPUs (Central Processing Units) and GPUs (Graphics Processing Units). Some or all of these components are implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array). It may be realized by cooperation of software and hardware. The program may be stored in advance in the storage unit 150 (a storage device having a non-transitory storage medium), or may be a removable storage medium such as a DVD or CD-ROM (non-transitory storage medium). , and may be installed in the storage unit 150 by loading the storage medium into the drive device.

The storage unit 150 is implemented by an HDD, flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), RAM (Random Access Memory), or the like. The storage unit 150 stores, for example, programs referred to by the processor, in-vehicle device information 152, and list information 154. FIG. The in-vehicle equipment information 152 is information indicating a list of in-vehicle equipment VE installed in the vehicle M. FIG.

FIG. 5 is a diagram showing an example of the contents of the list information 154. As shown in FIG. The list information 154 is, for example, information indicating instructions that cannot be executed simultaneously. This instruction is, for example, an instruction included in the voice uttered by the passenger, and is an instruction related to the operation of the vehicle-mounted device VE. In the list information 154 shown in FIG. 5, as instructions that cannot be executed simultaneously, "an instruction to the same controlled object", "an instruction to designate two destinations", and "an in-vehicle device VE1 that cannot be controlled simultaneously". and the on-vehicle device VE2”, and “an instruction to the on-vehicle device VE3 that cannot be controlled simultaneously and the on-vehicle device VE4”.

Returning to the description of FIG. 2 , the acquisition unit 121 acquires audio data and other information from the microphone 106 .

When the agent data received by the communication unit 102 from the server device 200 includes voice control content, the voice synthesis unit 122 generates artificial synthesized voice corresponding to the voice data instructed by an utterance as voice control. do. The artificial synthesized speech generated by the speech synthesizing unit 122 is hereinafter also referred to as agent speech.

The communication control unit 123 causes the communication unit 102 to transmit the voice data acquired by the acquisition unit 121 to the server device 200 . The communication control unit 123 causes the communication unit 102 to receive the agent data transmitted from the server device 200 .

The identifying unit 124 identifies instructions to the vehicle-mounted device VE included in the agent data. For example, in order to identify the on-vehicle device VE included in the agent data, the identifying unit 124 searches for the agent data using each on-vehicle device VE included in the on-vehicle device information 152 as a search key. The identifying unit 124 identifies one or a plurality of vehicle-mounted devices VE identified as being included in the agent data as the vehicle-mounted device VE to be instructed. The agent data is, for example, an example of "utterance contents".

When there are a plurality of vehicle-mounted devices VE to be instructed identified by the identifying unit 124, the determining unit 125 determines whether or not the plurality of instructions can be executed simultaneously based on the list information 154. judge.

Note that the determination unit 125 may determine whether a plurality of instructions can be executed simultaneously without using the list information 154 for instructions that cannot be executed simultaneously. Instructions that cannot be executed simultaneously are, for example, contradictory instructions to the same referent. Specifically, "Increase the volume of the audio (instruction A), lower the volume (instruction B)" or the like, or "Stop the vehicle M (instruction A), increase the speed of the vehicle M (instruction B)” or the like.

If the determination unit 125 determines that the multiple instructions included in the agent data cannot be executed simultaneously, the priority addition unit 126 assigns priority to the multiple instructions included in the agent data. Append. The details of the process of adding priority by the priority adding unit 126 will be described later.

The output control unit 127 causes the speaker 108 to output the agent voice when the voice synthesizing unit 122 generates the agent voice according to the instruction included in the agent data. In addition, the output control unit 127 causes the display unit 110 to display image data according to instructions included in the agent data. In addition, the output control unit 127 may cause the display unit 110 to display an image of the speech data recognition result (text data such as phrases).

Here, when a priority is added to each of the plurality of instructions by the priority adding section 126, the output control section 127 sequentially selects an instruction having a higher priority among the plurality of instructions according to the instruction. The voice of the agent received is output from the speaker 108, and the designated image data is displayed on the display unit 110. FIG.

[Configuration of server device]
FIG. 6 is a diagram showing an example of the configuration of the server device 200 according to the embodiment. The server device 200 according to the embodiment includes, for example, a communication unit 202, a control unit 210, and a storage unit 230.

The communication unit 202 includes a communication interface such as NIC. The communication unit 202 communicates with the agent device 100 or the like mounted on each vehicle M via the network NW.

The control unit 210 includes, for example, an acquisition unit 211 , an utterance segment extraction unit 212 , a voice recognition unit 213 , an agent data generation unit 214 and a communication control unit 215 . These components are implemented by executing a program (software) by a processor such as a CPU or GPU. Some or all of these components may be implemented by hardware (including circuitry) such as LSI, ASIC, and FPGA, or may be implemented by cooperation of software and hardware. good too. The program may be stored in advance in the storage unit 230 (a storage device having a non-transitory storage medium), or may be a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM. , and may be installed in the storage unit 230 by loading the storage medium into the drive device.

Storage unit 230 is implemented by an HDD, flash memory, EEPROM, ROM, RAM, or the like. The storage unit 230 stores, for example, a program referred to by the processor as well as answer information 232 and the like.

FIG. 7 is a diagram showing an example of the content of the reply information 232. As shown in FIG. In the answer information 232, for example, semantic information is associated with control contents to be executed by the control unit 120. FIG. The semantic information is, for example, the meaning recognized by the speech recognition unit 213 from the entire utterance content. The contents of control include, for example, vehicle-mounted device control related to instructions (control) to vehicle-mounted device VE, voice control for outputting agent voice, image control for display on display unit 110, and the like. For example, in the response information 232, for the semantic information of "starting up the air conditioner", on-vehicle device control of "starting up the air conditioner", voice control of "starting up the air conditioner", vehicle interior temperature, and set temperature are displayed. is associated with display control.

Returning to FIG. 6 , the acquisition unit 211 acquires voice data transmitted from the agent device 100 by the communication unit 202 .

The speech period extraction unit 212 extracts a period during which the passenger speaks (hereinafter referred to as a speech period) from the voice data acquired by the acquisition unit 121 . For example, the speech segment extraction unit 212 may use the zero-crossing method to extract the speech segment based on the amplitude of the audio signal included in the audio data. In addition, the utterance segment extraction unit 212 may extract utterance segments from the speech data based on a Gaussian mixture model (GMM), or may extract a speech segment from the speech data using a templated speech signal specific to the utterance segment. A speech segment may be extracted from the audio data by performing matching processing.

The speech recognition unit 213 recognizes speech data for each speech period extracted by the speech period extraction unit 212, and converts the recognized speech data into text, thereby generating text data including speech content. For example, the speech recognition unit 213 separates the speech signal of the speech period into a plurality of frequency bands such as low frequency and high frequency, and Fourier transforms each classified speech signal to generate a spectrogram. The speech recognition unit 213 obtains a character string from the spectrogram by inputting the generated spectrogram to a recursive neural network. A recursive neural network, for example, learns in advance by using teacher data in which known character strings corresponding to learning speech are associated as teacher labels for spectrograms generated from learning speech. It can be. Then, the speech recognition unit 213 outputs the character string data obtained from the recursive neural network as text data.

The speech recognition unit 213 also parses the natural language text data, divides the text data into morphemes, and recognizes the meaning of the sentences included in the text data from each morpheme.

The agent data generation unit 214 refers to the semantic information of the answer information 232 based on the meaning of the utterance content recognized by the speech recognition unit 213, and acquires the control content associated with the matching semantic information. Note that when meanings such as "turn on the air conditioner" and "turn on the power of the air conditioner" are recognized as a result of recognition, the agent data generation unit 214 converts the above meanings into the standard character information "start up the air conditioner". etc. As a result, it is possible to easily acquire the control content matching the instruction even when there is fluctuation in the utterance content of the instruction.

In addition, the agent data generation unit 214 generates agent data for executing processing corresponding to the acquired control content (for example, at least one of vehicle-mounted device control, voice control, and display control).

The communication control unit 215 causes the communication unit 202 to transmit the agent data generated by the agent data generation unit 214 to the agent device 100 . Thereby, the agent device 100 can execute control corresponding to the agent data by the control unit 120 .

[Regarding the process of adding priority]
Details of the priority adding process of the priority adding unit 126 will be described below with reference to FIGS. 8 and 9. FIG. FIG. 8 is a diagram showing an example of a scene in which priority is added to instructions. FIG. 9 is a diagram showing an example of a scene where the information output device is caused to output information based on the added priority.

In FIG. 8, the passenger utters "go to the zoo" as the utterance SP1 instructing to set the destination. Based on the agent data generated in response to the utterance SP1, the output control unit 127 causes the navigation device to identify the moving route to the "zoo", and the display unit 110 displays an image IM1 indicating the moving route identified by the navigation device. to display. Furthermore, the passenger utters "Go to the convenience store" as the utterance SP2 instructing to set the destination.

In this case, the specifying unit 124 recognizes that the utterance SP1 and the utterance SP2 are instructions to the navigation device, which is the in-vehicle device VE, and include a plurality of instructions for setting a destination. In this case, since there are a plurality of instructions for the in-vehicle device VE identified by the identification unit 124, the determination unit 125 determines whether or not these instructions can be executed simultaneously based on the list information 154. judge. As described above, the list information 154 includes "instructions for the same controlled object" as instructions that cannot be executed simultaneously. Therefore, the determination unit 125 determines that it is impossible to simultaneously execute a plurality of instructions to the vehicle-mounted device VE identified by the identification unit 124 . When determination unit 125 determines that multiple instructions to vehicle-mounted device VE cannot be executed simultaneously, priority addition unit 126 adds a priority to each of the multiple instructions.

[(1) Addition of priority based on content displayed on display]
The priority adding unit 126 adds a priority based on the relevance to the content displayed on the display unit 110, for example, when the passenger speaks (utterances SP1 to SP2). As described above, in the scene where the utterance SP1 is uttered, the display unit 110 displays the image IM1 showing the moving route to the "zoo". Therefore, between the instruction indicated by the utterance SP1 and the instruction indicated by the utterance SP2, the instruction indicated by the utterance SP1 is more relevant to the content of the display unit 110 (in this case, the route to "zoo"). Therefore, the priority adding unit 126 adds a higher priority to the instruction included in the utterance SP1, and a lower priority to the instruction included in the utterance SP2 than the instruction included in the utterance SP1.

For example, based on the agent data related to the utterance SP1 and the agent data related to the utterance SP2, the priority adding unit 126 combines the instructions included in the agent data related to the utterance SP1 and the instructions included in the agent data related to the utterance SP2. identify each. The priority addition unit 126 acquires information indicating the control history of the output control unit 127 or information related to the control state of the vehicle-mounted device VE (in this case, the navigation device), and identifies the content displayed on the display unit 110. do. Then, based on the specified various information, the priority adding unit 126 selects an instruction that is highly related to the content displayed on the display unit 110, out of the instruction related to the utterance SP1 and the instruction related to the utterance SP2. Identify and prioritize.

Based on the priorities added by the priority adding unit 126, the output control unit 127 executes control according to the instructions in descending order of priority. Based on the priority added by the priority adding unit 126, the output control unit 127 causes the information output device to output information corresponding to the instructions in descending order of priority.

In FIG. 9, the output control unit 127 causes the navigation device to identify the moving route to the "zoo" based on the agent data generated in response to the speech SP1, and the agent data generated in response to the speech SP2. , the "convenience store" existing on the way to the "zoo" is specified as a waypoint. Then, the output control unit 127 causes the display unit 110 to display an image IM2 showing the travel route to the "zoo" and showing the "convenience store" existing on the travel route to the "zoo" as a waypoint. The output control unit 127 causes the speaker 108 to output the agent voice SD1 related to the travel route to “zoo” generated by the voice synthesis unit 122, and then outputs the agent voice SD2 related to “convenience store” generated by the voice synthesis unit 122. is output from the speaker 108 . The agent voice SD1 is, for example, a voice such as "The destination is set to the zoo. I will guide you on the route." added to."

Here, as shown by the images IM1 and IM2, near the vehicle M, there is a convenience store CS1 located on the way to the zoo, and a convenience store CS1 which is the closest to the current position of the vehicle M, although it is off the route to the zoo. There is a convenience store CS2. When priority addition by the priority addition unit 126 is not performed, the output control unit 127 sequentially processes instructions according to the speech SP. Therefore, the output control unit 127 causes the navigation device to identify the travel route to the zoo based on the instruction indicated by the utterance SP1, and then causes the navigation device to cancel the travel route to the zoo based on the instruction indicated by the utterance SP2. , to identify the moving route to the nearest convenience store CS2. In this case, the vehicle M stops at the convenience store CS2, which is off the route to the zoo, and then heads for the zoo, resulting in a detour to the zoo.

On the other hand, when priority is added by the priority adding unit 126, the output control unit 127 processes instructions in descending order of priority. In this case, since the vehicle M stops at the convenience store CS1 on the route to the zoo while heading to the zoo, it is possible to move efficiently to the zoo. Therefore, according to the agent device 100 of the present embodiment, a plurality of uttered instructions can be executed in an order appropriate for the occupant of the vehicle M.

[(2) Addition of priority based on the order of utterances]
In the above description, the priority adding unit 126 adds priority based on, for example, the relationship with the content displayed on the display unit 110. However, the present invention is not limited to this. For example, the priority adding unit 126 may not associate any of a plurality of instructions included in the utterance SP with the content displayed on the display unit 110, If so, priority may be added based on the order of the speech SPs. In this case, the priority adding unit 126 gives a higher priority to the instruction included in the utterance SP1 that was uttered earlier, and gives a higher priority to the instruction included in the utterance SP2 that was uttered after the utterance SP1. give it a lower priority than the instructions given.

[(3) Addition of priority based on conjunction]
In addition, the priority adding unit 126 may, for example, set a plurality of instructions included in the utterance SP as not relating to the content displayed on the display unit 110, or , the priority may be added based on the conjunctions connecting the referent phrases. For example, ``(instruction A) and then ``then'' (instruction B)'' (example 1) or ``(instruction A) and ``the same'' (instruction B). The passenger's utterance SP such as example 2) includes conjunctions indicating the execution order of instructions, such as "then" and "similarly." In this case, the storage unit 150 stores conjunction information (not shown) in which information indicating the conjunction and information indicating the order (or priority) of the words before the conjunction and the words after the conjunction are associated with each other. ) are stored, and the priority adding unit 126 adds a priority to each of the plurality of instructions based on the conjunction information.

If the instruction A contains a conjunction indicating that the instruction A is to be executed before the instruction B, as in (Example 1) or (Example 2), the priority adding unit 126 assigns a higher priority to the instruction A. A lower priority than the instruction A is added to the instruction B.

[(4) Addition of priority based on words indicating order]
In addition, the priority adding unit 126 may, for example, set a plurality of instructions included in the utterance SP as not relating to the content displayed on the display unit 110, or priority may be added based on the order of instructions. For example, "do (instruction B) after" (instruction A). ) and the like include conjunctions indicating the execution order of instructions, such as "after" and "before". In this case, in the storage unit 150, information indicating the words and phrases indicating the order and information indicating the order of the words and phrases before the words and phrases indicating the order and the words and phrases after the words and phrases indicating the order are associated with each other. Information (not shown) is stored, and the priority adding unit 126 adds a priority to each of a plurality of instructions based on the order phrase information.

As in (Example 3), if the instruction A contains a phrase indicating that the instruction A should be executed before the instruction B, the priority adding unit 126 adds a higher priority to the instruction A, and gives the instruction B a higher priority. is assigned a lower priority than instruction A. Further, if the instruction B contains a phrase indicating that the instruction B is to be executed before the instruction A, as in (Example 4), the priority adding unit 126 adds a higher priority to the instruction B, Instruction A is given a lower priority than instruction B.

[(5) Addition of priority based on words indicating timing]
In addition, the priority adding unit 126 may, for example, set a plurality of instructions included in the utterance SP as not relating to the content displayed on the display unit 110, or , the priority may be added based on a phrase indicating the timing of the instruction. For example, in the passenger's utterance SP such as "Arrive at the elementary school at '17:00' (instruction A) and arrive at the swimming school at '18:00' (instruction B)" (Example 5), the timing for giving the instruction Contains phrases that indicate In this case, the priority adding unit 126 adds a priority to each of the multiple instructions based on the phrase indicating the timing of each instruction so that the instruction with earlier timing has a higher priority.

If the instruction A contains a phrase indicating that the instruction A is to be executed before the instruction B, as in (Example 5), the priority adding unit 126 adds a higher priority to the instruction A, and gives the instruction B a higher priority. is assigned a lower priority than instruction A.

[Processing flow]
Next, the flow of processing of the agent system 1 according to the embodiment will be explained using a flowchart. In the following description, processing by the agent device 100 and processing by the server device 200 will be described separately. Also, the flow of processing described below may be repeatedly executed at a predetermined timing. The predetermined timing is, for example, the timing at which a specific word (for example, a wake-up word) that activates the agent device is extracted from voice data, or the switch that activates the agent device 100 among various switches mounted on the vehicle M. is the timing at which the selection of is received.

FIG. 10 is a flow chart showing a series of processes of the agent device 100 according to the embodiment. First, the acquisition unit 121 determines whether or not voice data of the passenger has been collected by the microphone 106 (that is, whether or not the passenger has spoken) after the wakeup word is recognized (step S100). The acquisition unit 121 waits until the passenger's voice data is collected. Next, the communication control unit 123 causes the server device 200 to transmit the voice data to the communication unit 102 (step S102). Next, the communication control unit 123 causes the communication unit 102 to receive the agent data from the server device 200 (step S104).

The identification unit 124 identifies the instruction for the vehicle-mounted device VE included in the received agent data (step S106). The determination unit 125 determines whether or not an instruction to the vehicle-mounted device VE has been identified by the identification unit 124 (that is, whether or not the utterance content includes an instruction to the vehicle-mounted device VE) (step S108). If the specification unit 124 does not specify an instruction to the vehicle-mounted device VE, the determination unit 125 ends the process.

When determining that an instruction to the vehicle-mounted device VE is included, the determination unit 125 determines whether or not the agent data includes a plurality of instructions to the vehicle-mounted device VE (step S110). If the determination unit 125 determines that the agent data includes instructions to the vehicle-mounted device VE, but does not include a plurality of instructions (there is one instruction), the process proceeds to step S114. If the determining unit 125 determines that the agent data includes a plurality of instructions for the in-vehicle device VE, it determines whether or not the plurality of instructions can be executed simultaneously based on the list information 154. (Step S112). When determining unit 125 determines that a plurality of instructions to vehicle-mounted device VE can be executed simultaneously, the process proceeds to step S114.

The output control unit 127 causes the information output device to simultaneously output information corresponding to one or more instructions to the vehicle-mounted device VE identified by the identification unit 124 (step S114).
The output control unit 127 controls the in-vehicle device VE according to instructions included in the agent data, for example. Further, for example, when an agent voice is generated by the voice synthesizing unit 122 according to an instruction included in agent data, the output control unit 127 causes the speaker 108 to output the agent voice. In addition, the output control unit 127 causes the display unit 110 to display the instructed image data according to the instruction included in the agent data.

The fact that the output control unit 127 simultaneously outputs information corresponding to a plurality of instructions to the information output device means that, for example, the agent voice related to the agent data of each instruction is simultaneously output to one of the speakers 108A to 108C. It is to output or display an image related to the agent data of each instruction on one of the display units 110A to 110C. Note that the output control unit 127 sequentially outputs the agent voice related to the agent data of each instruction to one of the speakers 108A to 108C (for example, one), regardless of the priority, or The images related to the instructed agent data may be sequentially displayed on one of the display units 110A to 110C (for example, one) regardless of the priority.

When the judgment unit 125 judges that a plurality of instructions to the vehicle-mounted device VE cannot be executed at the same time, the priority addition unit 126 adds a priority to each of the plurality of instructions (step S116). For example, the priority adding unit 126 may add a priority to each of a plurality of instructions based on the relevance to the content displayed on the display unit 110, and may add the priority based on the order of the utterance SP. may be added, the priority may be added based on the conjunction that connects the phrases indicating the instruction, the priority may be added based on the phrase indicating the order of the instruction, and the priority may be added based on the phrase indicating the timing of the instruction Priority may be added based on phrases.

The output control unit 127 causes the information output device to output information corresponding to a plurality of instructions to the vehicle-mounted device VE identified by the identifying unit 124 in order of priority (step S118).

FIG. 11 is a flow chart showing an example of the processing flow of the server device 200 according to the embodiment. First, the communication unit 202 acquires voice data from the agent device 100 (step S200). Next, the speech segment extraction unit 212 extracts speech segments included in the voice data (step S202). Next, the speech recognition unit 213 recognizes the utterance contents from the speech data in the extracted utterance period. Specifically, the speech recognition unit 213 converts the speech data into text data, and finally recognizes the words included in the text data (step S204). The agent data generation unit 214 generates agent data based on the meaning of the entire speech content (step S206). Next, the communication control unit 215 transmits the agent data to the agent device 100 via the communication unit 202 (step S208).

[Summary of embodiment]
As described above, the agent system 1 of the present embodiment includes the acquisition unit 121 that acquires voice data representing the voice uttered by the user (a passenger in this example), and the voice data acquired by the acquisition unit 121. Based on this, a speech recognition unit 213 that recognizes the content of the utterance of the passenger, a specification unit 124 that specifies an instruction included in the content of the utterance, and information that responds to the instruction specified by the specification unit 124, information including the display unit 110. an output control unit 127 that causes an output device to output; a determination unit 125 that, when a plurality of instructions are identified by the identification unit 124, determines whether or not the plurality of identified instructions can be executed simultaneously; If the determining unit 125 determines that it is impossible to execute a plurality of instructions at the same time, the plurality of instructions included in the utterance content are determined based on the relevance to the content displayed by the display unit 110. and a priority adding unit 126 for adding priority to each priority, and when a plurality of instructions are specified by the specifying unit 124, the output control unit 127 selects from the instructions with high priority added to the priority adding unit 126. In order, the information output device is made to output the information corresponding to the instruction. As a result, the agent system 1 of the present embodiment can execute a plurality of uttered instructions in an order appropriate for the occupant of the vehicle M. FIG.

<Modification>
In the above-described embodiment, the agent device 100 mounted on the vehicle M and the server device 200 are different devices, but the present invention is not limited to this. For example, the constituent elements of the server device 200 related to the agent function may be included in the constituent elements of the agent device 100 . In this case, the server device 200 may function as a virtual machine that is virtually realized by the controller 120 of the agent device 100 . An agent device 100A including the components of the server device 200 will be described below as a modified example. In addition, in the modified example, the same components as in the above-described embodiment are denoted by the same reference numerals, and detailed description thereof is omitted here.

FIG. 12 is a diagram showing an example of an agent device 100A according to a modification. The agent device 100A includes, for example, a communication section 102, a microphone 106, a speaker 108, a display section 110, a control section 120a, and a storage section 150a. The control unit 120a includes, for example, an acquisition unit 121, a speech synthesis unit 122, a communication control unit 123, a specification unit 124, a determination unit 125, a priority addition unit 126, an output control unit 127, and a speech segment extraction unit. It includes a unit 212 , a speech recognition unit 213 and an agent data generation unit 214 .

Further, the storage unit 150a includes, for example, in-vehicle device information 152, list information 154, and answer information 232 in addition to programs referred to by the processor. The answer information 232 may be updated with the latest information obtained from the server device 200 .

The processing of the agent device 100A is, for example, after the processing of step S100 of the flowchart shown in FIG. 10, the processing of steps S202 to S206 of the flowchart shown in FIG. is a process for executing the process of

According to the agent device 100A of the modified example described above, in addition to the same effects as those of the first embodiment, it is not necessary to communicate with the server device 200 via the network NW each time the voice from the passenger is acquired. Therefore, the utterance content can be recognized more quickly. Further, even when the vehicle M cannot communicate with the server device 200, it is possible to generate agent data and provide information to the occupants.

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

Reference Signs List 1 agent system 100, 100A agent device 102, 202 communication unit 106, 106, 106A, 106B, 106C, 106D, 106E microphone 108, 108A, 108B, 108C, 108D, 108E speaker 110 , 110A, 110B, 110C... display unit 120, 120a, 210... control unit 121... acquisition unit 211... acquisition unit 122... speech synthesis unit 123... communication control unit 215... communication control unit 124... identification Unit 125 Determination unit 126 Priority addition unit 127 Output control unit 150, 150a, 230 Storage unit 152 In-vehicle device information 154 List information 200 Server device 212 Speech segment extraction Part 213... Voice recognition part 214... Agent data generation part 232... Answer information M... Vehicle SD1, SD2... Agent voice SP, SP1, SP2... Speech VE, VE1, VE2, VE3, VE4... In-vehicle machine

Claims (10)

an acquisition unit that acquires data indicating a voice uttered by a user;
a speech recognition unit that recognizes utterance content of the user based on the data acquired by the acquisition unit;
an identification unit that identifies an instruction included in the utterance content;
an output control unit that causes an information output device including a display unit to output information responding to the instruction specified by the specifying unit;
a determining unit that, when the specifying unit specifies a plurality of the instructions, determines whether the specified plurality of the instructions can be executed at the same time;
If the determination unit determines that the plurality of instructions cannot be executed at the same time, the relationship between the plurality of instructions included in the utterance content and the content displayed by the display unit a priority adding unit that adds priority based on the nature of the
When the specifying unit specifies a plurality of instructions, the output control unit outputs information corresponding to the instructions to the information output device in order of priority added to the priority adding unit. to output to
agent device. The determination unit refers to list information of instructions that cannot be executed simultaneously, and determines whether or not the specified plurality of instructions can be executed simultaneously.
The agent device according to claim 1. The output control unit causes the information output device to output information in response to the plurality of instructions when the determination unit determines that the plurality of instructions can be executed at the same time.
3. The agent device according to claim 1 or 2. an acquisition unit that acquires data indicating a voice uttered by a user;
a speech recognition unit that recognizes utterance content of the user based on the data acquired by the acquisition unit;
an identification unit that identifies an instruction included in the utterance content;
an output control unit that causes an information output device including a display unit to output information responding to the instruction specified by the specifying unit;
a priority adding unit that adds priority to the plurality of instructions included in the utterance content based on relevance to the content displayed by the display unit;
When the specifying unit specifies a plurality of instructions, the output control unit outputs information corresponding to the instructions to the information output device in order of priority added to the priority adding unit. to output to
agent device. If none of the plurality of instructions included in the speech content are related to the content displayed by the display unit, or all of the instructions are related to the content displayed by the display unit If so, adding the priority based on the order in which the multiple instructions were spoken;
An agent device according to any one of claims 1 to 4. If none of the plurality of instructions included in the speech content are related to the content displayed by the display unit, or all of the instructions are related to the content displayed by the display unit if so, adding said priority based on a conjunction connecting multiple said instructions;
An agent device according to any one of claims 1 to 5. If none of the plurality of instructions included in the speech content are related to the content displayed by the display unit, or all of the instructions are related to the content displayed by the display unit in the case, adding the priority based on a phrase indicating the order of the instructions included in the utterance content;
Agent device according to any one of claims 1 to 6. If none of the plurality of instructions included in the speech content are related to the content displayed by the display unit, or all of the instructions are related to the content displayed by the display unit in the case, adding the priority based on a phrase indicating the timing of the instruction included in the utterance content;
Agent device according to any one of claims 1 to 7. the computer
Acquire data indicating the voice uttered by the user,
recognizing the utterance content of the user based on the acquired data;
identifying an instruction included in the utterance content;
causing an information output device including a display to output information responding to the specified instruction;
if multiple instructions are identified, determining whether the identified instructions can be executed simultaneously;
when it is determined that the plurality of instructions cannot be executed simultaneously, based on the relevance of the plurality of instructions included in the utterance content to the content displayed by the display unit add priority,
when a plurality of the instructions are specified, outputting information corresponding to the instructions to an information output device in order from the added instructions with the highest priority;
agent method. to the computer,
Acquire data indicating the voice uttered by the user,
Recognizing the utterance content of the user based on the acquired data,
identify instructions included in the utterance content;
causing an information output device including a display to output information responding to the specified instruction;
if a plurality of said instructions are identified, determining whether said plurality of said identified instructions can be executed simultaneously;
when it is determined that the plurality of instructions cannot be executed at the same time, based on the relevance of the plurality of instructions included in the utterance content to the content displayed by the display unit add priority,
When a plurality of said instructions are specified, causing an information output device to output information corresponding to said instructions in order from said added instructions with higher priority;
program.

Images