JP2000231427A - Multi-modal information analyzing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize unification without any semantic inconsistency with a real objective world by interpreting each input modality based on the knowledge of the objective world of a user. SOLUTION: A recognized result such as the gesture of a user is inputted to a spatial base information under consideration detecting part 21, and what a user pays attention to is specified by using the knowledge of an objective world in an objective position storage part 23, and a recognized result such as an uttered voice is inputted to a context base information under consideration detecting part 22, and what the user pays attention to and how the user does is specified by using the knowledge of an objective world in an objective characteristics storing part 24, and they are unified under the consideration of the state of the objective world or the state of this system managed by a state managing part 26 in a consideration information unifying part 25. Thus, the information under consideration of the user can be detected.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an input device of a computer, and more particularly, to a method of integrating a direct instruction such as a gesture and a line of sight of a human and a voice of a human into an integrated system. The present invention relates to a multi-modal information analysis device that outputs an analysis result as attention information of a user as to what and how to pay attention.

[0002]

2. Description of the Related Art Humans point to an object of interest,
They pursue their eyes and speak the content they are paying attention to as voice, expressing their own intentions and attention information.
2. Description of the Related Art Conventionally, as a means for analyzing human attention information, there is a multimodal interface technology that integrally uses a plurality of input modalities such as a direct instruction by a human gesture and a content instruction by an uttered voice. In this multimodal interface, in order to recognize and integrate human gestures and uttered voices, a framework is generally used in which each recognition result is linked from information of each recognition time and output as a final analysis result.

As a general framework for handling a plurality of input modalities in a multimodal interface, there is a multimodal input analyzer disclosed in Japanese Patent Publication No. Hei 7-122879. In this multi-modal input analysis device, input data of each modality to which a set of an actual input start time and an input end time is added,
Input analysis is performed by applying a syntax analysis rule capable of describing a time constraint by using a data sequence arranged based on the input time.

A multi-modal interface using human gestures and uttered voices is disclosed in Japanese Patent Laid-Open No. 9-11 / 1991.
No. 4634. In this method, a recognition time is added to each recognition result of a gesture and an uttered voice, and the recognition result is linked based on the recognition time, and a final analysis result is output.

[0005]

However, in such a conventional multimodal interface, it is only necessary to manage the recognition time of each input modality and to associate each modality with the recognition time as a reference. There was a problem that it was not possible to say that what made sense by actually associating them was integrated. For example, in the multi-modal interface disclosed in the above-mentioned Japanese Patent Application Laid-Open No. Hei 9-114634, a semantic structure of each input modality analyzed from its recognition result is analyzed using a temporal reference. By integrating, the semantic structure as the final analysis result is output, but when analyzing the semantic structure of each modality, the user does not use the knowledge of the target world,
Only the semantic structure in terms of defining gestures and the semantic structure in terms of linguistic definition of speech are analyzed. For this reason, for example, when designating the output destination of a video playback image, simply pointing to a certain output destination screen and saying “play back” requires a user request because the screen itself does not have a playback function. It happens that the operation cannot be realized. This indicates that there is no contradiction in terms of time and the semantic structure of each input modality, but there is a semantic contradiction as an operation in the actual target world.

Further, since the recognition rate of various kinds of recognition units is generally not 100%, for example, those which do not actually exist at the stage of speech recognition are targeted, and consequently inconsistent interactive processing is performed as a whole. There is a problem that often occurs. In such a case, if there is a framework that provides feedback to the recognition system based on the current analysis results, narrow down the words to be recognized and the target areas to be recognized according to the current situation. Will be possible. For example, when the user is looking in the direction of a television, words such as TV and TV, sentences including those words, and words such as video if a video is connected to the television are sent to the voice recognition unit. The recognition rate can be stabilized by increasing the adoption rate of the included speech voice. Also, for example, when the age of a certain person is inquired to the system, but the person is not known, the image recognition unit is informed of the target world that the target is a person to inquire of the age. Utilizing this method, it is possible to stabilize the recognition by narrowing down the recognition targets in advance to reduce the possibility of recognition errors.

Further, in the conventional multimodal interface technology, emphasis is placed on how to associate a plurality of input modalities with a time lag, depending on the current situation of the world to which the user is directed. However, the weight of each input modality has not yet been considered. For this reason, although a certain modality may not be necessary depending on the situation, the information of the modality is used, and thus there is a problem that inconsistency may occur in a final analysis result. For example, consider a case in which the system returns a question as to what is to be realized while performing an interactive operation with the system. In this case, it is set so that the user can determine the target by directly instructing the target with his / her arm or the like, and it is also possible to determine the target by speaking a name unique to the target. However, considering the target selection and decision operation in a case where there is no question return, direct instruction of the target with the arm etc. allows the target selection, but not the determination,
Realizing the final determination of the target by uttering “this” or the like can provide a natural operation feeling, and integration that makes use of the meaning of each modality can be performed.

According to the present invention, when integrating a plurality of input modalities, an association between modalities based on a temporal reference or an association between modalities based on a temporal reference and a semantic structure of each input modality is provided. In the integration by, the user is not required to address the problem that even if there is no inconsistency in the semantic structure after each input modality, there is a semantic inconsistency based on the knowledge of the user's target world. It is an object of the present invention to obtain a multimodal information integration device that can realize integration that is semantically consistent with the actual target world by interpreting each input modality based on knowledge of the world.

Further, since the present invention cannot provide feedback to the recognition system based on the analysis result at the present time, it is possible to flexibly cope with the recognition system by narrowing down the recognition target according to each situation. For problems that cannot be achieved, build a flexible system according to each situation by applying appropriate feedback to each recognition unit based on the analysis results of each input modality and the integrated analysis results. It is an object of the present invention to provide a multi-modal information analysis device capable of performing the above. In addition, the present invention solves the problem that the weight of each input modality cannot be changed, and thus the unnecessary information is integrated and the interpretation is inconsistent. Multimodal information analysis that can more flexibly derive semantically consistent analysis results by appropriately changing the importance of each input modality according to the current state of each target and the state of system dialogue The aim is to obtain a device.

[0010]

To achieve the above object,
The multi-modal information analysis device according to the first aspect of the present invention provides a multi-modal information analysis apparatus that allows a user to perform spatial analysis based on an image recognition signal detected from a gesture or a line of sight of the user and spatial knowledge in a target world stored in a target position storage unit. A space-based attention information detection unit that detects information of interest in and outputs this as a space-based attention information signal;
Detects the information that the user is paying attention to based on the speech recognition signal detected from the user's uttered voice and the knowledge of the functions and characteristics of each target in the target world stored in the target characteristic storage unit And a context-based attention information detection unit that outputs this as a context-based attention information signal; and a current information of each object included in the object world managed by the space-based attention information signal, the context-based attention information signal, and the state management unit. An attention information integration unit that detects information on what the user is paying attention to and how to receive the information as an attention information integration signal, using the state and the state of interaction with the system as inputs, An integrated signal is input, and the image recognition signal and the voice recognition signal are output according to the content of the attention information integrated signal. For the image recognition unit and the speech recognition unit, a feedback generation unit that outputs a feedback signal is provided, and the interactive control unit receives the attention information integrated signal as an input, and if it can be actually realized, To execute the operation, and if it is not feasible, to return a response to the user, to manage the dialogue between the user and the system. The storage unit stores and manages target knowledge such as commands for each target.

The multi-modal information analysis device according to the second aspect of the present invention provides the multi-modal information analysis device according to claim 1, wherein the space-based attention information detection unit newly outputs the image recognition signal within a predetermined time interval set with respect to a time when the image recognition signal is input. If there is an input of an image recognition signal, the image recognition signal is output as a group of signals, and if no new image recognition signal is input within the predetermined time interval, the image recognition signal is output as it is. Providing a signal management unit, the image recognition signal from the image recognition signal management unit as an input, to the attention target identification unit, using spatial knowledge in the target world stored in the target position storage unit, It is designed to specify what the user is actually paying attention to.

A multimodal information analyzing apparatus according to a third aspect of the present invention provides a language for specifying the semantic content of a text by performing language processing and analysis using the speech recognition signal as an input to the context-based attention information detecting unit. An analysis unit is provided, and a linguistic analysis output from the linguistic analysis unit is used as an input, and knowledge of functions and characteristics of each object in a target world stored in the target characteristic storage unit is used, and the attention content specifying unit is used. In this case, what the user actually pays attention to is specified.

In the multimodal information analyzing apparatus according to a fourth aspect of the present invention, the attention information integration section includes a space-based attention information signal output from the space-based attention information detection section and the context-based attention information detection section. The context-based attention information signal output from the input as an input, and it is sufficient to analyze the respective attention information signals in an integrated manner with respect to the valid time section of each attention information signal, or one of them alone. A time-consistency verification unit for verifying whether or not the analysis should be performed, and a space-based attention information signal with a synchronization verification flag and a context-based attention information signal with a synchronization verification flag output from the time-consistency verification unit, In the consistency verification unit, the content of each synchronization verification flag and the current status managed by the state management unit are stored. Consider the subject of world configuration subject to state and conversational state of the system, one in which the user has so as to verify whether you are paying attention to what it against what the target world.

According to a fifth aspect of the present invention, in the multimodal information analyzing apparatus, the state management unit sequentially reads current states of a plurality of constituent objects included in the target world or constituting the target world. Add, edit, store,
A management target state management unit is provided so that the system state management unit sequentially reads, adds, edits, stores, and manages the contents of the dialogue between the user and the system.

Further, in the multimodal information analyzing apparatus according to the invention of claim 6, the feedback signal output by the feedback generating section is based on the accumulated data of the object position accumulating section and the object characteristic accumulating section which are knowledge of the object world. , Which are output as an image recognition feedback signal and a speech recognition feedback signal, respectively.

Further, in the multimodal information analyzing apparatus according to the invention of claim 7, the interactive control unit receives an attention information integration signal output from the attention information integration unit as an input.
Providing a feasibility verification unit that verifies whether or not the operation can be actually performed on the target world, and using a feasibility verification signal output from the feasibility verification unit as an input,
The operation content generation unit generates a command sequence for executing an operation on the user's target of interest in accordance with the target world knowledge stored in the target operation content storage unit. The content of the operation content signal output by the generation unit is interpreted, and the operation content signal is output to an actual target.

Further, the multimodal information analyzing apparatus according to the invention of claim 8 is provided with an attention information storage section for accumulating and managing the attention information integration signal output from the attention information integration section.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a multimodal information analysis device according to the present invention. In FIG. 1, the multimodal information analysis device includes an image input unit 11, a voice input unit 12, an image Recognition unit 13, voice recognition unit 14, time management unit 15, attention information detection unit 2 according to the present invention, and dialogue management unit 3
And Among these, the image input unit 11 has a function of inputting a user's gesture or line of sight from an image input device such as a camera and outputting the image as an image signal 101. The voice input unit 12 inputs a user's uttered voice from a voice input device such as a microphone, and outputs a voice signal 10.
2 is provided.

The image recognizing unit 13 uses the difference and change of the density and color of the image included in the image signal 101 output from the image input unit 11 to determine the user's gesture, gaze, position, and the like. It has a function to detect information. For example, when a user directly points a target in space with an arm or the like, the method of specifying the pointing position in space is to use widely used methods such as image recognition and pattern recognition. Realize. In addition, by using a difference method or a binarization method based on threshold selection, which is a general method of image recognition, a difference between an image of a space captured in advance and an image currently captured is calculated. Identify the position of the user and shoot the camera according to the position,
The body angle and the arm of the user are identified from the human body shape model prepared as a template in advance by adjusting the photographing angle and the like and using a template matching method generally used in pattern recognition processing. Then, for example, by using a stereo method, which is a general method of acquiring three-dimensional information, calculate the correspondence of the model identified between a plurality of cameras, acquire three-dimensional depth information and the like,
It is possible to specify which area or position in the space the user's arm is pointing to.

Further, the image recognizing unit 13 stores an information type identifier for indicating that the processing has been performed by the image recognizing unit 13, a detection start time and a detection end time, and a user-specified position in the detected space. For example, it is described in the information content as an area number in the space set in advance, and is output as the image recognition signal 103 to the space-based attention information detection unit 21 in the attention information detection unit 2. As a result, it is possible to detect which region or position in the space the user is paying attention to, and use it as an important information source for knowing what the user is actually paying attention to. It becomes possible.

FIG. 2 shows an example of a description format of the image recognition signal 103 output from the image recognition section 13. The description format for the image recognition signal includes an information type identifier for identifying whether or not the signal is an image recognition unit, information content indicating an image recognition result such as a spatial area number and a designated position coordinate, and a recognition start time. And time information indicating the recognition end time and the like. Also,
The information area is not limited to a single spatial area number, and may take a form including a plurality of pieces of information. The image recognition signal is a fixed-length or variable-length signal. In addition, the information type identifier, information content,
The arrangement of the time information and the time information is not limited to the form shown in FIG. 2, and other information such as a user identifier indicating which user the information is may be included in the image recognition signal.

On the other hand, the voice recognition unit 14 converts, for example, a user's uttered voice into a character code by using the state or change of the amplitude and frequency of the voice signal 102 output from the voice input unit 12. It has a function to do.
As a method of converting a user's uttered voice to a character code, a widely known continuous voice recognition technique is used, for example, a pronunciation model of a word included in a dictionary for recognition prepared in advance and an utterance model. The user's uttered voice is output as a character code by matching a voice input model or using a model that determines the probability that a certain word appears continuously.

The speech recognition unit 14 further includes an information type identifier for indicating that the speech recognition unit 14 has processed the speech,
The recognition start time and the recognition end time, and the character code converted from the uttered voice are described in the information content as a group of texts, for example, and the context-based attention information detection unit 22 in the attention information detection unit 2 is used as the speech recognition signal 104. Output to As a result, the content spoken by the user can be processed as characters, and can be used as an important information source for knowing what the user is paying attention to in the speech content.

FIG. 3 shows an example of a description format of the speech recognition signal 104 output from the speech recognition section 14. The description format for the speech recognition signal includes an information type identifier for identifying whether the signal is a speech recognition unit signal, information content indicating a recognition result such as a text obtained from the uttered speech, a recognition start time and a recognition end time. And time information indicating time. The speech recognition signal is a fixed-length or variable-length signal. Also,
The arrangement of the information type identifier, the information content, and the time information in the voice recognition signal is not limited to the form shown in FIG. 3, and other information such as a user identifier indicating which user the information belongs to may be used. A configuration included in the speech recognition signal may be used.

Next, the time management unit 15 integrally manages time information relating to the image recognizing unit 13 and the voice recognizing unit 14, and stores time information such as a recognition start time and a recognition end time in each of the recognizing units 14, 13. Is output to each of the recognition units 14 and 13. The handled time information is not limited to the absolute time information, and may include relative time information. The attention information detection unit 2 includes a space-based attention information detection unit 21 and a context-based attention information detection unit 22.
And a target position storage unit 23, a target characteristic storage unit 24, an attention information integration unit 25, a state management unit 26, and a feedback generation unit 27.

FIG. 4 shows a space-based attention information detection unit 2 which is a part of the configuration of the multimodal information integration apparatus shown in FIG.
1 shows a detailed configuration. Space-based attention information detection unit 21
Are the image recognition signal management unit 211 and the attention target identification unit 21
And the image recognition signal management unit 2
11 receives an image recognition signal 103 output from the image recognition unit 13 as an input, and
3 is set at a fixed time interval W with respect to the time T when the next image recognition signal is received, and when the next image recognition signal is input within the time interval W, these image recognition signals are set as a group of image recognition signals 2101. It has a function to output. of course,
If another image recognition signal is not input within the time interval W, the image recognition signal 103 is output as it is.

In general, when a user designates a target area or the like in a series of operations, the user does not always designate only a single area. Conventionally, when a plurality of regions are designated, the designated start time and designated end time of each region are separately managed, and each recognition signal is treated as a completely different signal. However, from the viewpoint of the interaction between the user and the system, when considering a case where a plurality of regions are designated, the designated start time and the designated end time of each region are not separately managed, but the designated time interval of each region is set. If it is within a certain time, it is assumed that a plurality of areas are continuously specified, and the time when the first area is specified is the instruction start time, finally no area is pointed, or a certain area is left pointing It is more effective to manage the time as an instruction end time, for example, when the instruction continues for a certain period of time or more.

For example, if the recognition start time of the first input image recognition signal is T1 and the end time of the last image recognition signal input within the time interval is E2, these are grouped into a single image recognition signal. It is treated as a signal and output as recognition start time = T1, recognition end time = E2. Further, it is assumed that the area numbers described in the respective image recognition signals are the area 1, the area 3 and the area 4, and if the input order is the area 4, the area 3 and the area 1, the information content is (area 4 , Area 3 and area 1), the output order is output in the form of specifying a plurality of areas in which the input order is stored.

For example, when an uttered voice such as "from here to here" is input, it is only necessary to know which region is being pointed and in which order plural regions are being pointed out. It is possible to derive an analysis result indicating that the region 1 to the region 5 are sequentially pointed from the region 5 to the region 1. In this way, even when a plurality of areas are specified, it is possible to derive the analysis result in an integrated manner even if, for example, the number of descriptive words included in the uttered speech differs from the number of indicated areas in space. . As described above, the image recognition signal management unit 211 sets a fixed time interval W with respect to the time when the image recognition signal 103 is input to the image recognition signal management unit 211 so as to be able to cope with a plurality of area designations. It has a function of setting, and if a new image recognition signal is input within the time interval, it is possible to handle a plurality of image recognition signals as a group of image recognition signals.

The attention target specifying unit 212 receives the image recognition signal 2101 output from the image recognition signal management unit 211 as input, and stores the contents of the image recognition signal 2101 and the target world stored in the target position storage unit 23. It has a function to specify what the user actually pays attention to using knowledge. Here, the target world knowledge is, for example, knowledge of a plurality of constituent objects included in the target world or constituting the target world, such as absolute positions and relative positions in space, names, functions, characteristics, and the like. .

Of these, the target position storage unit 23 includes:
Spatial positional information such as an absolute position and a relative position is accumulated. For example, assume that information indicating that the user's designated area number is 1 is obtained from the image recognition signal 2101. In this case, it is assumed that the target position storage unit 23 is inquired about what is present in the area number 1 and, for example, the target “television” existing in the area number 1 is detected. As a result, it is possible to estimate what the user is actually paying attention to in the target world, and to write "TV" in the target identifier of the target information content. Further, the attention target identification unit 212, after identifying what the user actually pays attention to, an information type identifier indicating that the information source is the image recognition unit 13, and
Describe the start time and end time at the time of recognition, the attention information content including the identifier of the object of interest, and output the space-based attention information signal 201 to the time consistency verification unit 251 in the attention information integration unit 25. I do.

FIG. 5 shows the space-based attention information signal 201 output from the space-based attention information detection unit 21 and the attention content identification unit 22 in the context-based attention information detection unit 22.
2 based context information signal 20
2 shows an example of each description format. Each attention information signal includes an information type identifier indicating which recognition unit the information source is, a recognition start time and a recognition end time in the recognition unit,
Attention information indicating what the user is actually paying attention to in the target world. In addition,
Each attention information signal is a fixed-length or variable-length signal. Further, the arrangement of the information type identifier, the recognition start time and the end time, and the content of the attention information in each attention information signal is not limited to FIG. The information may be included in the attention information signal.

FIG. 6 shows an example of a description format of the information of interest shown in the description format of FIG. The attention information content is a target to which the user is paying attention, a target identifier which is an identifier of a person or a target object, and an operation content identifier indicating what kind of operation the user is requesting, and , A request content identifier indicating what is being requested with the operation. For example, if the target of interest is person A and you want to inquire about the name of person A, (person A, inquiry,
Name). The content of the attention information is a fixed length or a variable length. In addition, the arrangement of the target identifier, the operation content identifier, and the request content identifier in the content of the attention information is not limited to FIG. Other information such as a person and a target identifier may be included in the attention information content.

The target position storage unit 23 has a function of sequentially reading, adding, editing, storing, and managing knowledge regarding absolute position information and relative position information of each target among the target world knowledge. FIG. 7 is a diagram illustrating an example of the target world knowledge stored in the target position storage unit 23. In this example, for example, a television is located in area 1 and
It can be understood that it is located on the table. In addition, it is possible to know information such as what object exists in the area 1 and where it is located in the object space. The knowledge of the target world described in the target position storage unit 23 is not limited to using knowledge created in advance. For example, the target space is photographed using an image input device such as a camera, and the components included in the space are detected by image recognition, and then the components are labeled, and the position obtained for each target is obtained. The knowledge described in the target position storage unit 23 can also be acquired by a method such as managing information and labeled targets as a set.

FIG. 8 shows a detailed configuration of the context-based attention information detection unit 22 which is a part of the configuration of the multimodal information integration device shown in FIG. The context-based attention information detection unit 22 includes a language analysis unit 221 and an attention content identification unit 222. Of these, the language analysis unit 2
The speech recognition unit 21 receives the speech recognition signal 104 output from the speech recognition unit 14 as an input, and performs various language processing such as morphological analysis and syntax analysis, thereby obtaining the speech recognition signal included in the speech recognition signal. From the text,
It has a function of specifying the meaning of the text by extracting each constituent word, specifying part of speech information of each word, analyzing syntax information, and the like. This language analysis unit 221
For example, the information type identifier indicating that the information source is the speech recognition unit 14, the recognition start time and end time, and information of the linguistic analysis contents, for example, the input text is separated for each word, reading for each word, It is output as a language analysis signal 2201 describing part of speech information and the like.

The attention content specifying unit 22 receives the linguistic analysis signal 2201 output from the linguistic analysis unit 21 as an input, and based on the knowledge of the target world stored in the target characteristic storage unit 24, the user can actually It has a function to specify what kind of thing you are paying attention to. For example, when a linguistic analysis result relating to the text “Kore Attachment” is input, information such as “This: demonstrative pronoun” and “attach: verb” is obtained from the linguistic analysis signal 2201.
In this case, an inquiry is made as to whether or not there is a target having the characteristic of “turn on” in the target characteristic storage unit 24. For example, it is determined that the target “television” having the characteristic of “turn on” is detected. I do. As a result, it is possible to estimate what the user is actually looking at in the target world, and as the information of interest (TV, power supply,
Attached).

Further, the attention content specifying section 222 specifies what content the user is actually paying attention to, and thereafter, an information type identifier indicating that the information source is the voice recognition section 14 and a recognition type identifier. The start time, the end time, and the attention information content describing the content of interest are described, and the attention information integration unit 25 outputs the context-based attention information signal 202.
To the later-described time consistency verification unit 251. Here, the target characteristic storage unit 24 has functions of sequentially reading, adding, editing, storing, and managing knowledge about the names, functions, characteristics, and the like of each target among the target world knowledge. FIG. 9 is a diagram illustrating an example of the target world knowledge stored in the target characteristic storage unit 24. In this example, for example, it is known that a video can be operated using various characteristic values such as playing, rewinding, stopping, and recording with respect to a characteristic of a tape. be able to. In addition, as the video output destination, information that an external output destination such as a television or a projector is prepared can be known.

The knowledge of the target world described in the target characteristic storage unit 24 is not limited to the use of previously created knowledge. For example, if it is assumed that a unique management tag that can be communicated by wire or wireless is attached to each object in a distribution warehouse or the like, the name or function of each object is determined from a wire or wireless signal transmitted from the management tag. , Characteristics, etc., and the knowledge in the target characteristic storage unit 24 can be automatically configured.

FIG. 10 shows a detailed configuration of the attention information integration unit 25 which is a part of the configuration of the multimodal information integration device shown in FIG. The attention information integration unit 25 includes a time consistency verification unit 251 and a knowledge consistency verification unit 252, of which the time consistency verification unit 251 is output from the space-based attention information detection unit 21. The space-based attention information signal 201 and the context-based attention information signal 202 output from the context-based attention information detection unit 22 are received as inputs, and which attention information signals should be combined and analyzed, or a certain attention. Verify whether it is better to analyze the information signal alone.

For example, the recognition start time T in the image recognition unit 13 described in the space-based attention information signal 201 is T
1. Assume that the recognition end time is E1 and the time frame set for waiting for another attention information signal is W1. Here, a time section between time A and time B is [A,
B], the time section including the time frame W1 is [T
1, E1 + W1]. The time frame referred to here is set by adding it to the recognition end time described in a certain information signal of interest, and eliminates a time lag due to a difference in the load of recognition processing in each recognition unit. ,
It plays a role in absorbing differences in characteristics between the recognition units and differences in processing time in the attention information detection unit. This time frame can be set freely according to each attention information signal.

The context-based attention information signal 2
02, the recognition start time in the speech recognition unit 14 described in 02 is T2, the recognition end time is E2, and the time frame is W2, the effective time section [T1, E1 + W1] and the valid time interval [T2, E2 + W2] of the context-based attention information signal 202.
If there is a time overlapping with each other, it is determined that it is necessary to combine and analyze the space-based attention information signal 201 and the context-based attention information signal 202, and the result of both attention information signals is determined. Are added to the space-based attention information signal 201 and the context-based attention information signal 202, for example, by indicating a synchronization verification flag, for example, (space = 1, context = 1) to indicate that In addition, as a space-based attention information signal 2501 with a synchronization verification flag, at the same time, a context-based attention information signal 250 with a synchronization verification flag
As No. 2, it has a function of outputting to the knowledge consistency verification unit 252. Conversely, if the valid times of both attention information signals do not overlap, a synchronization verification flag indicating that the analysis should be performed independently, for example, (space = 1, context =
0) or (space = 0, context = 1) may be added to each attention information signal and output.

FIG. 11 shows an example of a description format of the attention information signal with the synchronization verification flag output from the time consistency verification section 251 in the attention information integration section 25. The attention information signal with the synchronization verification flag is an information type identifier indicating from which attention information detection unit the signal is output,
It consists of information contents and a synchronization verification flag. In addition,
The attention information signal with the synchronization verification flag is a fixed-length or variable-length signal. The arrangement of the information type identifier, the information content, and the synchronization verification flag in the attention information signal with the synchronization verification flag is not limited to FIG. 11, and other information such as the user identifier and the reception time of each attention information signal may be synchronized. It may be configured to be included in the attention information signal with the verification flag.

Here, the synchronization verification flag means that when a plurality of input modalities are analyzed in an integrated manner, information between which input modalities and which modules are integrated from the viewpoint of temporal consistency. This flag indicates whether the analysis should be performed, or whether the analysis should be performed by a single input modality or module. For example, when the set of modules to be integrated is a space-based attention information detection unit and a context-based attention information detection unit, the space-based attention information is described as (space = 1, context = 1) as described above. In the case of only the detection unit, it is described as (space = 1, context = 0). Thereby, at the time of final analysis, it is possible to know whether the analysis is realized after integrating a plurality of modalities and modules, or whether analysis can be performed with a single modality or module.

The knowledge consistency verification unit 252 receives as input the space-based attention information signal 2501 with the synchronization verification flag and the context-based attention information signal 2502 with the synchronization verification flag output from the time consistency verification unit 251, and Taking into account the contents of the verification flag and the current state of the configuration object of the target world managed by the state management unit 26 and the state of dialogue with the system, the user can determine what to do with the target world. Has the function of specifying whether the user is paying attention to For example, when the synchronization verification flag is (space = 1, context = 1), it is necessary to integrate and analyze both the space-based attention information and the context-based attention information. Assume that the target identifier of the attention information content in 2501 is “projector”, and the attention information content in the context-based attention information signal 2502 with the synchronization verification flag is (video, playback, tape). In addition, televisions and projectors are prepared as video output destinations all over the target world, and it may not be clear to which of the video reproduction destinations to output.

However, it is possible to detect from the space-based attention information signal 2501 with the synchronization verification flag that output to the “projector” is required. This allows
It is possible to specify the user's attention information such as ((video, playback, tape) → projector) that plays back the video and outputs it to the projector. Further, the knowledge consistency verification unit 252 responds to a user's request to reproduce video and output the video to the projector.
An inquiry is made to the state management unit 26 as to whether the video power is on. Here, if the video is on,
(Video, power, turn on) can be omitted. If the power of the video is not turned on, the actual operation may be realized in the flow of (video, power, turn on), ((video, playback, tape) → projector). However, because we know that the video is on,
It is possible to satisfy the user's request only with the information ((video, playback, tape) → projector). As a result, unnecessary operations can be omitted, and the current state can be used to respond more efficiently to the user, thereby improving the operational feeling for the user. Becomes possible. In addition, the attention information integration unit 25 outputs, for example, a description in which the final output destination is added to the content of the attention information,
3 and outputs the same signal 204 as the attention information specifying signal 203 to the dialog management unit 3.

FIG. 12 shows a detailed configuration of the state management unit 26 of the multimodal information integration device shown in FIG. The state management unit 26 includes a target state management unit 261 and a system state management unit 262. Of these, the target state management unit 261 includes a plurality of components included in the target world or constituting the target world. It has functions such as sequential reading, addition, editing, storage, and management of the current state of the target. For example, if there is an object called person B, television, and video in a room, the state of what operation is performed by person B, whether the television is on, whether the video is in a playback state, and the like. , And their states can be sequentially read and used, stored, and managed.

On the other hand, the system state management unit 262 has functions of sequentially reading, adding, editing, storing, managing, and the like, the contents of the dialog between the user and the system. For example, in the dialogue with the system, the current system state is such that the system knows only the content of the request to turn on, and does not know what the object corresponding to actually turns on. , And their states can be sequentially read and used, stored, and managed. As a result, for example, when the user inquires "How old is it?" And the situation of the request is known but the target is uncertain, the answer is "Mr. A" in the uttered voice. Alternatively, it is possible to narrow down the target by directly instructing Mr. A with an arm or the like. Conversely, if it is not in an indeterminate state,
With direct instructions by arms etc., final refinement is not performed,
It is possible to make settings such that direct instructions are used only in the form accompanying the input of operation contents and request contents by uttered voice. As a result, it is possible to integrate the input modalities by weighting the input modalities according to the current state, and it is possible to realize an operation with a natural feeling for the user.

Further, the feedback generation section 27 outputs the attention information integrated signal 203 output from the attention information integration section 25.
Is received as an input, and the contents of the attention information integrated signal 203 are compared with the contents of the target position storage unit 23 and the target characteristic storage unit 24, which are knowledge of the target world, and the image recognition feedback signal 105 is output to the image recognition unit 13. And outputs the voice recognition feedback signal 106 to the voice recognition unit 14.

For example, it is assumed that the user directly designates an area with his / her arm or the like, and the object included in the area is a television. In this case, the content of the attention information recorded in the attention information integrated signal 203 is, for example, (TV, operation content unknown, request content unknown). From the contents of the attention information, for example, the speech recognition unit 14 is usually provided with a word including the word “TV” or a word including the word “TV”, a word such as turning on the power which is a characteristic related to the TV, and a sentence including those words. It is possible to output the speech recognition feedback signal 106 so as to make settings such as increasing the adoption rate at the time of recognition than at the time of recognition. Also, if the video is connected to a TV, the adoption rate of the word video, sentences containing the word video, words such as playing a tape that is a characteristic of video, and sentences containing those words, etc. will be increased. Settings can be made to stabilize recognition.

In addition, for example, when the user makes an inquiry using an uttered voice of "How old are you?", For example, (object unknown, inquiry, age) or the like is written as the attention information content. From the contents of the attention information, the target characteristic storage unit 24 that a person is a promising candidate for inquiring the age to the image recognition unit 13, for example.
By outputting the image recognition feedback signal 105 for narrowing down the recognition target or the recognition area from the target candidates in the target position storage unit 23 in advance by using the knowledge in the above, the possibility of the recognition error is reduced and the recognition is stabilized. It is also possible to do. As a result, it is possible to apply feedback according to the scene to the recognition unit, and it is possible to more smoothly and stably interact with the user and the system. The dialog management unit 3 includes a dialog control unit 31 and a dialog operation content storage unit 32.

FIG. 13 shows a detailed configuration of the dialogue control unit 31 in the dialogue management unit 3. The dialog control unit 31 includes a feasibility verification unit 311, an operation content generation unit 312, an operation content execution unit 313, and a response generation unit 314.
Among these, the feasibility verification unit 311 receives the attention information integration signal 204 output from the attention information integration unit 25 as an input, and determines whether it is possible to actually perform an operation on the target world. Has a function to verify.

For example, when the content of the attention information described in the attention information integrated signal 204 is described as (TV, power supply, turn on), etc., the system is set to the attribute of adding the characteristic of the power supply of the target TV. You can understand that it should be done. Therefore, the feasibility verification unit 3
Numeral 11 denotes a feasibility identifier for indicating whether or not operation of the system can be realized. In this case, an identifier indicating the feasibility is added to the attention information integrated signal 204 to form a feasibility verification signal 3101. The description is sent to the operation content generation unit 312 and at the same time, the feasibility verification signal 3
A signal 3102 identical to 101 is output to the response generation unit 314.

Further, for example, if (Person A, operation content unknown, request content unknown) is described, it means that the operation is related to person A, but the system can understand. The system does not understand what kind of operation is performed. That is, in this case, the operation cannot be realized as a system. At this time, the feasibility verification unit 311 adds a feasibility identifier indicating that the system operation is not feasible to the attention information integration signal 204 to the response generation unit 31 as a feasibility verification signal 3102.
4 is output. In this case, the feasibility verification signal 310
Although it is possible to output 1 to the operation content generation unit 312, it is not necessary to output it.

Next, the operation content generation unit 312 is a feasibility verification signal 3101 which is an attention information integrated signal after verifying the feasibility output from the feasibility verification unit 311.
Is received as input, and the user performs an operation on the target of interest in accordance with the content of the target information in the feasibility verification signal 3101 and the content of the target world knowledge stored in the target operation content storage unit 32. It has a function of generating a command string and the like. For example, the feasibility verification signal 3101 is
(Video, playback, tape), the video that is actually operated
By reading from the target operation content storage unit 32 a command string or the like indicating the operation content necessary for reproducing the tape,
It is possible to generate a command sequence required for an actual operation. Then, the generated command sequence is transmitted to the operation content signal 310.
As 3, it has a function of outputting to the operation content execution unit 313.

The operation content execution unit 313 interprets the content of the operation content signal 3103 output from the operation content generation unit 312 and directly outputs the operation content signal to an actual object.
In addition, it has a function of outputting images, sounds, and the like via a system instead of directly outputting to an actual target. For example, if the operation content signal 3103 describes a command sequence necessary to realize an operation of inquiring the target person A about the age, “execution (person A, inquiry, age)”, this command sequence Is executed, the output destination required to execute the operation is specified, and the operation execution signal is output to the system side, for example, to fulfill the role of responding to the user's request. Thereby, it is possible to realize an operation in the actual target world according to the attention information of the user.

The transmission destination of the operation execution signal is the actual target device itself in the case of a television or the like, or a computer constituting a system in the case of inquiring about the age of a person. Is not limited. Also, when transmitted to the actual target device itself, images and sounds are output to the target device itself and connected output destinations, etc., according to the characteristics of the device, and are sent to computers included in the system. When transmitted, images such as computer graphics and video images, various sound effects and synthesized sounds are output from the computer itself, or output from other devices via the computer.

The response generation unit 314 includes the feasibility verification unit 3
11, a feasibility verification signal 3102 is received as an input, and according to the information as to whether the system can realize the operation, and what the operation content of the object is,
It has a control function for outputting a response sentence, a response image, and the like to the user. For example, suppose that as the contents of the feasibility verification signal 3102, information that the system cannot perform the operation and incomplete attention information contents (person A, operation contents unknown, request contents unknown) are obtained. In this case, ask the user "What do you want to do with A?"
A response sentence such as “Operation cannot be performed on the user” is generated and output using an audio output device such as a speaker. The output destination of the response generation unit 314 is not particularly limited and is included in the system. Various images, sound effects, and synthetic sounds are output via a computer display, a sound output device, or the like, or output using a screen, a speaker, or the like.

The target operation content storage unit 32 sequentially reads, adds, edits, stores, manages, etc., knowledge of a series of operation contents, command strings, etc. necessary for realizing the operation of each target in the target world knowledge. It has the function of FIG.
4 shows an example of the target world knowledge stored in the target operation content storage unit 32. In this example, for example, to play a videotape, first execute the command “execute (video, power, on)” to turn on the video, and then specify the command in the dialogue with the system. To turn on the power of the output destination such as a TV or projector that can
Execute the command “execute (output destination, power supply, on)”. Then, in order to set the output of the video to an output destination such as a television and reproduce the tape, a series of commands of “connection (video, output destination)” and “execute (video, tape, reproduction)” are executed. Know that you need it. Note that the knowledge of the target world described in the target operation content storage unit 32 is not limited to the one created in advance. For example, when a new operation content is added or a function is updated by adding a driver file or the like, the target operation content storage unit 3 is connected to the Internet or the like.
It is also conceivable that the knowledge in 2 is automatically updated.

In the first embodiment, an image signal input from an image input device such as a camera, and an audio signal input from an audio input device such as a microphone, in order to detect user's attention information. However, the information used to detect the user's attention information is not limited to image information and audio information. For example, as means for detecting the user's spatial attention information, information input from a pointing device such as a mouse or a laser pointer may be used to detect the user's attention information, in addition to the user directly giving instructions using the arm. May be used as information for the purpose.

FIG. 15 is a block diagram showing a multimodal information analyzer according to another embodiment of the present invention. This embodiment has the same function and configuration as the multi-modal information analysis apparatus shown in FIG. 1, but the image recognition signal management section 211 in the space-based attention information detection section 21 shown in FIG. Separated from the identification unit 212,
The difference is that the time consistency verification unit 251 and the knowledge consistency verification unit 252 in the attention information integration unit 25 are provided separately.

The image recognition signal management unit 41 in the attention information detection unit 4 accepts the image recognition signal 103 output from the image recognition unit 13 as an input, and may output only a single image recognition signal. Alternatively, it has a function of managing whether to output the image recognition signal in which the attention information content and the time information are re-described as a group of a plurality of image recognition signals. For example, assume that the first image recognition signal is input at a certain time T. The information content of the input image recognition signal is (region 3), and the time information is [T1, E
1]. At this time, a time interval W is set for a certain time T, and the next image recognition signal is transmitted in a time interval [T, T
+ W], it is described as a new image recognition signal as a unit with the first image recognition signal. At this time, it is assumed that the information content of the next input image recognition signal is (area 2) and the time information is [T2, E2].
In this case, the information content of the newly generated image recognition signal is stored in the form of preserving the input order (region 3, region 2).
And the time information is [T1, E2]. Time interval W
Is reset each time there is a new input, and the process ends if there is no input within the time interval.

If the image recognition signal management section 41 includes a plurality of information contents, the image recognition signal management section 41 rewrites the information contents, for example, in a state where the order of the area numbers is preserved. Are re-described as a new recognition start time and a new recognition end time corresponding to
01 is output to the time consistency verification unit 42. The time consistency verification unit 42 receives the image recognition signal 401 output from the image recognition signal management unit 41 and the voice recognition signal 104 output from the voice recognition unit 14 as inputs, and from the viewpoint of time of both recognition signals. Is verified, and whether it is sufficient to analyze the recognition signals in an integrated manner or whether it is better to analyze a certain recognition signal alone is verified. For example, the recognition start time in the image recognition unit 13 described in the image recognition signal 401 is T11, the recognition end time is E11, and the time frame set to wait for a recognition signal from another modality is W11. And

Here, the time section including the time frame W11 can be expressed as [T11, E11 + W11].
The time frame referred to here is set in a form to be added to the recognition end time described in a certain recognition signal, and eliminates a time lag due to a difference in a load of recognition processing in each recognition unit, It has the role of absorbing differences in the characteristics of each recognition unit. This time frame can be set freely according to each recognition signal.

If the recognition start time of the speech recognition unit 14 described in the speech recognition signal 104 is T22, the recognition end time is E22, and the time frame is W22,
Effective time section of image recognition signal 401 [T11, E11 +
W11] and the valid time section [T2
2, E22 + W22], the image recognition signal 401 and the speech recognition signal 104
And a synchronization verification flag for indicating that the results of both recognition signals should be analyzed in an integrated manner from a time point of view,
For example, (image = 1, sound = 1) is added to the image recognition signal 401 and the sound recognition signal 104, and the image recognition signal 402 with the synchronization verification flag is sent to the attention target specifying unit 43.
At the same time, it has a function of outputting to the context-based attention information detection unit 44 as a speech recognition signal 403 with a synchronization verification flag. Conversely, if the valid times of both recognition signals do not overlap, a synchronization verification flag indicating that analysis should be performed independently, for example (image = 1, audio = 0),
Alternatively, (image = 0, sound = 1) may be added to each recognition signal and output.

As described above, the provision of the time consistency verification unit 42 allows the attention target identification unit 43 and the context-based attention information detection unit 44 to be unaffected by differences in processing time and message transmission time. It is possible to verify temporal consistency at the time of recognition. That is, it is possible to verify the temporal consistency at a point closer to the user's input than in the case of the first embodiment. It is possible to realize the consistency verification based on the knowledge in.

FIG. 16 is a block diagram showing a multi-modal information analyzing apparatus according to still another embodiment of the present invention. In this embodiment, an attention information integration signal output from attention information integration section 25 is used. It differs from the embodiment shown in FIG. 1 in that it has an attention information storage unit 5 having a function of storing and managing. The attention information storage unit 5 sequentially reads the attention information integrated signal output from the attention information integration unit 25,
It has functions such as addition, editing, accumulation, and management. For example, it manages the accumulated attention information integrated signal in chronological order in a certain time frame, and can observe attention time and attention frequency for each attention target. By managing the information, it is possible to sequentially read out any attention information integrated signal by designating time, attention target, or the like.

As a result, it is possible to grasp the transition of the target of interest and the contents of the attention of the user, and it is possible to grasp the objects and contents of high frequency of attention. And more efficiently and stably detect the attention information by referring to the past attention information, and realize a smoother operation by constructing a natural operation environment more in line with the user's sense It is possible to do. Further, in the multi-modal information integration device shown in FIG. 1, there may be a form in which a portion for accumulating and managing each attention information signal is added to the space-based attention information detection unit 21 and the context-based attention information detection unit 22.

Also, in each of the above-described embodiments, as in the case of the embodiment shown in FIG. 1, the use of only image information and audio information to detect the user's attention information is limited. is not. For example, as means for detecting the user's spatial attention information, in addition to the user's direct instruction using the arm, information input from a pointing device such as a mouse or a laser pointer is used to detect the user's attention information. May be used as information.

[0069]

As described above, according to the present invention, based on the image recognition signal and the speech recognition signal, after verifying the temporal consistency between the input modalities, the image recognition with the synchronization verification flag is performed. A signal and a speech recognition signal with a synchronization verification flag are output, and the space-based attention information detection unit uses the knowledge of the target world to look at what the user spatially focuses on from the image recognition signal with the synchronization verification flag. The context-based attention information detection unit detects how the user speaks from the speech recognition signal with the synchronization verification flag using the knowledge of the target world. Is detected as a context-based attention information signal, and the space-based attention information signal and the context-based attention information are output. Degree from and, in the attention information integration unit, using the current state as well as knowledge of the object world,
Since it is configured to detect what the user actually pays attention to and how to output this as the attention information integrated signal, it is not simply to integrate because the input time of the input modality is near. Instead, it is possible to realize a multimodal information integration device that integrates to match the actual target world, and as a result, based on the gestures of direct instructions by the human arm and the utterance voice, the human complexities The effect is obtained that it is possible to analyze and judge a plurality of pieces of input information in an integrated manner.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a multimodal information analysis device according to an embodiment of the present invention.

FIG. 2 is a format diagram showing a description format of an image recognition signal output from an image recognition unit in FIG. 1;

FIG. 3 is a format diagram showing a description format of a speech recognition signal output from a speech recognition unit in FIG. 1;

FIG. 4 is a block diagram illustrating details of a space-based attention information detection unit in FIG. 1;

FIG. 5 is a format diagram showing a description format of each attention information signal output from a space-based attention information detection unit and a context-based attention information detection unit in FIG. 1;

FIG. 6 is a format diagram showing a specific example of a description format of attention information content in the description format of FIG. 5;

FIG. 7 is an explanatory diagram showing a specific example of description contents of target world knowledge stored in a target position storage unit in FIG. 1;

FIG. 8 is a block diagram illustrating details of a context-based attention information detection unit in FIG. 1;

FIG. 9 is an explanatory diagram showing a specific example of description contents of target world knowledge stored in a target characteristic storage unit in FIG. 1;

FIG. 10 is a block diagram showing details of an attention information integrating unit in FIG. 1;

11 is a format diagram illustrating a description format of each information signal of interest with a synchronization verification flag output from the time consistency verification unit in FIG. 1;

FIG. 12 is a block diagram illustrating details of a state management unit in FIG. 1;

FIG. 13 is a block diagram showing details of a dialog control unit in the dialog management unit in FIG. 1;

14 is an explanatory diagram showing a specific example of description contents of target world knowledge stored in a target operation content storage unit in FIG. 1;

FIG. 15 is a block diagram showing a multimodal information integration device according to another embodiment of the present invention.

FIG. 16 is a block diagram showing a multimodal information integration device according to another embodiment of the present invention.

[Explanation of symbols]

 Reference Signs List 21 space-based attention information detection unit 211 image recognition signal management unit 212 attention target identification unit 22 context-based attention information detection unit 221 language analysis unit 222 attention content identification unit 23 target position storage unit 24 target characteristic storage unit 25 attention information integration unit 251 Time consistency verification unit 252 Knowledge consistency verification unit 26 State management unit 261 Target state management unit 262 System state management unit 27 Feedback generation unit 31 Dialogue control unit 32 Target operation content storage unit 311 Feasibility verification unit 312 Operation content generation unit 313 Operation content execution unit 314 Response generation unit

Claims (8)

[Claims] 1. A multi-modal analysis apparatus that receives a plurality of modalities such as gestures, eyes, facial expressions, and voices used in normal communication by a human and outputs an analysis result by integrating the input modalities. Based on image recognition signals detected from gestures and eyes of the user and spatial knowledge in the target world accumulated by the target position storage unit, information that the user is paying attention to in space is detected and A space-based attention information detection unit that outputs a notice information signal, and a speech recognition signal detected from the user's uttered speech and the knowledge of the functions and characteristics of each object in the object world stored in the object characteristic accumulation unit Information that the user is paying attention to in terms of utterance content, and A context-based attention information detection unit that outputs as an information signal; a space-based attention information signal, the context-based attention information signal, and a current state of each object included in an object world managed by a state management unit and a conversation state with the system. As an input, detecting information on what and how the user is paying attention to, an attention information integration unit that outputs this as an attention information integration signal, and the attention information integration signal as an input, According to the content of the attention information integration signal, a feedback generation unit that outputs a feedback signal to the image recognition unit and the speech recognition unit that output the image recognition signal and the speech recognition signal, respectively, As an input, if an operation is actually feasible, the operation is actually performed in the target world. A dialogue control unit that returns a response to the user and manages a part of the dialogue between the user and the system, and a target that accumulates and manages target knowledge such as commands for each target when the dialogue control unit actually implements operations. A multimodal information analysis device comprising an operation content storage unit. 2. If the space-based attention information detection unit receives a new image recognition signal within a predetermined time interval set with respect to the time when the image recognition signal is input, the image recognition is performed. An image recognition signal management unit that outputs a signal as a group of signals and outputs the image recognition signal as it is if no new image recognition signal is input within the predetermined time interval; and an image from the image recognition signal management unit. A recognition target signal specifying unit that specifies what the user is actually paying attention to by using spatial knowledge in the target world stored in the target position storage unit as an input of the recognition signal; The multimodal information analysis device according to claim 1, wherein: 3. The context-based attention information detection unit performs language processing and analysis using the speech recognition signal as an input, and inputs a language analysis unit that specifies the meaning of text and a language analysis output from the language analysis unit. The content of interest that specifies what the user is actually paying attention to, using the knowledge of the functions and characteristics of each object in the target world stored in the target characteristic storage unit. The multimodal information analysis device according to claim 1, further comprising a specifying unit. 4. The attention information integration section receives a space-based attention information signal output from the space-based attention information detection section and a context-based attention information signal output from the context-based attention information detection section as inputs. And a time consistency verification unit for verifying whether the respective attention information signals should be analyzed in an integrated manner with respect to the effective time section of each attention information signal, or whether any of them should be analyzed alone. The space-based attention information signal with synchronization verification flag and the context-based attention information signal with synchronization verification flag output from the time consistency verification unit are input, and the content of each synchronization verification flag and the state management unit are managed. Considering the current state of the configuration object in the target world and the dialogue state of the system, the user Multimodal information analysis apparatus according to claim 1, characterized in that it comprises a knowledge consistency verification unit for verifying whether the focus on what it for. 5. The state management unit according to claim 1, further comprising: sequentially reading out, adding, and reading a current state of a plurality of constituent objects included in the target world or constituting the target world.
Claims characterized by comprising a target state management unit for editing, storing, and managing, and a system state managing unit for sequentially reading, adding, editing, storing, and managing the content of a dialogue between the user and the system, and the state. Item 2. The multimodal information analysis device according to item 1. 6. A feedback signal output by the feedback generation unit, according to data stored in the target position storage unit and the target characteristic storage unit, which is knowledge of a target world.
2. The multi-modal information analysis device according to claim 1, wherein the multi-modal information analysis device is an image recognition feedback signal and a voice recognition feedback signal that are respectively output. 7. An interactive control unit, which receives an attention information integrated signal output from the attention information integration unit as an input, and verifies whether it is possible to actually perform an operation on a target world. A feasibility verification unit, and a feasibility verification signal output from the feasibility verification unit is input, and an operation is performed on a target of interest of the user according to the target world knowledge stored in the target operation content storage unit. And an operation content execution unit that interprets the content of the operation content signal output by the operation content generation unit and outputs the operation content signal to an actual target. The multi-modal information analysis device according to claim 1. 8. The multi-modal information analysis apparatus according to claim 1, further comprising an attention information storage unit that accumulates and manages the attention information integration signal output from the attention information integration unit.