JP2008015585A - Living task support system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To support by presenting information and guidance according to a step when a user executes a life task composed of a plurality of steps.
A task model acquisition unit 22 converts each step of a life task selected by a user from a task teaching material database 13 into a task model represented by a more detailed substep, and generates a state transition model. The task progress control unit 23 acquires a context from the user's action / state obtained from the sensor information, and specifies a substep of the state transition model that matches this context. The interaction control unit 24 presents the task contents of this sub-step to the user via the video presentation unit 26 and performs voice guidance via the voice output unit 25 to control the interaction. The video retrieval unit 27 retrieves the teacher video data of the requested daily task step and the video data of the work performed by the user, and outputs it by the video presentation unit 26.
[Selection] Figure 1

Description

  The present invention relates to a life task support system that supports a user's life task using an interactive function. More specifically, when a user performs a daily task consisting of multiple steps, an interactive dialog function is inferred from the sensor information to indicate the context representing the action of the user being executed or the situation being placed. TECHNICAL FIELD OF THE INVENTION

  The life task is a work or operation for a certain purpose performed in a user's general life or other various activities, and is composed of a plurality of time-series steps.

  With a ubiquitous environment as a background, a life support system that is installed in a user's living space and autonomously supports the user's life has been realized. In order to maintain a comfortable environment and support user activities, this life support system infers the environment and user behavior based on various sensor information, and is connected to a network-connected appliance (home appliance with a communication function). ), And provides various services for supporting the life of the user by controlling the information terminal device and the like.

  For example, the life support system of Non-Patent Document 1 analyzes the user's movement / posture and room conditions based on sensor information and infers that the user is in a relaxed state. By playing music that suits the user's preference, or maintaining indoor temperature and humidity with an air conditioner, the user can realize a comfortable living environment without consciously operating these appliances. Yes.

In addition, as shown in the dialogue system of Patent Document 1, the present inventor has a life support system that provides a life support service by independently controlling an appliance that is constructed in the entire living environment of a user, and a life support system. Has realized a system including an interactive robot that interacts with a user by voice to explain the service provided by the user (see Patent Document 1).
Mino Tetsuhiko, “The Purpose and Summary of the Yukari Project: UKARI Project Report No. 1”, IPSJ 66th National Convention, 2004 WO2005 / 086051A1

  As described above, the life support system can execute various life services according to the context inferred based on information obtained by sensing the user's operation and the environment surrounding the user. For example, it is possible to execute a service for controlling the appliance adaptively to the situation, such as “adjusting the temperature and humidity of the room with an air conditioner” according to the user's operation and surrounding environment (context).

  However, there is a need for a service that can provide more advanced support based on the contents of the user's daily tasks. For example, when performing tasks such as cooking tasks, cleaning tasks, and nursing tasks that are performed according to a series of procedures for a predetermined purpose, the task is being executed by inferring contexts such as user actions and surrounding conditions. There is a need for a service that can provide appropriate support according to the user's actions and conditions.

  As a conventional system that supports cooking tasks, which is one of the daily tasks, the user's actions during cooking are shot with a video camera, and the user temporarily stops cooking tasks for some reason, and then the cooking task There is known a system that enables the user's operation to be confirmed by playing back video data in which the user's operation immediately before the interruption is recorded when the operation is restored. However, in this conventional system, since 5W1H information about the procedure and method in the cooking task, that is, information “who, what, when, where, how” is not accumulated on the system side, From the user's recorded video data, the part that was performed just before the specific time (at the time of interruption) was simply played back to confirm the user's operation.

  The cooking task has a plurality of predetermined procedures (steps), and each step is defined to perform a predetermined operation. The cooking task is considered successful when the specified actions can be performed in the order of steps.

  Therefore, if a step to be performed is omitted or an operation to be performed in a step cannot be completely performed, the cooking task will fail. For example, a user who is cooking may not be able to remember “whether seasoning has already been added or not yet added” when cooking is temporarily interrupted due to a telephone response and then returned to cooking. . In such a situation, there is a possibility that the step of “seasoning” during the cooking task may be missed or performed repeatedly, leading to failure of the cooking task.

  In addition, if the user is a beginner, even if he intends to perform a predetermined action at each step of cooking, there is a risk that he / she may miss the step or perform an incorrect action without his / her awareness. The task will fail.

  In order to eliminate such failures as much as possible, it is necessary to provide information and guidance according to the steps the user is performing.

  In addition, there is a request on the user side to confirm a situation that causes a failure such as “which step has failed” or “what operation error has occurred” after the cooking task is completed.

  It is an object of the present invention to acquire a context representing a user's operation and situation from sensor information in a living task support system having an interaction function with a user, and the user executes a living task composed of a plurality of steps. In this case, the system provides a system that supports the user's life task by presenting information and guidance according to the step being performed by the user.

  In addition, another object of the present invention is to present the user with the life task execution information after the user has performed the life task, and further present the user's life task. It is to provide a system that supports

  The present invention accumulates implicit knowledge information about the user's environment, operations performed in the living task, etc., and functions to model the living task, and sensed information to support the living task performed by the user. , A function that estimates the context that represents the user's actions and surrounding circumstances, and a function that interactively presents the information required by the user who is performing a daily task at an appropriate timing, media, and content granularity And

  The present invention relates to task knowledge information that defines the vocabulary used in the life task, task information including the operation of each life task, the equipment used, and the place where it is executed, and the life of the user. Inferred from the sensor information measured in the space where the task is executed, the action / state information related to the user's action and state during the life task execution inferred from the sensor information, and the action / state information. A living task support system having a distributed environment behavior database for storing a context, and 1) a task model storing a task model in which contents and procedures of a living task composed of a plurality of steps are described in a plurality of substeps Storage means, and 2) a task model of a life task selected by the user from the task model storage means. A task model acquisition means for outputting, and 3) an interaction control for controlling an interaction with the user by performing a process of generating and outputting an inquiry to the user and a process of interpreting a response content of the user in response to the inquiry And 4) a task for generating a state transition model configured in accordance with user actions and states that can be detected as sensor information stored in the distributed environment behavior database based on the extracted task model And 5) estimating a context representing the user's action or state in the life task based on the user's action information or state information generated from the sensor information or the response content, and from the state transition model Task progress that identifies substeps of the task model that match the context And control means, 6) the contents of the life tasks associated with the identified sub-step, characterized in that it comprises a task content presenting means for presenting to the user.

  The present invention includes task model storage means for storing a task model in which the contents and procedure of a living task are described in a plurality of substeps in order to support the living task designated by the user. Then, the task model acquisition unit acquires the task model of the life task selected by the user from the task model storage unit. Further, the task description means generates a state transition model configured corresponding to the user's operation and state that can be detected as sensor information based on this task model.

  Then, the task progress control means estimates a context representing the user's action or state in the life task based on the user's action / state information generated from the sensor information and the response content, and the context is converted from this state transition model. Identify matching task model substeps. And the content of the life task linked | related with the specified substep is shown with respect to a user by a task content presentation means. In addition, the interaction control means generates a question to the user based on the context while the life task is in progress, and converts this question into voice data and outputs it, or voice data uttered by the user in response to the question The voice response is interpreted and the response content to the question is interpreted.

  As a result, information corresponding to the step being executed can be presented to the user who is executing the life task.

  In the present invention, the task description means acquires task teaching material data in which the contents and definition of the life task are described, and generates a task model in which each step of the task teaching material data is described in a plurality of substeps.

  Further, the task description means searches the distributed environment behavior database for a context related to the life task requested by the user, generates a task model of the life task based on the information described in the context, A state transition model is generated from this task model.

  The present invention further includes a task teaching material database for storing task teaching material data in which the contents and procedures of daily tasks are described, and the task teaching data is acquired from the task teaching material database by the task description means. To do.

  In addition, the present invention provides a teacher video database that stores teacher video data that records a model action at each step of the life task, and teacher video data related to the identified life task step from the teacher video database. Video search means for searching, and the searched teacher video data is output by the task content presentation means.

  In addition, the present invention includes a user video database that stores user video data that records the actions of a user who is performing a life task, and user video storage processing means that stores user video data in the user video database. When there is a video presentation request from the user by the retrieval means, the teacher video database is searched for the teacher video data corresponding to either the requested action or state or the life task step, and the request is received from the user video database. Such user video data is retrieved, and the task content presentation means can output the retrieved teacher video data and user video data, for example, in a state that the user can compare. Accordingly, the user can present both the contents of the action performed by the user and the information used as a model action for performing the life task, and can be compared with the user.

  Further, the interaction control means of the present invention includes a dialog dictionary database for accumulating language information for generating a dialog with a user, a question generating means for generating a query for the user based on the context, It is characterized by comprising voice synthesis means for converting a question into voice data and speaking, and voice recognition means for recognizing the contents of voice data spoken by a user. As a result, it is possible to realize interaction with the user by voice data. This allows interaction with the user by voice.

  Further, the present invention generates a user state model for the user state information from the user state obtained from the sensor information or the progress state of the life task obtained from the context by the interaction control means, and the user state model is also generated. And a predetermined parameter for determining the content of the interaction. The predetermined parameters are, for example, contents to be notified to the user, content granularity (degree of detail), and media to be used (speech, image, text, etc.). As a result, the user can interact with the contents according to the skill of the user.

  The living task support system of the present invention includes 1) task teaching material data in which the contents and procedure of a living task composed of a plurality of steps are described, and a teacher who records a model operation in each step of the living task. Task teaching material database that stores video data, and 2) When a user requests video presentation, identify the daily tasks required for the request based on the distributed environment behavior database, and meet the requirements in the identified daily tasks A presentation target specifying means for estimating a context representing an action or a state, and identifying a step of a living task according to a request from task teaching material data based on the estimated context; and 3) teaching video data corresponding to the identified step. , Video search means for extracting from the task teaching material database, and 4) the extracted teacher video data Outputs a and a video presentation unit.

  In the present invention, when there is a video presentation request from the user, the requested living task is specified based on the distributed environment behavior database by the presentation target specifying means, and the action or The context representing the state is estimated, and the procedure of the life task related to the request is specified from the task teaching material data based on this context. Then, the teacher video data corresponding to the identified procedure is extracted from the task teaching material database by the video search means, and the extracted teacher video data is output by the video presentation means.

  As a result, when the user wants to check the procedure after completing the life task, the user can view the corresponding teacher video data in response to the teaching material data presentation request.

  In addition, the present invention includes a user video database that stores user video data that records the actions of a user who is performing a life task, and user video storage processing means that stores the user video data in the user video database. Then, the video search means searches the user video data from the user video database for the requested step, and the video presentation means outputs the searched teacher video data and user video data.

  According to the present invention, with respect to a life task that is completed after a plurality of steps are sequentially performed, task teaching material data describing task contents and operations in each step, video data recording a model operation, etc. From the sensor information and user's utterances, and the context surrounding the user who is working is extracted. Based on this context, the steps to be executed by the user, the contents of the tasks to be executed at each step, the following The user can be presented with guidance to the steps.

  As a result, it is possible to prevent a user who is executing a task from skipping a necessary step or making a mistake in the task contents at the step, and assisting the user to complete the life task without failure.

  Further, according to the present invention, it is possible to present both video information as an example of operation at each step of a daily task and video information of a task actually executed by a user. As a result, it is possible to display the step and task contents inquired by the user so that the model video and the user's own video can be compared. As a result, the user can compare the model action with his or her own action to confirm the location of the failure and the cause of the failure. Can help.

  Although the present invention is an object of support for daily tasks, it is not limited to tasks performed in a narrow sense of life, and tasks in any field can be applied as long as a plurality of steps are sequentially performed as described above. Can be supported. According to the present invention, for example, it is possible to support a work task called “a stand system” in which a single worker completes a predetermined assembly work through a plurality of processes. For this reason, it is possible to explain the contents of the work in correspondence with each worker, and it is expected to improve the efficiency and accuracy of the work.

  Hereinafter, as a best mode for carrying out the present invention, a case where a user performs a “cooking task” as a living task will be described as an example.

  FIG. 1 is a diagram showing a configuration example of a life task support system of the present invention.

  The living task support system 1 includes a distributed environment behavior database 11, a task model database 12, a task teaching material database 13, a user video database 15, a sensor information collection unit 21, a task model acquisition unit 22, a task progress control unit 23, and an interaction control unit 24. , An audio output unit 25, a video presentation unit 26, a video search unit 27, and a state transition model storage unit 28.

  The distributed environment behavior database 11 detects event information / state information, events detected based on information detected by various sensors, such as the user, objects existing around the user, and the environment (space state) surrounding the user. It is a database that hierarchically manages and accumulates contexts inferred from information / state information. The distributed environment behavior database 11 is a platform that can manage, store, and use information of “people / things / environments” in a so-called living space. The distributed environment behavior database 11 is realized by using the technique described in Patent Document 1 (WO2005 / 086051A1).

  2 and 3 show examples of the configuration of the distributed environment behavior database 11. The distributed environment behavior database 11 includes four managements: a sensor information database (DB) layer 111, an event / state management database (DB) layer 113, a context database (DB) layer 115, and a domain / task knowledge database (DB) layer 117. Composed of layers.

  As shown in FIG. 3, the sensor information database layer 111 is data management means for accumulating and managing sensor information collected via sensors (sensing devices) arranged in a living space where a user performs a living task.

  There are various user living spaces such as RFID tag sensors, wireless LAN positioning systems, floor sensors, pyroelectric sensors, cameras, microphones, sleep sensors, home appliance status sensors, robot sensors (sensors installed in interactive robots, etc.) Sensors are distributed. Each sensor information collected by the sensors is distributed and stored for each sensing type as binary data of a structure having a time stamp, data type, and data size as a header.

  The event / state management database layer 113 detects events and states that occur in the living space based on the sensor information in the sensor information database layer 111, and accumulates and manages the detection results (event information and state information). Data management means. As the event and state detection processing, specifically, position detection processing, person identification processing, motion detection processing, speech recognition processing, and other processing are performed. These processing results are stored as data in a format in which time information, person / thing / environment ID information (identification information), and position information are the main keys, and the contents of events and states and the certainty factor.

  The context database layer 115 is data management means for inferring a context representing a user's action and surrounding state based on information in the event / state management database layer 113, and storing and managing this context data. The context database layer 115 includes a main case (indicating the subject of the context such as a person, an object, and an environment), an action case (indicating the action of the main case), a target case (indicating the object of the action), and a place case (in which the action is performed). Context data described in XML (eXtensible Markup Language) format such as information such as confidence level (degree of confidence that is an estimated context) and the like are stored.

  FIG. 4 is an example of context data stored in the context database layer 115. The context data shown in FIG. 4 indicates that the user with the specified ID (primary case) extracts the salad oil (target case) from the specified position (location case) with a certainty factor of 0.85 during the described time zone. (Operation case) ".

  The domain / task knowledge database layer 117 is data management means for accumulating and managing domain knowledge information and task knowledge information. In the domain / task knowledge database layer 117, domain knowledge data that defines service domains such as vocabulary of life tasks executed by the user, operations in life tasks, appliances to be used, correspondence in the execution place, etc. can be diverted. Task knowledge data that defines implicit task content is stored in XML or RDF (Resource Description Framework) format.

  FIG. 5 shows an example of domain / task knowledge data in the domain / task knowledge DB layer 117. FIG. 5A is an example of task knowledge data indicating tacit knowledge about cooking. Here, the content that “the cooking operation of Boil requires a pan and is executed in the range” is defined by the RDF description that shows the relationship between the cooking utensil and the utensil / place, etc. is doing. FIG. 5B is a diagram illustrating a description example of main cooking operations. In addition to “Boil”, “Fire extinguishing (closeFire), stopping water (closeWater), etc.” are defined.

  The task model database 12 is data storage means in which task models are stored. The task model is data that describes the contents and procedure of a daily task as a structure with a plurality of steps.

  The task teaching material database 13 is a data storage unit that stores task teaching material data. Task teaching material data is data in which the contents and procedures of daily life tasks are described in a natural language in a form that can be understood by humans. In this example, the task teaching material data is cooking recipe data. In addition, the task teaching material database 13 accumulates teacher video data in which actions that serve as examples in each procedure (step) of the life task are recorded.

  In this example, cooking recipe data and teacher video data are accumulated in the task teaching material database 13.

FIG. 6 shows an example of cooking recipe data. The cooking recipe data in FIG. 6 is information in which the ingredients necessary for the cooking task of “Fry this oyster sauce when beef” and the cooking operation to be executed are described in time series. Here, it is divided into a plurality of steps indicating the details of the work as follows.
“S1. Dissolve 1/2 tablespoon of starch starch in 1/2 tablespoon of water.
S2. 250g of beef thigh is chopped along the fiber.
S3. Add 1 tablespoon of sake, 1 tablespoon of soy sauce, 2 tablespoons of water-soluble cauldron (dissolved in the same amount of water), and 1 tablespoon of salad oil to the beef thigh.
S4. Place the beef leg for about 20 minutes,
S5. Take a stone of 1 pack of maitake and 1 pack of shimeji… ”.

  The teacher video data is video data of an operation that serves as an example of a cooking operation executed in each step.

  The contents of life tasks can usually be divided into a number of smaller steps. The task teaching material data is divided into several steps, and the contents are described for each step. However, the task model stored in the task model database 12 divides each step of the task teaching material data into finer steps ( It is divided into sub-steps).

  As will be described later, if the task model of the life task to be supported is accumulated in the task model database 12, the task model acquisition unit 22 uses the accumulated task model to change the state transition model for task progress control. Is generated. In addition, if there is no task model of the life task to be supported, a task model is generated from the task teaching material data. Therefore, the living task support system 1 only needs to include either the task model database 12 or the task teaching material database 13.

  The user video database 15 is data storage means for storing user video information obtained by photographing a cooking operation of a user who is executing a cooking task. The user video information is photographed by a fixed camera installed on a kitchen work table as one of the sensors, and stored in the user video database 15 via the sensor information collection unit 21.

  The sensor information collection unit 21 is a processing unit that collects sensor information detected by the sensors distributed and installed in the user's living space and stores it in the distributed environment behavior database 11. In kitchens where cooking tasks are performed in the user's living space, floor sensors that detect the user's position and movement, reed switches that detect the opening and closing of doors such as cupboards and food shelves, and RFID tags used in the task RFID reader that detects the position of attached objects (cooking utensils, food storage containers, tableware, etc.), a sensing function that detects the operating state of appliances such as refrigerators, a microphone that collects voice data spoken by the user, A camera to shoot the top board will be installed.

  The task model acquisition unit 22 is a processing unit that performs task model acquisition processing, task model description processing, state transition model generation processing, and the like.

  The task model acquisition unit 22 extracts a task model of a life task selected by the user from the task model database 12 as a task model acquisition process.

  Also, as task model description processing, cooking recipe data of the cooking task selected by the user is extracted from the task teaching material database 13, and each step of the cooking recipe data is converted into substeps divided into more detailed cooking operations. Describe the model. Here, some of the sub-steps generated in a certain step may describe operations and states performed in an array.

  Further, as the state transition model generation process, a state transition model configured corresponding to the user's operation and state that can be detected as sensor information is generated based on the task model and stored in the state transition model storage unit 28.

  The state transition model generated here is state data (event / state management database layer 113) of cooking utensils, tableware, ingredients, etc. (objects) used in cooking tasks, stored in the distributed environment behavior database 11. Based on task knowledge data related to cooking tasks, it is configured with user actions and states that can be detected as sensor information as elements.

  The task progress control unit 23 refers to the task learning data, and performs sensor event detection processing, context extraction processing, and presentation target in order to execute real-time support processing according to the user's life task operation acquired by the sensor information. This is a processing means for controlling the progress of the life task to be supported by performing specific processing. The task progress control unit 23 controls the task model acquisition unit 22, the interaction control unit 24, the video search unit 27, and the like.

  The task progress control unit 23 acquires the user action / state information generated from the sensor information and the user response content obtained from the interaction control unit 24 as sensor event detection processing. Furthermore, as context extraction processing, a context representing the user's action or state in the cooking task is estimated based on the sensor information or response content, and the substep of the cooking task that matches this context data is identified from the state transition model. To do.

  Further, the task progress control unit 23 estimates a context representing an operation or a state that matches the requested content in the requested life task when the presentation of the teacher video data is requested by the user as the presentation target specifying process. Then, the corresponding step of the task teaching material data is specified based on the context data.

  The interaction control unit 24 controls, as an interface with the user, dialogue processing with the user, display processing of the contents of the cooking recipe data stored in the task teaching material database 13, comparison display processing between the user video data and the teacher video data, and the like. Processing means.

  The interaction control unit 24 is a dialogue dictionary database that accumulates language information for generating a dialogue with the user and a context extracted by the task progress control unit 23 as a processing means for dialogue processing with the user. A question generating unit for generating a question to the user, a voice synthesizing unit for converting the generated question into voice data, and a voice recognition unit for recognizing the contents of the voice data spoken by the user.

  As an interactive process with the user, the interaction control unit 24 guides cooking contents and cooking operations performed in the steps of the cooking recipe data specified by the task progress control unit 23, introduction guidance to the next step, and completion of the cooking operation. Converts questions such as confirmation to voice data and outputs it. Also, the user's utterance to the question is input, voice recognition is performed, the response content is analyzed, and the task progress control unit 23 is passed.

  Moreover, the interaction control part 24 converts the content of the cooking recipe relevant to the substep of the cooking task specified by the task progress control part 23 into voice data, and outputs it as display processing of the content of cooking recipe data.

  Further, the interaction control unit 24 arranges and displays the corresponding portions of the user video data and the teacher video data extracted by the video search unit 27 when there is a video presentation request from the user as a video data comparison display process. To display.

  The audio output unit 25 is a processing unit that outputs the audio data generated by the interaction control unit 24. The video presentation unit 26 is processing means for displaying the contents of cooking recipes and video data on a display.

  FIG. 7 shows a display example of the display. FIG. 7A shows a display during support for cooking tasks. The display screen 200 displays a cooking task content display 201 corresponding to a cooking task step, a teacher video data display 203 of a cooking operation as a model, and the like. FIG. 7B is a video display example in response to a search request. On the display screen 200, a user video data display 205, which is the user's own cooking operation, and a teacher video data display 207 for the corresponding steps and operations are displayed side by side.

  The video search unit 27 is a processing unit that searches the task teaching material database 13 or the user video database 15 and acquires video data related to the requested location or operation when a video presentation request is received from the user.

  Hereinafter, the cooking task support processing by the life task support system 1 will be described in more detail.

(1) Detection of Sensor Event The sensor information collected by the sensor information collection unit 21 is accumulated in the sensor information database layer 111 of the distributed environment behavior database 11. In the distributed environment behavior database 11, when sensor information is accumulated, a sensor event is detected. Then, person identification, movement, conversation content (conversation text), etc. are detected and recognized from the sensor information. As the processing results, user actions (event information), kitchen status (status information), and the like are estimated and stored in the event / status management database layer 113. Furthermore, the collected event information and state information are inferred in an integrated manner, and the context is estimated. The context data is stored in the context database layer 115.

  Further, the sensor event is notified to the task progress control unit 23.

(2) Cooking Task Modeling In order to support cooking tasks, it is necessary to present cooking recipe data appropriately as the user progresses cooking. In order to manage the user's cooking progress, the cooking task is modeled and the state is defined in order to determine which substep of the cooking task corresponds to the user's action or state that can be specified by the sensor event.

  When the user requests a support service for the cooking task of “Fry this oyster sauce when beef”, the task model acquisition unit 22 retrieves the cooking recipe data of “Fry this oyster sauce when beef” from the task teaching material database 13. get.

  Then, each step described in the cooking recipe data of “Stir-fried beef with oyster sauce” is divided into more detailed sub-steps and converted into a cooking task model.

  Specifically, as shown in FIG. 8, task knowledge data stored in the domain / task knowledge database layer 117 of the distributed environment behavior database 11, event / state information stored in the event / state management database layer 113. Each step of cooking recipe data is converted into a cooking task model with reference to the above.

For example, implicit task knowledge data "A container is required to melt A with B. Melting operation is normally performed on a workbench ..." and status data "container is bowl object1, The bowl object2 exists in the upper left cupboard of the kitchen .... "is accumulated, step S1" 1. Melt 1/2 tablespoon of starch starch with 1/2 tablespoon of water "in the cooking recipe data is as follows. Are divided into six sub-steps.
“SubS1: Place the bowl on the workbench,
SubS2: Place starch on the workbench,
SubS3: Place the measuring spoon on the workbench,
(SubS1 to 3 are in no particular order),
SubS4: Put potato starch into the bowl with a measuring spoon.
SubS5: Put water into the bowl with a measuring spoon,
SubS6: Stir the starch. "

  The task model acquisition unit 22 further recognizes user events (operations) that can be recognized by sensor events from the cooking task model so that cooking task progress control can be performed using user event information / status information that can be acquired by sensor events. ) And surrounding states are extracted as elements, a state transition model is generated, and stored in the state transition model storage unit 28. For example, in the case of the above step S1 “dissolve starch with water”, the user's action or state that can be recognized by the sensor event is “the bowl appears on the workbench”, “the starch powder container appears on the workbench”, There are three: “Measuring spoon appears on the workbench”. Therefore, in the state transition model of the above step, these operations or states are included in the elements.

(3) Acquisition of context The task progress control unit 23 uses the event / state information generated by the sensor event to identify substeps that match the action of the user who is executing the cooking task and the surrounding state. Acquire the context.

  In this example, in the sensor event, sensor information for knowing the operation and situation of the user during cooking is collected via a sensor as shown in FIG.

  ・ Cameras: As shown in FIG. 9, two cameras are installed on the kitchen workbench and on the IH cooking heater, respectively, and objects related to cooking tasks (bowl, cutting board, frying pan, etc.) in each shooting area. , Tableware, food containers, etc.). For example, ARTToolKit (see http://www.hitl.washington.edu/artoolkit) is used to detect the position of the object. A marker compliant with ARTToolKit is pasted on the object in advance. For an object with a marker attached, it is possible to detect the position information of the object by detecting the marker from the captured image.

  In addition, a marker is affixed to the inner bottom of the bowl in advance, and when the bowl is empty, the marker is photographed, and when the bowl is filled with ingredients, the marker is hidden by the placed ingredients. It is possible to detect the state information of the object by utilizing that the image is not taken.

  Furthermore, the video of the user's hand on the work table or the IH cooking heater taken by the camera is stored in the user video database 15 as user video data.

  -Reed switch: Install a reed switch on the knife holder that holds the knife, and attach a magnet to the knife. As a result, it is possible to detect an event that “the knife has been removed from the knife holder”. In addition, by installing a reed switch on the door of a cupboard or food storage and detecting the opening / closing of the door, it is possible to detect “the object has been accessed”.

  ・ Vibration sensor: A vibration sensor is installed on the workbench in the kitchen to perform cooking operations such as “An object is placed on the workbench”, “Turn off an object (foodstuff)”, and “Crush the object (foodstuff).” Can be detected.

  In the actual sensing environment, the accuracy of the recognition process of the user's action and the state of the kitchen is low, and sometimes the change of the action and the state cannot be recognized at all. Therefore, the user's will may be explicitly input. For example, by cooperating with the interaction control unit 24, the user asks for confirmation of the user's operation and the surrounding state to be confirmed in the progress of the cooking task, and acquires the operation and kitchen state by the user's response.

  Alternatively, a confirmation button or the like may be installed in the kitchen, and the user may decide to press the button at the end of the cooking operation and detect the button press to recognize the user operation or the state of the kitchen. .

  By associating such sensor events with the state transition model generated from the cooking task model, the user can acquire the context of “when and what object, what operation was performed”. .

  As shown in FIG. 10, while the user performs a cooking task, the user's actions and the kitchen are detected from sensor information such as the position of the cooking utensil, the user's posture and orientation, and the user's utterance detected by the sensor group. Event information including content such as the state of the event and the certainty level is generated. For example, event information “a cooking utensil has appeared on the work table” is generated from a sensor event that the camera has detected a marker attached to the bowl.

  The event information is input to the state transition model of the corresponding step of the cooking recipe. Event information generated from the sensor event, such as “a cooking utensil (bowl) has appeared on the workbench”, “a food (boiled starch) has entered the cooking utensil (bowl)”, or the like, is included in each step (Sn, Sn + 1) of the cooking task. ,...), And the context in the cooking task is acquired. This makes it possible to manage which step or sub-step is being executed.

  When the context that the last sub-step of a certain step Sn is completed is acquired by the input event information, the task progress control unit 23 switches to the next step Sn + 1 of the cooking recipe data.

  The relationship between the cooking task model and the sensor event will be described with reference to FIGS. 11 and 12. Here, in the cooking recipe data (see FIG. 6), the process proceeds in the order of step S1 “melting starch starch with water”, step S2 “slicing beef leg meat”, and so on.

  FIG. 11 is a diagram showing the relationship between the substep of step S1 “melting starch with water” and the sensor event expected to be input.

  When the user removes the bowl and the starch starch container from the kitchen cupboard according to the instructions of the cooking task uttered by the interaction control unit 24 and puts them on the workbench, the camera pastes them to the bowl and the starch starch container. The detected marker is detected. Event information generated from this sensor event, such as “a bowl marker appears on the workbench” and “a starch marker on the workbench appears” is input to the state transition model. The contexts “placed”, “the user placed the starch on the workbench”, and “the user placed the measuring spoon on the workbench” are acquired. As a result, the task progress control unit 23 knows that the user has performed the operations of the sub-steps SubS1 to SubS3.

  After that, when the user puts potato starch into the bowl and the marker affixed to the bottom of the bowl becomes invisible, the event information that “the marker on the bowl disappears” is input, and “the user puts potato starch into the bowl”. Thus, it can be seen that the user has performed the operation of sub-step SubS4.

  However, a cooking operation in which a user puts water into a bowl with a measuring spoon and stirs potato starch may not be recognized by sensing with a camera or a vibration sensor. Therefore, the user is inquired whether or not the operation of melting the starch is finished by using the interaction control described later, and acquires the context that “the user melts the starch with water” from the content of the user's response, The end of S1 “dissolve starch with water” is determined.

  Thereafter, as shown in FIG. 12, the control target is shifted to step S <b> 2 “slicing beef leg meat”. Similar to the control in step S1, when the user removes the beef container from the refrigerator, the kitchen knife from the storage unit, and the cutting board according to the instructions of the system and places them on the workbench, the marker attached to the container and cutting board by the camera is displayed. User event information generated from sensor events such as detection and detection of the removal of a kitchen knife is input, and "user puts meat on workbench", "user puts kitchen knife on workbench ”,“ The user places a cutting board on the workbench ”is acquired.

  As a result, the task progress control unit 23 knows that the user has performed the operations of the sub-steps SubS10 to Sub12. Thereafter, when the user is cutting the meat, the context that “the user is cutting the object (meat) on the workbench” can be obtained from the sensor event of the vibration sensor of the workbench, so that the user can execute the sub-step SubS13, It can be seen that 14 operations have been performed.

(4) Interaction control The interaction interface between the daily life task support system 1 and the user is realized by an interaction control unit 24, an audio output unit 25, and a video presentation unit 26 having a display as an output unit.

  In this example, the interaction control unit 24 and the voice output unit 25 are implemented by a dialogue robot constituting the dialogue system of Patent Document 1 described above, and realized by, for example, an existing anthropomorphic voice dialogue agent tool kit (Galatea Toolkit). (For details, see Patent Document 1).

  When the task progress control unit 23 is given a cooking task state (context) derived by matching the sensor event and the cooking task model in the task progress control unit 23, the interaction control unit 24 generates an interaction according to the state.

  Here, the following interactions are generated.

・ "Ask whether the current step is finished"
When the progress of the cooking task cannot be recognized depending on the sensor event, a question for confirming whether or not the step in progress is completed is generated and the user is uttered. The interactive robot asks the user, “Is the cooking step I'm done now?”, And if the user gives a positive response, “I'm done.” Can be acquired, and the control target is advanced to the next step.

・ “Next Step Guidance”
As shown in FIG. 13, when it is recognized by the context based on the sensor information that the user has reached the operation corresponding to the last sub-step of the current step Sn, first in the next step Sn + 1. Guidance on sub-steps (operations) to be performed is performed. Then, when the user starts the operation of the sub-step to be instructed, the task progress control in step Sn + 1 is performed such that the cooking recipe data of the corresponding step is presented.

・ "Warning for unexecuted cooking action"
When the execution of the cooking operation to be performed cannot be recognized due to the sensor event, a warning is given that there is an unexecuted sub-step (operation).

  The interaction control unit 24 generates a model (user state model) for the user state information in the same manner as the state transition model based on the user operation (event information) during execution of the cooking task.

  As shown in FIG. 14, the user state model includes user state information obtained from sensor events, task state information obtained from the cooking task model, user profile information stored in the distributed environment behavior database 11 (user cooking skill). Described in dynamic Bayesian based on degrees). The interaction control unit 24 determines the interaction control parameters such as the granularity (detail level) of the guidance of the cooking task, the timing of the guidance presentation, the warning to the user, and the selection of the media to be presented based on this user state model.

  By using these interaction control parameters, if the user is a beginner cooking, the system will carefully check the cooking operation of each step and warn if the user tries to proceed to the next step without performing the cooking operation. The error recovery function can be realized at a level adapted to the user.

(5) Review Support by Cooking Video Search When the user requests presentation of cooking task teacher video data from the user, the task progress control unit 23 determines the cooking task step or the sub-step of the task model from the content of the request. Identify cooking actions.

  Then, the video search unit 27 searches and acquires the teacher video data corresponding to the step or sub-step of the cooking task specified by the task progress control unit 23 from the task teaching material database 13. Further, the user video data corresponding to the specified cooking task step or sub-step is retrieved from the user video database 15 and acquired.

  The video presentation unit 26 displays the acquired video data side by side on the display.

  For example, the user feels that there is a problem with how the meat is fried after the cooking is finished, and wants to review his cooking operation and requests “I want to see what happens when the meat is fried”. The interaction control unit 24 extracts search keywords such as “meat” and “sauté” from a request issued by the user through speech recognition processing, morphological analysis processing, and the like. This search keyword is passed to the task progress control unit 23.

  The task progress control unit 23 searches the cooking recipe data in the task teaching material database 13 for the cooking task step including the search keyword. The video search unit 27 acquires teacher video data corresponding to the searched cooking task step. Further, the task progress control unit 23 searches the context database layer 115 of the distributed environment behavior database 11 for time information when the user performs the corresponding cooking recipe step. The video search unit 27 acquires the corresponding user video data from the user video database 15 based on the obtained time information.

  Then, the video presentation unit 26 displays the teacher video data and the user video data side by side on the display. Thus, the user can review the model image and his / her cooking in a calm state after cooking is completed.

  While the present invention has been described above with reference to the embodiments, it is obvious that the present invention can be variously modified within the scope of the gist thereof.

  For example, the life task support system 1 can be implemented as a system incorporated in the dialogue system of Patent Document 1 described above. In this case, the distributed environment behavior database 11, the task model database 12, the task teaching material database 13, the user video database 15, the sensor information collection unit 21, the task model acquisition unit 22, the task progress control unit 23, the video of the living task support system 1 The presentation unit 26, the video search unit 27, and the like are implemented by a life support system of a dialog system. Further, the interaction control unit 24 and the voice output unit 25 are implemented as a dialog robot of the dialog system.

  In the present invention, a position detection system using an RFID tag can be used to detect the position of an object (tool, instrument, etc.) used in a daily task.

  The present invention can also be implemented as a processing program that is read and executed by a computer. The processing program for realizing the present invention can be stored in an appropriate recording medium such as a portable medium memory, a semiconductor memory, and a hard disk that can be read by a computer. It is provided by transmission / reception using various communication networks via a communication interface.

It is a figure which shows the structural example of the life task assistance system of this invention. It is a figure which shows the structural example of a distributed environment action database. It is a figure which shows the structural example of a distributed environment action database, and the data processing of each layer. It is a figure which shows the example of context data. It is a figure which shows the domain knowledge data and the task knowledge data example. It is a figure which shows the example of cooking recipe data. It is a figure which shows the example of a display of a display. It is a figure for demonstrating modeling of a cooking task. It is a figure which shows the relationship between the sensor event by various sensors, and event information / state information. It is a figure for demonstrating the relationship of a sensor event, generation | occurrence | production of event information / state information, and the input to a state transition model. It is a figure for demonstrating the relationship between a cooking task model and a sensor event. It is a figure for demonstrating the relationship between a cooking task model and a sensor event. It is a figure for demonstrating the interaction of the guidance of the next step. It is a figure for demonstrating the production | generation of a user state model.

Explanation of symbols

1 Living Task Support System 11 Distributed Environmental Behavior Database (DB)
12 Task model database (DB)
13 Task teaching material database (DB)
15 User video database (DB)
DESCRIPTION OF SYMBOLS 21 Sensor information collection part 22 Task model acquisition part 23 Task progress control part 24 Interaction control part 25 Audio | voice output part 26 Image | video presentation part 27 Image | video search part 28 State transition model memory | storage part

Claims (10)

For the daily tasks performed by the user, task knowledge information that defines the vocabulary used in the daily tasks, task information including the operation of each daily task, the equipment used, and the place where it is executed, and the user's daily tasks are executed. Sensor information measured in a space, motion / state information related to a user's motion and state during life tasks inferred based on the sensor information, and context inferred from the motion / state information. A living task support system with a distributed environment behavior database that accumulates,
A task model storage means for storing a task model in which contents and procedures of a life task composed of a plurality of steps are described in a plurality of sub-steps;
Task model acquisition means for extracting a task model of a life task selected by the user from the task model storage means;
An interaction control means for controlling the interaction with the user by performing a process of generating and outputting an inquiry to the user and a process of interpreting a response content of the user in response to the inquiry;
Based on the extracted task model, task description means for generating a state transition model configured corresponding to a user's operation and state detectable as sensor information stored in the distributed environment behavior database;
A context representing the user's action or state in the daily life task is estimated based on the user's action information or state information generated from the sensor information or the response content, and matches the context from the state transition model Task progress control means for identifying substeps of the task model;
A life task support system, comprising: task content presentation means for presenting the content of the life task associated with the specified substep to the user. The task description means acquires task teaching material data in which the contents and procedure of the life task are described, generates a task model in which each step of the task teaching material data is described in a plurality of substeps, and the generated task model The life task support system according to claim 1, wherein a state transition model is generated from. The task description means searches the distributed environment behavior database for a context related to a life task requested by a user, and generates a task model of the life task based on information described in the searched context. The life task support system according to claim 1, wherein a state transition model is generated from the generated task model. A task teaching material database that stores task teaching material data describing the contents and procedures of daily tasks is provided.
The living task support system according to claim 2, wherein the task description means obtains task teaching material data selected by a user from the task teaching material database and generates a task model. A teacher video database for storing teacher video data recording a model action in each step of the life task;
Video search means for searching for teacher video data related to the sub-steps of the identified daily tasks from the teacher video database;
The life task support system according to claim 1, wherein the task content presentation unit outputs the searched teacher video data. A user video database for storing user video data recording the actions of the user who is performing the life task;
The user video database comprises user video storage processing means for storing the user video data,
When there is a video presentation request from the user, the video search means searches the teacher video database for teacher video data corresponding to either the operation or state relating to the request or the step of a living task, and the user Search the video database for user video data that has executed the step related to the request,
The living task support system according to claim 5, wherein the task content presentation means outputs the searched teacher video data and user video data. The interaction control means includes:
A dictionary database for dialogue for storing language information for generating dialogue with the user;
A question generating means for generating a question to the user based on the context;
Speech synthesis means for converting the generated question into speech data and speaking;
The living task support system according to any one of claims 1 to 6, further comprising: a voice recognition unit that recognizes the content of voice data spoken by the user. The interaction control means generates a user state model for the user state information from the state of the user obtained from the sensor information or the progress state of the life task obtained from the context, and based on the user state model The living task support system according to any one of claims 1 to 7, further comprising: calculating a predetermined parameter for determining the content of the interaction. For the daily tasks performed by the user, task knowledge information that defines the vocabulary used in the daily tasks, task information including the operation of each daily task, the equipment used, and the place where it is executed, and the user's daily tasks are executed. Sensor information measured in a space, motion / state information related to a user's motion and state during life tasks inferred based on the sensor information, and context inferred from the motion / state information. A living task support system with a distributed environment behavior database that accumulates,
A task teaching material database storing task teaching material data in which contents and procedures of a living task composed of a plurality of steps are described, and teacher video data in which a model operation is recorded in each step of the living task;
When there is a video presentation request from the user, a life task related to the request is specified based on the distributed environment behavior database, and a context representing an operation or a state that matches the request in the specified life task is estimated. Presenting target specifying means for specifying a step of a living task related to the request from the task teaching material data based on the estimated context;
Video search means for extracting teacher video data corresponding to the identified step from the task teaching material database;
A life task support system comprising: video presentation means for outputting the extracted teacher video data. A user video database for storing user video data recording the actions of the user who is performing the life task;
The user video database comprises user video storage processing means for storing the user video data,
The video search means searches the user video data from the user video database for user video data that has undergone the request step,
The living task support system according to claim 9, wherein the video presentation unit outputs the searched teacher video data and user video data.

Images