JP2023084966A - Information recording support method, information recording support device, information recording support program, and information recording support system - Google Patents

Abstract

An information recording support method, an information recording support device, an information recording support program, and an information recording support system are provided that naturally support information recording by means of a vehicle or the like without causing discomfort to passengers. do. Kind Code: A1 An information recording support method includes a vehicle equipped with an exterior camera 29 and an agent system 15 which is an information recording assistance device for supporting information recording at an arbitrary timing using the exterior camera 29 via an agent unit 23. 100. In this information recording support method, the movement and/or speech of the passenger 10 are detected, and the object of interest of the passenger 10 outside the vehicle 100 is identified based on the movement and/or speech of the passenger 10 . When the object of interest is identified, the agent system 15 notifies the passenger 10 of the identification of the object of interest, and the camera 29 outside the vehicle records the object of interest. [Selection drawing] Fig. 2

Description

The present invention relates to an information recording support method, an information recording support device, an information recording support program, and an information recording support system that support information recording at arbitrary timing using an outside camera that photographs the outside of the vehicle.

Patent Document 1 discloses an image/audio recording/reproducing apparatus including a navigation apparatus installed in a vehicle and a recording/reproducing apparatus, which is a computer installed at home or the like. The image/audio recording/reproducing device of Patent Document 1 records video data and audio data in the vehicle with the navigation device, and transmits these data to the recording/reproducing device, thereby saving the trouble of taking pictures by the passengers themselves. In addition, the recorded video data and audio data can be easily reproduced by a computer.

JP 2012-105174 A

2. Description of the Related Art In recent years, a vehicle or a device mounted on the vehicle (hereinafter referred to as a vehicle or the like) may have a function of assisting a passenger in operating the device or obtaining information based on, for example, interactive voice input. As one of such support functions, an information recording support function for automatically recording a moving image or the like as a memory of a trip or the like by a vehicle or the like is under consideration.

However, if the vehicle etc. automatically records videos etc. and then provides the automatically recorded videos etc. to the passengers as a memory of the trip etc., the passengers will have a negative impression of the vehicle etc. that automatically recorded the videos etc. may form. Specifically, if the vehicle, etc., does not show any reaction to the recording of information such as moving images, and the vehicle, etc., selfishly records information, the recorded information will be used later as a memory of the passenger or a memory with the passenger. Passengers may feel uncomfortable even if it is provided as

The present invention provides an information recording support method, an information recording support device, an information recording support program, and an information recording support system that naturally support information recording by means of a vehicle or the like without giving passengers a sense of discomfort. intended to provide

An aspect of the present invention is an information recording support device that has an exterior camera that captures an image of the exterior of the vehicle and an agent unit that functions as an interface for passengers, and supports information recording at an arbitrary timing using the exterior camera via the agent unit. and an information recording support method by a vehicle. In this information recording support method, a passenger's motion and/or speech is detected, and an object of interest of the passenger outside the vehicle is identified based on the passenger's motion and/or speech. Then, when the target of interest is specified, the information recording support device notifies the passenger that the target of interest has been specified, and the target of interest is recorded by the camera outside the vehicle.

According to the present invention, the recording of information can be naturally assisted by the vehicle or the like without making the passenger feel uncomfortable.

FIG. 1 is an explanatory diagram showing a schematic configuration of a vehicle and an information recording support system. FIG. 2 is a block diagram showing the configuration of the agent controller. FIG. 3 is a flow chart for starting automatic information recording. FIG. 4 is a flow chart for ending automatic information recording. FIG. 5 is a flowchart showing a specific example of interest loss determination. FIG. 6 is an explanatory diagram of a scene in which automatic information recording is started. FIG. 7 is an explanatory diagram of a scene during automatic information recording. FIG. 8 is an explanatory diagram of a scene in which automatic information recording ends.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First embodiment]
FIG. 1 is an explanatory diagram showing a schematic configuration of a vehicle 100 and an information recording support system 101. As shown in FIG. The vehicle 100 uses the agent system 15 to support the operation and information acquisition of the passenger 10 by accepting, for example, interactive operation input with the passenger 10 of the vehicle 100 and/or providing various types of information. do. In this embodiment, the vehicle 100 supports the recording of images or videos (moving images), voices, or other information (hereinafter referred to as moving images, etc.) at arbitrary timing by the agent system 15 . Therefore, vehicle 100 including agent system 15 constitutes information recording support system 101 . Also, the passenger 10 is a driver and/or a fellow passenger. In this embodiment, the passenger 10 is assumed to be the driver of the vehicle 100 for the sake of simplicity.

As shown in FIG. 1 , a vehicle 100 includes an agent system 15 , a driving operation unit 11 , a car navigation system 12 , a communication device 13 , a speaker 14 , a vehicle controller 16 and sensors 17 .

The driving operation unit 11 is an operation unit operated by the passenger 10 to operate the vehicle 100 . The driving operation unit 11 is, for example, a steering wheel, an accelerator pedal, a brake pedal, and the like.

The car navigation system 12 is a guidance device that guides the passenger 10 on a route to a destination and the like. The car navigation system 12 has its own display and operation keys. The passenger 10 uses the operation keys of the car navigation system 12 to perform operations necessary for route guidance, such as setting a destination. In addition, the passenger 10 can receive information including route guidance and the like through the display of the car navigation system 12 and/or the sound or voice output from the speaker 14 .

The car navigation system 12 has map information 21, landmark information 22, and the like.

The map information 21 is a database or the like including maps and traffic information associated with the maps. The map information 21 can be appropriately updated by information communication with the outside of the vehicle 100 using the communication device 13, for example.

Landmark information 22 is a database containing information about landmarks. Landmarks are buildings (such as skyscrapers and bridges), structures (such as monuments), natural objects (such as mountains), and spaces or locations (such as parks) that are generally considered to be famous and distinctive. is. For example, famous buildings such as Tokyo Tower (registered trademark) and Sky Tree (registered trademark), and Mt. Fuji are examples of landmarks. In this embodiment, a building or the like registered in advance in the landmark information 22 is treated as a landmark. Landmarks that are actually around the vehicle 100 can be identified according to the position, orientation, and the like of the vehicle 100 . In this embodiment, the vicinity of the vehicle 100 is, for example, a range that can be visually recognized by the passenger 10 regardless of the distance from the vehicle 100. Landmarks around the vehicle 100 A landmark that can be partially or wholly visible. The landmark information 22 can be appropriately updated by information communication with the outside of the vehicle 100 using the communication device 13, for example.

The map information 21 and the landmark information 22 are not only used for route guidance by the car navigation system 12, but can also be used appropriately in each part of the vehicle 100 such as the agent system 15 and the vehicle controller 16. FIG.

The communication device 13 is a communication interface that wirelessly communicates with nearby vehicles, transportation equipment, and/or any other system, server, or the like. The communication device 13 is connected directly or via an Internet line or the like to a device or the like to be communicated, and transmits/receives various kinds of information. In this embodiment, the vehicle 100 has the communication device 13, but the communication device 13 may be provided as part of the car navigation system 12 or the agent system 15. FIG. Regardless of the affiliation of the communication device 13 , each part of the vehicle 100 , that is, the car navigation system 12 , the agent system 15 , the vehicle controller 16 and the like, uses the communication device 13 to appropriately communicate with a server or the like outside the vehicle 100 . can communicate.

The speaker 14 outputs sound, voice, etc. toward the inside of the vehicle 100 . Each part of the vehicle 100 can access the speaker 14 as necessary to output sound or the like. In this embodiment, the car navigation system 12 outputs sound and voice from the speaker 14 for route guidance and the like. Also, the agent system 15 outputs sounds, voices, and the like from the speaker 14 in order to interact with the passenger 10 and provide information to the passenger 10 .

The agent system 15 is a system that supports the operation of the vehicle 100 or various devices and systems mounted on the vehicle 100 and the provision of information that the vehicle 100 should perform. The agent system 15 provides the above support through interactive communication in which information and the like are exchanged interactively with the passenger 10 . In particular, in this embodiment, the agent system 15 provides support for recording a moving image or the like that becomes a memory of a trip or the like at an arbitrary timing. That is, the agent system 15 constitutes an information recording support device that supports information recording at arbitrary timing.

Specifically, the agent system 15 includes an agent section 23 , an agent controller 24 , a display 25 and a recording device 26 .

The agent part 23 is a part that functions as an interface for the passenger 10 . That is, the agent system 15 uses the agent unit 23 to perform interactive communication with the passenger 10 .

The agent unit 23 is, for example, a mascot (also referred to as a mascot character or simply a character) or an assistant represented in the form of a person, animal, plant, object, or other symbol. The agent unit 23 is physically configured using a mechanical mechanism like a robot, or is virtually configured using computer graphics or the like. In this embodiment, the agent section 23 is a virtual configuration displayed on the display 25 .

The agent part 23 is configured in an anthropomorphic manner at least to the extent that the passenger 10 can communicate with it. That is, the agent unit 23 is configured to be able to express its own emotions and emotions, which are changes in emotions, by sounds or voices and/or changes in form. In this embodiment, the shape of the agent part 23 means the overall shape of the agent part 23 (for example, the orientation and balance of the face and limbs) and the partial shape (for example, the shape, orientation, and balance of parts such as eyes and eyebrows). , and pattern or color (eg, complexion), and the like. Therefore, the morphological change is, for example, the action of the face or body of the agent part 23, the change in the line of sight of the agent part 23, the change in the facial expression of the agent part 23, or a combination thereof. If the tone of the voice output by the agent unit 23 can be changed, the change in tone is also included in the above-mentioned form change.

The agent unit 23 displays an object to which the agent unit 23 pays attention (hereinafter referred to as an object of interest) or performs an object-of-interest display operation to indicate that there is an object of interest, by means of a change in form or an alarm such as voice. conduct. In this embodiment, the agent unit 23 can execute the first action, the second action, the third action, or a combination of these actions as the target display action.

The first operation is an operation of displaying the target of interest or indicating that there is the target of interest by changing the form of operating at least part of the agent unit 23 toward the target of interest. For example, an action of performing a gesture according to the presence/absence, position, property, or the like of a target of interest is the first action. More specifically, the first action is an action of directing the line of sight to the target of interest, an action of pointing at the target of interest, an action of directing the face or body to the target of interest, or a combination of these actions.

The second operation is an operation of displaying the target of attention of the agent unit 23 or indicating that there is a target of attention of the agent unit 23 by changing the form of changing the color and/or pattern of the agent unit 23 . . For example, the second action is the action of displaying the feeling or emotion according to the presence/absence, position, or nature of the target of interest. More specifically, the action of exhibiting an expression corresponding to the target of interest is the second action.

The third operation is an operation of displaying a target of attention of the agent unit 23 or displaying that there is a target of attention of the agent unit 23 by the agent unit 23 emitting sound or voice. For example, the action of issuing a message by sound or voice according to the presence/absence, position, property, or the like of the target of interest is the third action.

The agent unit 23 can perform three types of attention target display operations, ie, a first operation, a second operation, and a third operation. In the present embodiment, the object of interest is displayed or the presence of the object of interest is displayed by a composite action combining the first action, the second action, and the third action. However, the agent unit 23 can perform an attention target display operation by arbitrarily combining one or more of the first operation, the second operation, and the third operation.

Agent controller 24 is a control unit that controls the operation of agent system 15 . More specifically, the agent controller 24 is a computer in which information provided by the vehicle 100 to the passenger 10 and the operation of the agent system 15 according to the action of the passenger 10 such as the action of the agent unit 23 are programmed. . A specific configuration of the agent controller 24 will be described in detail later. A program that operates the agent system 15 so as to support the recording of a moving image or the like that becomes a memory of a trip is an information recording support program. The information recording support program can be provided in the form of a recording medium that can be read by agent system 15, vehicle 100, or information recording support system 101. FIG. Furthermore, part or all of the information recording support program can be updated after implementation using a recording medium or by so-called OTA (Over The Air) technology.

The display 25 is a display unit that displays moving images and various information. When the agent section 23 is configured virtually, the agent section 23 is displayed on the display 25 . In addition, the display 25 can display, for example, moving images captured by the vehicle 100 . In this embodiment, the display 25 is a HUD (Head-Up Display) and displays the agent part 23 in a part of the visual field of the passenger 10 . In addition, the passenger 10 can visually recognize the photographing range of an exterior camera 29, which will be described later, through the display 25, which is a HUD. That is, the display 25 displays the shooting range of the exterior camera 29 .

Note that the display 25 is provided as part of the agent system 15 in this embodiment. However, instead of providing the display 25 on the agent system 15 or in addition to the display 25 of the agent system 15 , a display may be provided on the vehicle 100 or the car navigation system 12 . In this case, agent system 15 can display agent section 23 and the like on a display provided in vehicle 100 or car navigation system 12 .

The recording device 26 is a storage that temporarily or permanently records moving images, etc., that serve as memories of trips and the like. The recording device 26 is composed of a memory, a hard disk drive, a solid state drive, cloud storage provided on the web, or the like.

The agent system 15 configured as described above is substantially integrated with the vehicle 100 in this embodiment. Accordingly, some or all of the functionality of agent controller 24 may be configured by vehicle controller 16 . In this case, vehicle controller 16 functions as agent controller 24 . Also, the agent system 15 may be configured by a separate device independent of the vehicle 100 . For example, when a device such as a smartphone owned by passenger 10 can cooperate with vehicle 100 , agent system 15 can be implemented in the device owned by passenger 10 .

The vehicle controller 16 is a control unit that controls the operation of the vehicle 100 in an integrated manner. The vehicle controller 16 controls an electric motor and/or an engine that drives the vehicle 100, a brake, etc., based on an input from the driving operation unit 11. FIG. The vehicle controller 16 is composed of, for example, one or a plurality of computers in which the operation of each component of the vehicle 100 is programmed. The vehicle controller 16 can appropriately refer to signals output by the sensor 17 and the like for controlling the vehicle 100 . In addition, the vehicle controller 16 acquires operations on the vehicle 100 and operations on various devices mounted on the vehicle 100 . In addition, the vehicle controller 16 obtains parameters necessary for controlling the vehicle 100 or various devices mounted on the vehicle 100 by performing calculations using signals obtained from the sensor 17 and the like.

The sensor 17 is one or a plurality of sensors that measure the condition of each part constituting the vehicle 100 and the condition inside or outside the vehicle 100 . The sensor 17 is, for example, a speed sensor, an acceleration sensor, an in-vehicle camera 27, an in-vehicle microphone 28, an outside camera 29, an outside microphone, and/or a position sensor 30 and the like. In this embodiment, the sensor 17 includes at least an in-vehicle camera 27 , an in-vehicle microphone 28 , an exterior camera 29 and a position sensor 30 .

The in-vehicle camera 27 is one or a plurality of photographing devices that photograph the interior of the vehicle 100 and obtain still images or moving images thereof. In this embodiment, the in-vehicle camera 27 captures at least the passenger 10 . Below, the still image or moving image of the passenger 10 captured by the in-vehicle camera 27 is referred to as an in-vehicle image 31 (see FIG. 2). According to the in-vehicle image 31, the line of sight of the occupant 10, the position and orientation of the face, body, etc., and changes in these can be detected.

The in-vehicle camera 27 can acquire the in-vehicle image 31 from time to time in response to a request from the agent system 15 and/or the vehicle controller 16 or the like. In this embodiment, the in-vehicle camera 27 intermittently or continuously acquires the in-vehicle image 31 in response to a request from the agent system 15 . The in-vehicle image 31 is temporarily stored and used in the agent system 15 to detect the line of sight of the passenger 10, the position and orientation of the face and body, and changes in these.

In-vehicle microphone 28 is one or more sound pickup devices that capture sounds or voices generated inside vehicle 100 . In this embodiment, the in-vehicle microphone 28 acquires at least the voice uttered by the passenger 10 . Below, the voice acquired by the in-vehicle microphone 28 is referred to as the in-vehicle voice 32 (see FIG. 2). According to the in-vehicle voice 32, the passenger 10 murmurs (talking to himself/herself), a conversation between a plurality of passengers 10, and a voice for dialogue between the passenger 10 and the agent unit 23 (hereinafter simply referred to as "conversation"). can be obtained.

In-vehicle microphone 28 may acquire in-vehicle audio 32 from time to time upon request from agent system 15 and/or vehicle controller 16 or the like. In this embodiment, the in-vehicle microphone 28 intermittently or continuously acquires the in-vehicle voice 32 in response to a request from the agent system 15 . The in-vehicle voice 32 is temporarily stored and used in the agent system 15 to acquire or detect the conversation of the passenger 10 .

Exterior camera 29 is one or a plurality of imaging devices that capture an image of the exterior of vehicle 100 and obtain a still image or moving image thereof. In this embodiment, the exterior camera 29 captures at least the front of the vehicle 100 . Below, the still image or moving image image|photographed with the camera 29 outside a vehicle is called the image 33 outside a vehicle (refer FIG. 2). According to the image 33 outside the vehicle, an object that the passenger 10 is interested in (hereinafter referred to as an object of interest) can be specified based on changes in the line of sight of the passenger 10 or the like.

The exterior camera 29 can acquire the exterior image 33 as appropriate in response to requests from the agent system 15 and/or the vehicle controller 16 or the like. In this embodiment, the vehicle exterior camera 29 intermittently or continuously acquires the vehicle exterior image 33 in response to a request from the agent system 15 . In principle, the outside image 33 intermittently or continuously acquired in this manner is temporarily retained and used for identifying the object of interest of the passenger 10, for example. In addition, the image 33 outside the vehicle may be recorded in the recording device 26 in order to provide the passenger 10 with memories of the trip or the like. The agent system 15 determines whether the outside image 33 acquired at arbitrary timing is recorded as a memory of travel or the like.

The position sensor 30 is a GPS (Global Positioning System) sensor or other GNN (Global Navigation Satellite System) sensor, and measures the position of the vehicle 100 . The position, orientation, traveling direction, and the like of the vehicle 100 can be specified based on the position information output by the position sensor 30 and its change. The position information output by the position sensor 30 is used by the car navigation system 12 for route guidance and the like. In addition, in this embodiment, the agent system 15 uses the position information output by the position sensor 30 when identifying landmarks around the vehicle 100 based on the position and orientation of the vehicle 100. .

FIG. 2 is a block diagram showing the configuration of the agent controller 24. As shown in FIG. As shown in FIG. 2, the agent controller 24 includes an occupant detection unit 41, an interest target identification unit 42, an interest target recording unit 43, an interface control unit 44, a loss of interest determination unit 45, and a shared recognition determination unit 46. .

The occupant detection unit 41 detects the occupant 10 using the in-vehicle image 31 and/or the in-vehicle audio 32 . The detection of the passenger 10 means detecting partial or total behavior of the passenger 10 , and includes not only the movement of the passenger 10 itself but also the voice uttered by the passenger 10 .

Specifically, the passenger detection unit 41 includes a motion detection unit 51 and a conversation detection unit 52 .

The motion detection unit 51 detects the motion of the passenger 10 using the in-vehicle image 31 . That is, the occupant detection unit 41 detects the line of sight of the occupant 10, the facial expression of the occupant 10 such as the position and orientation of the face and/or body of the occupant 10, and changes in these. The line of sight of the passenger 10 is detected, for example, based on the movement of the eyeballs of the passenger 10 . Moreover, the facial expression of the passenger 10 is detected based on the movements of the parts that make up the face of the passenger 10, such as the eyes, nose, mouth, eyebrows, and cheeks of the passenger 10. The facial and other bodily movements (including gestures) of the passenger 10 are detected based on changes in their appearance.

The conversation detection unit 52 uses the in-vehicle voice 32 to detect the conversation of the passenger 10 and its transition (change). For example, the conversation detection unit 52 detects the conversation of the passenger 10 by detecting predetermined keywords and/or phrases from the voice (voice data) of the passenger 10 acquired as the in-vehicle voice 32. . Further, the conversation detection unit 52 detects conversation topics and their transitions based on the classification and frequency of appearance of keywords and phrases. The topic here is the theme, object, etc. of the conversation.

Based on the outside image 33 and the detection result of the passenger 10 in the passenger detection unit 41, the target-of-interest identification unit 42 identifies a target to which the passenger 10 pays attention and/or shows interest (hereinafter referred to as , called the object of interest). That is, the subject-of-interest identifying unit 42 identifies the subject of interest of the passenger 10 from landmarks, other structures, natural objects, etc. outside the vehicle 100 based on changes in the line of sight of the passenger 10, conversations, and the like. do. The target-of-interest identification unit 42 also acquires information and the like related to the specified target-of-interest.

Specifically, the target-of-interest identification unit 42 includes an object detection unit 53 , an interest level estimation unit 54 , and a related information acquisition unit 55 .

The object detection unit 53 detects an object, which is one or a plurality of objects of interest, such as landmarks, other structures, or natural objects, from the image 33 outside the vehicle. The object detection unit 53 can detect various objects including objects other than predetermined landmarks. However, in this embodiment, for simplicity, the object detection unit 53 detects landmarks. Specifically, the object detection unit 53 acquires the landmark information 22 based on the position information of the vehicle 100, and identifies landmarks that can be visually recognized by the passenger 10 from the current position of the vehicle 100 as objects of possible interest. To detect.

The degree-of-interest estimation unit 54 estimates the degree of interest in the object detected by the object detection unit 53 based on the line of sight, face or other body of the occupant 10, or the topic of conversation, or based on a predetermined calculation or the like. presume. The subject-of-interest identification unit 42 determines the subject of interest of the passenger 10 from one or more detected objects based on the degree of interest estimated by the degree-of-interest estimation unit 54 . More specifically, the subject-of-interest identifying unit 42 determines that an object is the subject of interest of the passenger 10 when the degree of interest in the object is greater than a predetermined threshold value. That is, in the present embodiment, an object whose degree of interest is greater than the threshold is recognized as an object of interest to the passenger 10 .

In this embodiment, the degree of interest is an evaluation value representing the degree of interest of the passenger 10 in the object detected by the object detection unit 53 . For example, the time during which the passenger's 10 line of sight transitions on a specific object, or the time during which the passenger's 10 face or other body is directed toward the object detected by the object detection unit 53 represents the degree of interest. Also, the degree of matching or the degree of association with the topic of conversation represents the degree of interest. In principle, the interest level estimating unit 54 estimates the interest level for a specific object by calculation using the time when the line of sight, face, etc. of the passenger 10 is directed, the degree of relevance to the topic of conversation, and the like. In this embodiment, for the sake of simplicity, the interest degree estimation unit 54 determines the time when the line of sight of the passenger 10 or the direction of the face or body changes continuously or intermittently on a specific object for a predetermined time. Compare with Then, the interest level estimation unit 54 determines that an object to which the line of sight of the passenger 10 is directed for a predetermined time or longer is an object of interest to the passenger 10 .

The related information acquisition unit 55 acquires information related to the target of interest (hereinafter referred to as related information) when the target of interest is specified. For example, when the object of interest is a mountain, the related information includes the name and altitude of the mountain. Also, for example, when the object of interest is a bridge, the related information includes the name, length, and year of construction of the bridge. The related information acquisition unit 55 acquires related information from the Internet or the like using the communication device 13, for example.

When the target of interest is specified, the target of interest recording unit 43 causes the agent unit 23 to notify the passenger 10 that the target of interest has been specified, and the specified target of interest is photographed by the outside camera 29. Information recording support control for automatically recording the outside image 33 in the recording device 26 is executed. That is, in principle, the subject-of-interest recording unit 43 automatically records the image 33 outside the vehicle in which the identified subject-of-interest is captured. is notified to the passenger 10 by the agent unit 23. This allows the passenger 10 to know that the agent system 15 has specified the object of interest and that the outside image 33 showing the object of interest will be automatically recorded.

As described above, at least informing the passenger 10 that the target of interest has been identified is the minimum condition for the target of interest recording unit 43 to automatically record the outside image 33 showing the target of interest. . In addition, in the present embodiment, the subject-of-interest recording unit 43 further confirms that the passenger 10 has recognized that the agent system 15 has specified the subject-of-interest and automatically recorded it. condition.

Further, when recording the identified target of interest as a moving image, the target of interest recording unit 43 ends recording when the interest of the passenger 10 is lost. It should be noted that when recording the specified target of interest as a still image, the recording ends almost simultaneously with the start of the recording. Therefore, regardless of the loss of interest of the passenger 10 as described above, the recording is automatically terminated.

In addition, the subject-of-interest recording unit 43 can transmit information such as the automatically recorded outside image 33 to an external information device that cooperates with the vehicle 100 or the agent system 15 using the communication device 13 . The external information device here is a device that can display or reproduce the image 33 outside the vehicle or the like, and is, for example, a device such as a smartphone owned by the passenger 10, or a device that can be accessed from a device owned by the passenger 10. Cloud storage and the like. When devices such as the car navigation system 12 mounted on the vehicle 100 can display or reproduce the image 33 outside the vehicle, these devices are also included in the external information device. In order to allow the outside image 33 to be displayed or reproduced without depending on the vehicle 100 or the agent system 15, the subject-of-interest recording unit 43, for example, when the passenger 10 gets off the vehicle 100 or has got off the vehicle, The recorded outside image 33 and the like are transmitted to the external information device.

The interface control unit 44 provides the support function of the agent system 15 to the passenger 10 in a natural form by controlling the interface of the agent system 15 to the passenger 10 . Specifically, the interface control unit 44 comprehensively controls various operations to be performed by the agent unit 23, such as display, movement, expression, and voice output of the agent unit 23, and the display of the display 25 and the like. In particular, in the present embodiment, the interface control unit 44 acquires information related to the position of the identified target of interest, etc., from the target of interest identification unit 42 , for example, in response to a command or notification from the target of interest recording unit 43 . Then, the interface control unit 44 causes the agent unit 23 to perform an attention object display operation of displaying the specified interest object as the attention object of the agent unit 23, and/or displays the specified interest object on the display 25. emphasize in

The interface control unit 44 includes a first operation control unit 56 , a second operation control unit 57 , a third operation control unit 58 and a highlighting control unit 59 .

The first operation control unit 56 causes the agent unit 23 to perform an attention object display operation of displaying the attention object of the agent unit 23 by the first operation of operating at least a part of the agent unit 23 . In the present embodiment, the first operation control unit 56 causes the agent unit 23 to perform a first operation such as displaying the identified target of interest as the target of interest of the agent unit 23 . That is, when a target of interest is specified and the target of interest recording unit 43 automatically records the target of interest, the agent unit 23 is made to perform the first operation. Accordingly, the fact that the object of interest of the passenger 10 has been shared with the agent unit 23 is notified to the passenger 10 by, for example, a gesture of the agent unit 23 or the like.

The second operation control unit 57 causes the agent unit 23 to perform an attention object display operation of displaying the attention object of the agent unit 23 by the second operation of changing the color and/or pattern of the agent unit 23 . In the present embodiment, the second operation control unit 57 causes the agent unit 23 to perform a second operation such as displaying the specified target of interest as the target of interest of the agent unit 23 . That is, at least when a target of interest is specified and the target of interest recording unit 43 automatically records the target of interest, the second operation control unit 57 causes the agent unit 23 to perform the second operation. As a result, the fact that the object of interest of the passenger 10 has been shared with the agent unit 23 is notified to the passenger 10 by, for example, the facial expression of the agent unit 23 .

The third operation control unit 58 causes the agent unit 23 to perform an attention object display operation of displaying the attention object of the agent unit 23 by the third operation of causing the agent unit 23 to output sound, voice, or the like. In the present embodiment, the third operation control unit 58 causes the agent unit 23 to perform a third operation such as displaying the identified target of interest as the target of interest of the agent unit 23 . That is, at least when a target of interest is specified and the target of interest recording unit 43 automatically records the target of interest, the third operation control unit 58 causes the agent unit 23 to perform the third operation. As a result, the fact that the object of interest of the passenger 10 has been shared with the agent unit 23 is notified to the passenger 10 by, for example, a direct or indirect (metaphorical) voice message issued by the agent unit 23 .

Further, when the interested object recording unit 43 starts recording the interested object, the third operation control unit 58 notifies the passenger 10 of this fact by causing the agent unit 23 to perform the third operation. can be done. Similarly, when the interested object recording unit 43 finishes recording the interested object, the third operation control unit 58 notifies the passenger 10 of this fact by causing the agent unit 23 to perform the third operation. can be done. In this embodiment, the third operation control unit 58 causes the agent unit 23 to output a voice message to that effect when the interested object recording unit 43 starts and ends the recording of the interested object, thereby notifying the passenger 10. inform.

In addition, after the interest target recording unit 43 starts recording the interest target, the third operation control unit 58 causes the agent unit 23 to perform the third operation, thereby providing the passenger 10 with the relevant information acquisition unit 55 can provide relevant information obtained. In this embodiment, the third operation control unit 58 provides the relevant information as described above by the agent unit 23 .

Emphasis display control unit 59 emphasizes part or all of the display on display 25 by adjusting the display form on display 25 . In this embodiment, the highlighting control unit 59 highlights the specified target of interest on the display 25 . That is, at least when a target of interest is identified and the target of interest recording unit 43 automatically records the target of interest, the highlighting control unit 59 displays the identified target of interest, such as by framing the target of interest with a frame. Highlight. Accordingly, the object of interest of the passenger 10 specified by the agent unit 23 or the agent system 15 is notified to the passenger 10 by the display on the display 25 .

In the present embodiment, as described above, the interface control unit 44 executes the first action, the second action, and the third action of the agent unit 23 in combination with the highlighting on the display 25 . However, the interface control unit 44 can display that the agent unit 23 or the agent system 15 has identified the object of interest of the passenger 10 by one or any combination of these operations.

The loss of interest determination unit 45 continuously monitors the detection result of the passenger detection unit 41 and/or the target of interest identified by the target of interest identification unit 42, thereby detecting the interest identified by the target of interest identification unit 42. A determination is made as to whether the passenger 10 has lost interest in the object. In the present embodiment, the result of the determination by the interest loss determination unit 45 (hereinafter referred to as interest loss determination) is one of the triggers for the interest object recording unit 43 to end recording the outside image 33 of the interest object. Configure. That is, when the loss of interest determining unit 45 determines that the passenger 10 has lost interest in the identified target of interest, the target of interest recording unit 43 terminates continuous recording of the outside image 33 of the target of interest. do.

For example, when the topic (content) of the conversation by the passenger 10 transitions from a topic related to the target of interest to a topic not related to the target of interest, the interest loss determination unit 45 determines whether the interest of the passenger 10 in the identified target of interest is changed. can be determined to be lost. In addition, the interest loss determination unit 45 determines that the passenger 10 has lost interest in the previous interest target when the interest target of the passenger 10 is changed, for example, when a new interest target is specified. can do. In this embodiment, when the subject of the conversation changes, and the passenger 10 is not looking at the object of interest and the interest level decreases, and the object is no longer the object of interest, Determine that interest has been lost. As described above, the topic of conversation by the passenger 10 is estimated, for example, by the conversation detection unit 52 based on the keywords and phrases that appear in the conversation, their classification, frequency of appearance, and the like.

Here, the interested subject recording unit 43 automatically terminates the recording of the interested subject based on the result of the interest loss determination, but the present invention is not limited to this. For example, after starting recording of the specified target of interest, the interest recording unit 43 does not perform the loss of interest determination, or regardless of the result of the loss of interest determination, the target of interest recording unit 43 records the Recording can be terminated for the subject of interest. In this embodiment, when the passenger 10 loses interest in the target of interest and a certain period of time has elapsed since the start of recording, the target of interest recording unit 43 ends recording of the target of interest. Further, when the subject-of-interest recording unit 43 records the subject-of-interest as a still image based on the setting of the passenger 10, the recording is substantially automatically ended at the same time as the start. In the present embodiment, the target-of-interest recording unit 43 records the identified target-of-interest as a moving image.

The shared recognition determination unit 46 monitors the detection result of the passenger detection unit 41 after the passenger 10 is notified that the target of interest has been specified by at least the display on the agent unit 23 and/or the display 25 . Then, the shared recognition determination unit 46 determines whether or not the passenger 10 has recognized that the agent unit 23 or the agent system 15 has identified the object of interest of the passenger 10 . This determination by the shared recognition determination unit 46 (hereinafter referred to as shared recognition determination) determines whether or not the passenger 10 has recognized that the object of interest of the passenger 10 has been shared with the agent unit 23 or the agent system 15. is synonymous with Also, the passenger 10 usually recognizes the information recording support function of the agent system 15 . Thus, the shared awareness determination is essentially synonymous with determining the passenger's 10 willingness to agree or disagree with the automatic recording of information about the subject of interest by the agent system 15 .

In the present embodiment, the result of the sharing recognition determination constitutes one of triggers for the interested subject recording unit 43 to start recording the outside image 33 of the interested subject. That is, in the present embodiment, when the passenger 10 recognizes that the target of interest has been specified and that the target of interest has been shared with the agent unit 23 or the agent system 15, the target of interest recording unit 43 stores the specified target of interest. Recording of the vehicle exterior image 33 is started for the object of interest.

In the following, the operation of automatic information recording support in the vehicle 100 equipped with the agent system 15 configured as described above will be described. The information recording support program is a program that implements the processing for information recording support shown in FIGS. 3 to 5 below.

FIG. 3 is a flow chart for starting automatic information recording. As shown in FIG. 3, in step S101, the behavior of the passenger 10, that is, the gaze of the passenger 10, facial or other bodily movements, facial expression, and/or conversation, etc. are detected. In step S102, an object that may be of interest to the passenger 10 is detected. In this embodiment, a landmark that can be visually recognized by the passenger 10 is detected as an object that can be of interest. Then, in step S103, the degree of interest in the detected object is estimated based on the detected behavior of the passenger 10, and the estimated degree of interest is compared with a threshold. If the degree of interest is greater than the threshold, the object is recognized as being of interest to passenger 10 . It should be noted that the agent system 15 continues monitoring the behavior of the passenger 10 without selfishly recording the outside image 33 when the object of interest of the passenger 10 does not occur. On the other hand, when a subject of interest for passenger 10 is identified, agent system 15 assists in recording the subject of interest.

When a target of interest whose degree of interest is greater than the threshold is identified, the agent unit 23 performs an attention target display operation of displaying the interest target of the passenger 10 as the attention target of the agent unit 23 in step S<b>104 . Further, in step S105, the specified target of interest is highlighted in the display on the display 25. FIG. Through these operations, the passenger 10 is notified that the object of interest of the passenger 10 has been specified by the agent unit 23 or the like. As a result, the passenger 10 can recognize that the subject of interest is shared with the agent unit 23 or the like.

After that, in step S107, sharing recognition determination is performed. In this embodiment, if the passenger 10 does not recognize that the target of interest has been shared with the agent unit 23 or the like as a result of the sharing recognition determination, the recording of the outside image 33 of the target of interest is stopped. Here, the agent system 15 does not start recording for the subject of interest and ends information recording support for the subject of interest.

On the other hand, if it is determined in the sharing recognition determination that the passenger 10 has recognized that the subject of interest is shared with the agent unit 23 or the like, recording of the outside image 33 of the subject of interest is started in step S108. If the occupant 10 is aware of the sharing of the subject of interest, the occupant's 10 knowledge of the automatic recording of the subject of interest, i.e., substantial consent to the recording based on the functional understanding of the agent system 15 is obtained. recording of the object of interest is started.

When the recording of the target of interest is started, the start of recording is notified interactively by a voice message from the agent unit 23 in step S109. Also, when there is related information about the target of interest, the related information is provided by a voice message from the agent unit 23 in step S110.

FIG. 4 is a flow chart for ending automatic information recording. As shown in FIG. 4, when recording of an object of interest is started, interest loss determination is performed in step S201. As a result of the interest loss determination, if the passenger 10 has not lost interest in the object of interest, that is, if the interest in the object of interest related to the record continues, as shown in step S202, in principle, the object of interest is record continues.

On the other hand, when it is determined in the loss of interest determination that the passenger 10 has lost interest in the target of interest, it is determined in step S203 whether or not a certain period of time has elapsed since the start of recording of the target of interest. be done. When the passenger 10 loses interest in the recorded object of interest and a certain period of time has passed, the recording of the object of interest is ended in step S204. Also, when the recording of the subject of interest is finished, the end of the recording is notified interactively by a voice message from the agent unit 23 in step S205.

FIG. 5 is a flowchart showing a specific example of interest loss determination. As shown in FIG. 5, for example, in step S301, the interest loss determination is based on the detection result of the conversation detection unit 52 that is continuously monitored after the start of recording of the interest target, and the passenger 10 talks. is checked. Then, if the passenger 10 is having a conversation, it is checked in step S302 whether or not the conversation related to the subject of interest continues with respect to the topic of the conversation. When the topic related to the subject of interest continues, it can be assumed that the occupant 10 has not lost interest in the subject of interest for which recording was initiated, ie maintains interest in the subject of interest. Therefore, the process proceeds to step S202, where recording of the object of interest continues.

On the other hand, if it is determined in step S301 that the passenger 10 is not speaking, it can be assumed that the passenger 10 has already lost interest in the subject of interest for which recording has started. Similarly, although the passenger 10 is having a conversation, if it is determined in step S302 that the topic of the conversation has changed to a topic that is not related to the subject of interest, the passenger 10 can It can be assumed that interest has already been lost. Therefore, in these cases, the passenger 10 loses interest in the interest target for which recording is started based on the detection result of the motion detection unit 51 and/or the degree of interest estimated by the interest degree estimation unit 54. It is further checked whether or not

Here, in step S303, it is determined whether or not the passenger 10 is continuously or intermittently looking at the target of interest based on changes in the line of sight of the passenger 10 which is continuously monitored after the recording of the target of interest is started. It is confirmed. Recording was started when the passenger 10 was continuously or intermittently viewing an object of interest, even if the passenger 10 was not speaking or the topic had already transitioned to an irrelevant topic. It can be estimated that the passenger's 10 interest in the subject of interest has not been lost. Therefore, when it is determined in step S303 that the passenger 10 is visually recognizing the object of interest, the process proceeds to step S203, and after confirming that a certain period of time has elapsed, the recording of the object of interest is terminated. On the other hand, when the passenger 10 is not having a conversation, or when the topic has already changed to an irrelevant topic, and when it is confirmed that the passenger 10 does not visually recognize the target of interest, the recording It can be assumed that the occupant's 10 interest in the subject of interest that initiated the process has been lost. Therefore, when it is determined in step S303 that the passenger 10 has not visually recognized the object of interest, the process proceeds to step S204, and recording of the object of interest ends. In this way, the determination of loss of interest is made, for example, by comprehensive evaluation according to the presence or absence of conversation, the transition of topics, and the presence or absence of visual recognition of the target of interest.

A more specific application example of the information recording support will be described below.

FIG. 6 is an explanatory diagram of a scene in which automatic information recording is started. FIG. 6(A) is an explanatory diagram of a scene S1 in which an object of interest of the passenger 10 is specified. FIG. 6B is an explanatory diagram of a scene S2 in which the agent unit 23 performs an attention target display operation for the specified interest target. FIG. 6C is an explanatory diagram of a scene S3 in which sharing recognition determination is performed and recording of the target of interest is actually started. FIG. 6D is an explanatory diagram of a scene S4 for notifying the start of recording.

As shown in FIG. 6A, it is assumed that there is a mountain 62 as a landmark in the field of view 61 of the passenger 10 . When the passenger 10 visually recognizes this for a predetermined time or longer, the agent system 15 identifies the mountain 62 as the object of interest of the passenger 10 . In this case, as shown in FIG. 6B, for example, the agent unit 23 faces the direction of the mountain 62, directs its line of sight to the mountain 62, and sends a voice message suggesting recognition of the mountain 62, such as "It's beautiful, isn't it?" emitted. In addition, a frame line 63 surrounding the mountain 62 is displayed in the display area of the display 25 superimposed on the field of view of the passenger 10, and the mountain 62 is emphasized as the identified target of interest. Therefore, the passenger 10 recognizes that the object of interest of the passenger 10 has been specified by the operation of the agent unit 23 and the highlighting on the display 25 . Mountain 62 is shared as a common object of interest between passenger 10 and agent unit 23 . As a result, the mountain 62 becomes a shared memory between the passenger 10 and the agent unit 23 . Therefore, even if the outside image 33 showing the mountains 62 is provided to the passenger 10 after that, the passenger 10 does not perceive this as a record made by the agent system 15, but naturally feels uncomfortable as a shared memory. can be accepted without

As shown in FIG. 6(C), the passenger 10 responds to the voice message such as "You're beautiful" from the agent unit 23 by saying, for example, "That's right." Upon acknowledgment, recording of the mountain 62 of shared interest is initiated. Then, as shown in FIG. 6(D), the agent unit 23 issues a voice message directly or indirectly indicating the start of recording, such as "Remember," for example, and tells the passenger 10 to start recording. inform. This may particularly ensure that the passenger 10 is aware that the mountain 62 of shared interest is being recorded.

FIG. 7 is an explanatory diagram of a scene during automatic information recording. FIG. 7A is an explanatory diagram of a scene S5 in which the agent section 23 and the passenger 10 interact. FIG. 7B is an explanatory diagram of a scene S6 that provides related information.

As shown in FIG. 7(A), after the start of recording of the mountain 62 which is the object of shared interest, for example, when the passenger 10 says "I want to climb" the mountain 62, the following is shown in FIG. 7(B). In response to this, the agent unit 23 interactively provides the related information of the mountain 62 by issuing a voice message such as "that is a mountain called XX and its altitude is XXXm". This gives the occupant 10 a particular impression that the mountain 62 is of shared interest. As a result, the passenger 10 can naturally accept the image 33 outside the vehicle showing the mountain 62 as a shared memory without any sense of incongruity.

FIG. 8 is an explanatory diagram of a scene in which automatic information recording ends. A scene S7 shown in FIG. 8 is a scene in which the agent unit 23 notifies the passenger 10 of the end of recording.

As shown in FIG. 8, the interest of the passenger 10 in the subject of interest is lost, including the case where the interest is passively lost, such as when the mountain 62 of interest is out of the field of view 61, for example. recording is terminated. When the recording ends, the agent unit 23 notifies the passenger 10 of the end of the recording by issuing a voice message directly or indirectly indicating the end of the recording, such as "I remembered". This may particularly ensure that the passenger 10 is aware that the recording for the mountain 62 of shared interest has ended.

As described above, the information recording support method according to the present embodiment has the vehicle exterior camera 29 that captures the outside of the vehicle and the agent unit 23 that functions as an interface for the passenger 10 . and an agent system 15 that is an information recording support device that supports information recording at any timing using the vehicle 100. In this information recording support method, the movement and/or conversation of the passenger 10 is detected, and the object of interest of the passenger 10 outside the vehicle 100 is identified based on the movement and/or conversation of the passenger 10. . When the object of interest is identified, the agent system 15 notifies the passenger 10 of the identification of the object of interest, and the camera 29 outside the vehicle records the object of interest.

As described above, in the information recording support method according to the present embodiment, at an arbitrary timing when the object of interest of the passenger 10 occurs, the object of interest of the passenger 10 is specified when supporting the recording of the information (moving image, etc.). Passenger 10 is informed that it has been done. When an object of interest to the passenger 10 occurs and the agent unit 23 records the information without showing any reaction, the passenger 10 has no awareness of being provided with the recorded information as a memory. , you may feel uncomfortable. On the other hand, in the information recording support method according to the present embodiment, as described above, it is notified that the target of interest of the passenger 10 has been specified, so that the passenger 10 can use the agent unit 23 or the agent system 15, Recognize shared interests. Therefore, the passenger 10 can recognize the recorded information about the object of interest as memories shared with the agent unit 23 or the agent system 15 . As a result, according to the information recording assistance method according to the present embodiment, information recording can be assisted very naturally without causing the passenger 10 to feel uncomfortable.

In addition, if the agent unit 23 records the information without showing any reaction when the object of interest of the passenger 10 occurs, the trust of the passenger 10 in the agent unit 23 or the agent system 15 may be lowered. be. For example, if the agent unit 23 takes a record of an object of interest without showing any reaction and provides the record, the passenger 10 is selfishly recorded private information of the passenger 10 without the passenger 10's knowledge. may perceive that In other words, the non-responsive recording has low transparency regarding recording and recorded information, so the agent unit 23 or the agent system 15, which is the target of interactive communication, may lose the confidence of the passenger 10. be. On the other hand, in the information recording support method according to the present embodiment, as described above, when the target of interest is recorded, the passenger 10 is notified that the target of interest has been identified. Be transparent about what you record and what you are interested in recording for. Therefore, the information recording support method according to the present embodiment also has a secondary effect of maintaining or improving the confidence of the passenger 10 in the agent unit 23 or the agent system 15, which is the object of interactive communication.

The information recording support method according to the present embodiment has the following effects with regard to a specific aspect of informing the passenger 10 that the target of interest has been specified.

In particular, the information recording support method according to the present embodiment executes a target-of-interest display operation for displaying that the attention of the agent unit 23 has been directed to the target of interest by changing the form of the agent unit 23, so that the target-of-interest is displayed. is notified to the passenger 10 that the This makes it possible to particularly effectively make the passenger 10 recognize and impress that the subject of interest is shared between the agent unit 23 or the agent system 15 and the passenger 10 . As a result, the information support method according to this embodiment can support the recording of the object of interest particularly naturally.

In the information recording support method according to the present embodiment, the object-of-interest display operation is performed by the first operation of operating at least part of the agent unit 23 toward the object of interest. As a result, the passenger 10 almost unconsciously recognizes that the subject of interest has been shared between the agent unit 23 or the agent system 15 and the passenger 10, making it easier to make a strong impression. As a result, the information support method according to this embodiment can support the recording of the object of interest particularly naturally.

In the information recording support method according to the present embodiment, the attention target display operation is performed by the second operation of changing the color and/or pattern of the agent section 23 . As a result, feelings or emotions are shared between the agent unit 23 or the agent system 15 and the passenger 10 regarding the sharing of the subject of interest. Therefore, the information recording assistance method according to the present embodiment is particularly easy to impress on the passenger 10 that the object of interest is shared. As a result, the information support method according to this embodiment can support the recording of the object of interest particularly naturally.

In this embodiment, the agent system 15, which is an information recording support device, has a display 25 that displays the shooting range of the vehicle exterior camera 29. FIG. Then, in the information recording support method according to the present embodiment, by highlighting the target of interest on the display 25, the passenger 10 is notified that the target of interest has been specified. As a result, the passenger 10 is informed particularly clearly and intuitively that the target of interest of the agent unit 23 and the agent system 15 is the same as the target of interest of the passenger 10 . Therefore, the passenger 10 can almost certainly recognize that the agent unit 23 and the agent system 15 have shared the subject of interest without overlooking. As a result, in the information support method according to the present embodiment, it is possible to naturally support the recording of the target of interest without the fact that the target of interest has been recorded without the passenger 10 noticing.

The information recording support method according to the present embodiment has the following effects with respect to a specific mode for identifying the object of interest of the passenger 10 .

In the present embodiment, the vehicle 100 is provided with an in-vehicle camera 27 for photographing the passenger 10, and in the information recording support method according to the present embodiment, the motion of the passenger 10 is detected using the in-vehicle camera 27, and the detected boarding Information recording support is performed based on the actions of the person 10 . The behavior of the passenger 10, including the case where the passenger 10 is unconscious, vividly represents the object of interest of the passenger 10. Therefore, according to the information recording support method according to the present embodiment, the object of interest of the passenger 10 can be accurately recognized, so that the recording of the object of interest can be appropriately supported in a timely manner.

In particular, in the information recording support method according to the present embodiment, as the movement of the passenger 10, a change in the line of sight of the passenger 10 or a movement of the face or body of the passenger 10 is detected. These actions of the passenger 10 tend to show the occurrence of the object of interest particularly vividly. Therefore, according to the information recording support method according to the present embodiment, the object of interest of the passenger 10 is particularly accurately recognized, so that the recording of the object of interest can be timely and accurately supported.

Furthermore, in the information recording support method according to the present embodiment, an object to which the line of sight of the passenger 10 or the direction of the face or body of the passenger 10 is directed for a predetermined time or longer is specified as a target of interest. As a result, an object that is temporarily visually recognized by accident is not recognized as a target of interest, and the accuracy of the target of interest recognized by the agent system 15 is high. Therefore, according to the information recording support method according to the present embodiment, the object of interest of the passenger 10 is particularly accurately recognized, so that the recording of the object of interest can be timely and accurately supported.

Further, in the present embodiment, the vehicle 100 is provided with an in-vehicle microphone 28 that acquires the voice inside the vehicle, and in the information recording support method according to the present embodiment, the conversation of the passenger 10 is detected using the in-vehicle microphone 28, Information recording support is performed based on the detected conversation of the passenger 10 . As a result, even when the passenger 10 cannot take his/her eyes off the traffic situation, etc., the object of interest of the passenger 10 can be accurately identified and information about the object of interest can be recorded. In such a case, the need for information recording support is particularly high. Therefore, according to the information recording support method according to the present embodiment, the subject of interest of the passenger 10 can be accurately recognized, and the recording of the subject of interest can be appropriately supported in a timely manner.

In particular, in the information recording support method according to the present embodiment, the topic of the conversation of the passenger 10 is estimated, and the target of interest is specified according to the topic. Thus, the topic of conversation by passenger 10 is a true representation of the passenger's 10 current interests. Therefore, when an object corresponding to a topic is specified as a target of interest, the accuracy of the target of interest recognized by the agent system 15 is high. Therefore, according to the information recording support method according to the present embodiment, the object of interest of the passenger 10 is particularly accurately recognized, so that the recording of the object of interest can be timely and accurately supported.

In addition, in the information recording support method according to the present embodiment, landmarks around the vehicle 100 are extracted based on the position and orientation of the vehicle 100, and extracted based on the movement and/or conversation of the passenger 10. An object of interest is identified from among the identified landmarks. Landmarks are objects that are particularly likely to be of interest. Therefore, by selecting the target of interest from the landmarks around the vehicle 100 as described above, the target of interest can be easily and accurately identified. Therefore, according to the information recording assistance method according to the present embodiment, the recording of the target of interest can be accurately assisted.

Further, in the information recording support method according to the present embodiment, an interest level representing the degree of interest of the passenger 10 in an object that may be of interest is estimated based on the action and/or conversation of the passenger 10 . Then, when the degree of interest is greater than a predetermined threshold value, the object is specified as a target of interest. In this way, when the subject of interest is specified according to the degree of interest, which is a comprehensive evaluation based on the actions and/or conversations of the passenger 10, the subject of interest of the passenger 10 is particularly accurately specified. As a result, according to the information recording support method according to the present embodiment, the recording of the target of interest can be accurately supported.

The information recording support method according to the present embodiment has the following effects with respect to specific aspects of recording start conditions.

In the information recording support method according to the present embodiment, whether or not the passenger 10 has recognized that the agent system 15, which is the information recording support device, has identified the target of interest based on the actions and/or conversations of the passenger 10. is judged. Recording of the object of interest is started when the passenger 10 recognizes that the agent system 15 has identified the object of interest. That is, in the information recording support method according to the present embodiment, as a result of the sharing recognition determination, the passenger 10 recognizes that the subject of interest of the passenger 10 has been shared by the agent unit 23 or the agent system 15, and records it. Recording of the subject of interest is assisted when the occupant 10 has likely consented. Therefore, the subject of interest is not recorded without the passenger's 10 knowledge. As a result, the information support method according to this embodiment can support the recording of the object of interest particularly naturally.

The information recording support method according to the present embodiment has the following effects with respect to specific aspects regarding the conditions for ending recording.

In the information recording support method according to the present embodiment, it is determined whether or not the passenger 10 has lost interest in the subject of interest based on the actions and/or conversations of the passenger 10 . Recording of the object of interest is terminated when the passenger's 10 interest in the object of interest is lost. That is, in the information recording support method according to the present embodiment, when the loss of interest determination result indicates that there is a high probability that the passenger 10 has lost interest in the target of interest, recording of the target of interest is terminated. Therefore, for the purpose of recording memories shared with the agent unit 23, the unnaturalness of unnecessarily lengthy recording until after the interest of the object is lost does not occur. As a result, the information support method according to the present embodiment can support particularly natural recording of objects of interest.

Specifically, in the present embodiment, the vehicle 100 is provided with an in-vehicle microphone 28 that acquires the voice inside the vehicle. is obtained. Then, a topic in the conversation of the passenger 10 is estimated, and when the estimated topic transitions from a topic related to the subject of interest to a topic not related to the subject of interest, it is determined that the interest of the passenger 10 in the subject of interest has been lost. and the recording for the object of interest is terminated. As a result, the recording is appropriately terminated according to the transition of the topic. As a result, the information support method according to the present embodiment can support particularly natural recording of objects of interest.

In addition, in the information recording support method according to the present embodiment, it is determined that the passenger 10 has lost interest in the target of interest, particularly when the target of interest is changed, and recording of the target of interest is terminated. Thereby, when there is a change in the object of interest, the recording is appropriately terminated according to the change in the object of interest. As a result, the information support method according to the present embodiment can support particularly natural recording of objects of interest.

In addition, in the information recording support method according to the present embodiment, the recording of the target of interest is terminated when a predetermined period of time elapses after the recording of the target of interest is started. As a result, when a specific object enters the field of view 61 for a long time even though the user is not interested in it, such as during traffic jams, there is no problem that the specific object is recorded extensively. Therefore, the information support method according to the present embodiment can support natural recording of objects of interest.

The information recording support method according to the present embodiment has the following effects with respect to specific aspects regarding the operation of the agent unit 23 and the like.

In the information recording support method according to the present embodiment, when the recording of the target of interest is started, the agent unit 23 notifies the passenger 10 of the start of recording of the target of interest. This allows for greater transparency about what is being recorded and what is being recorded. Therefore, passenger 10's trust in agent unit 23 or agent system 15, which is the object of interactive communication, is improved. As a result, information recording assistance by the information recording assistance method according to the present embodiment is likely to be accepted by the passenger 10 as a natural thing.

Similarly, in the information recording support method according to the present embodiment, when the recording of the target of interest ends, the agent unit 23 notifies the passenger 10 that the recording of the target of interest is finished. This provides greater transparency about what was being recorded about the subject of interest. Therefore, passenger 10's trust in agent unit 23 or agent system 15, which is the object of interactive communication, is improved. As a result, information recording assistance by the information recording assistance method according to the present embodiment is likely to be accepted by the passenger 10 as a natural thing.

In addition, in the information recording support method according to the present embodiment, after the recording of the target of interest is started, information related to the target of interest is provided to the passenger 10 by the agent unit 23 . This increases the commonality of the recorded subject of interest between the agent unit 23 and the passenger 10 . As a result, information recording assistance by the information recording assistance method according to the present embodiment is likely to be accepted by the passenger 10 as a natural thing.

The information recording support method according to the present embodiment has the following effects with respect to specific aspects regarding the recorded matter of interest.

In the information recording support method according to the present embodiment, a record of an object of interest is transmitted to an external information device that cooperates with the vehicle 100 or the agent system 15, which is an information recording support device. This improves the convenience of recordings saved as memories, such as the ability to display or play recordings of interest independent of vehicle 100 or agent system 15 .

[Modification]
In the above-described embodiment, the result of the shared recognition determination is regarded as the manifestation of consent or disapproval of the passenger 10 with respect to the recording assistance of the target of interest, but the present invention is not limited to this. For example, recording of a subject of interest may begin when more explicit consent is obtained from the passenger 10 to record the subject of interest. In this case, the agent unit 23 clearly informs the subject of interest by a voice message or the like when recording is started. Then, the subject-of-interest recording unit 43 determines whether or not the passenger 10 agrees or disagrees with the recording of the subject-of-interest in accordance with the response of the passenger 10 to actions and/or conversations in the same manner as in the shared recognition determination of the above-described embodiment. will be determined, and recording will begin when consent is obtained. In this way, if the recording of the subject of interest is started when the express consent of the passenger 10 is obtained, it is not necessary to record even if there is an interest, against the will of the passenger 10. This prevents the agent system 15 from recording objects of interest.

Further, in the above embodiment, when it is determined that the passenger 10 recognizes that the agent unit 23 or the agent system 15 has identified the target of interest of the passenger 10 by the shared recognition determination, the record of the target of interest is recorded. Although it is started, it is not limited to this. For example, the subject-of-interest recording unit 43 can start recording a subject-of-interest when the subject-of-interest is identified. In this case, when it is determined by the shared recognition determination that the passenger 10 does not recognize that the agent unit 23 or the agent system 15 has identified the target of interest of the passenger 10, the target of interest recording unit 43 and discard any recordings of interest that have already been made. In this way, if the target of interest is identified and the recording of the target of interest is started before the shared recognition determination is made, the target of interest can be appropriately recorded without delay. In addition, when the passenger 10 is unaware that the object of interest is being shared and recorded, the previously started recording is interrupted and discarded. and the record is not provided to the passenger 10 as a memory.

In the above-described embodiment, a moving image of the vehicle exterior image 33 is saved as a record of the target of interest, but the present invention is not limited to this. In addition to the moving image by the image 33 outside the vehicle, the still image by the image 33 outside the vehicle, the image 31 inside the vehicle, the voice 32 inside the vehicle, the conversation of the passenger 10 which is part or all of the voice 32 inside the vehicle, or a combination thereof, etc. are related to the subject of interest. It can be recorded as information, ie memories. In addition, when the vehicle 100 is equipped with an external microphone that acquires external sound or voice, the audio or the like acquired by the external microphone can also be recorded as information related to the target of interest.

In the above embodiment, the subject-of-interest recording unit 43 records the subject-of-interest as memories of the passenger 10 or the like. , the results of detection and estimation by the target-of-interest identification unit 42, and the like can be recorded. That is, the subject-of-interest recording unit 43 classifies, as necessary, the actions and conversations of the passenger 10 related to the subject-of-interest, the degree of interest in the object that can be the subject-of-interest, the degree of interest in the subject-of-interest, the type of the subject-of-interest, and the like. , relevant information about the subject of interest, the search history of the car navigation system 12, etc., can be recorded as the history regarding the subject of interest of the passenger 10. FIG. Such history information reflects the passenger's 10 personal tastes and the like. For this reason, from the history information, it is possible to extract personal tendencies of the passenger 10, such as being likely to have an interest in mountains. Therefore, when the subject-of-interest recording unit 43 records the history information as described above, the subject-of-interest specifying unit 42 can detect objects and/or estimate the degree of interest based on the history information. For example, when the object detection unit 53 detects an object that can be a target of interest, the target-of-interest identification unit 42 can preferentially detect an object that the passenger 10 is likely to be interested in, based on history information. . Further, when the interest level estimating unit 54 estimates the interest level, the interest target identification unit 42 performs weighting or the like based on the history information. can be adjusted so that is estimated to be higher. In this manner, recording and utilizing historical information may allow more relevant interests to be identified based on passenger 10's personal tendencies.

The embodiments and modifications of the present invention have been described above. do not have. Each configuration shown in the above-described embodiment, modifications, and the like can be implemented by arbitrarily combining a part or all of them. Further, part of each configuration shown in the above embodiment, modifications, and the like can be omitted.

For example, in the above-described embodiment, the agent unit 23 outputs a voice message when displaying an object of interest, notifying the start of recording, notifying the end of recording, etc., but the present invention is not limited to this. Depending on the specific form of the agent unit 23, the various operations described above may be realized by barks or other sounds, facial expressions such as a special smile, or lights such as lighting of indicators.

Moreover, each part constituting the passenger detection unit 41 is an example of a configuration that realizes the function of the passenger detection unit 41 . Therefore, the motion detection unit 51 and/or the conversation detection unit 52 can be replaced with other configurations as long as the passenger detection function described in the above embodiment and the like is realized, and one of them can be omitted. be able to. The same applies to the subject-of-interest identification unit 42 and the interface control unit 44 . That is, each part constituting the target-of-interest specifying unit 42 can substitute the configuration described in the above-described embodiment or the like with another configuration or omit a part of the configuration, as long as the target-of-interest specifying function is realized. . Each unit constituting the interface control unit 44 can substitute other configurations for the configurations described in the above-described embodiments and the like, or part of the configurations can be omitted as long as the functions thereof are realized. In addition, the loss of interest determination unit 45 and the sharing recognition determination unit 46 may be replaced with other configurations or omitted as long as equivalent functions in supporting information recording can be achieved.

10: passenger, 11: driving operation unit, 12: car navigation system, 13: communication device, 14: speaker, 15: agent system (information recording support device), 16: vehicle controller, 17: sensor, 21: map information , 22: Landmark information, 23: Agent unit, 24: Agent controller, 25: Display, 26: Recording device, 27: In-vehicle camera, 28: In-vehicle microphone, 29: Out-of-vehicle camera, 30: Position sensor, 31: In-vehicle image , 32: voice inside the vehicle, 33: image outside the vehicle, 41: passenger detection unit, 42: interest object identification unit, 43: interest object recording unit, 44: agent control unit, 45: interest loss judgment unit, 46: shared recognition judgment Section 51: Action Detection Section 52: Conversation Detection Section 53: Object Detection Section 54: Interest Degree Estimation Section 55: Related Information Acquisition Section 56: First Action Control Section 57: Second Action Control Section 58: third operation control unit, 59: highlighting control unit, 61: field of view, 62: mountains, 63: frame line, 100: vehicle, 101: information recording support system

Claims (15)

An information recording support device that has an outside camera that captures an image of the outside of the vehicle and an agent unit that functions as an interface for passengers, and that supports information recording at an arbitrary timing using the outside camera via the agent unit. An information recording support method by a vehicle,
detecting movement and/or speech of the passenger;
identifying subjects of interest of the occupant external to the vehicle based on the occupant's movements and/or speech;
When the target of interest is specified, the information recording support device informs the passenger that the target of interest has been specified, and the external camera records the target of interest.
Information recording support method. The information recording support method according to claim 1,
The passenger is informed that the target of interest has been specified by executing a target-of-interest display operation for displaying that the attention of the agent has been directed to the target of interest by changing the form of the agent. ,
Information recording support method. The information recording support method according to claim 2,
The target-of-interest display operation causes at least part of the agent unit to operate toward the target of interest.
Information recording support method. The information recording support method according to claim 2 or 3,
The target-of-interest display operation changes the color and/or pattern of the agent unit.
Information recording support method. The information recording support method according to any one of claims 1 to 4,
The information recording support device includes a display that displays the shooting range of the outside camera,
notifying the passenger that the object of interest has been identified by highlighting the object of interest on the display;
Information recording support method. The information recording support method according to any one of claims 1 to 5,
The vehicle includes an in-vehicle camera that captures the passenger,
Using the in-vehicle camera, detecting a change in the line of sight of the passenger or a movement of the passenger's face or body,
Identifying, as the object of interest, an object to which the line of sight of the passenger or the direction of the face or body of the passenger is directed for a predetermined period of time or more;
Information recording support method. The information recording support method according to any one of claims 1 to 5,
The vehicle includes an in-vehicle microphone that acquires audio in the vehicle,
Detecting conversation of the passenger using the in-vehicle microphone,
estimating a topic in the conversation of the passenger;
identifying the subject of interest as a function of the topic;
Information recording support method. The information recording support method according to any one of claims 1 to 7,
extracting landmarks around the vehicle based on the position and orientation of the vehicle;
identifying the object of interest among the extracted landmarks based on the passenger's movements and/or speech;
Information recording support method. The information recording support method according to any one of claims 1 to 8,
estimating an interest level representing the degree of the passenger's interest in the possible object of interest based on the passenger's actions and/or conversations;
identifying the object as the subject of interest when the degree of interest is greater than a predetermined threshold;
Information recording support method. The information recording support method according to any one of claims 1 to 9,
Determining whether the passenger recognizes that the information recording support device has identified the target of interest based on the passenger's actions and/or conversations;
starting recording the target of interest when the passenger recognizes that the information recording support device has identified the target of interest;
Information recording support method. The information recording support method according to claim 1,
determining whether the passenger's interest in the subject of interest has been lost based on the passenger's actions and/or speech;
terminating the recording of the object of interest when the passenger's interest in the object of interest is lost;
Information recording support method. The information recording support method according to claim 11,
The vehicle includes an in-vehicle microphone that acquires audio in the vehicle,
Acquiring the passenger's conversation using the in-vehicle microphone,
estimating a topic in the conversation of the passenger;
determining that the passenger's interest in the target of interest is lost when the estimated topic transitions from a topic related to the target of interest to a topic not related to the target of interest;
Information recording support method. An information recording support device that is mounted on a vehicle having an exterior camera that captures images of the outside of the vehicle, has an agent unit that functions as an interface for passengers, and supports information recording at an arbitrary timing using the exterior camera via the agent unit. and
a passenger detection unit that detects the passenger's motion and/or conversation;
an object-of-interest identifying unit that identifies objects of interest of the passenger outside the vehicle based on the passenger's actions and/or speech;
a subject-of-interest recording unit that, when the subject of interest is identified, notifies the passenger that the subject of interest has been identified and records the subject of interest with the external camera;
An information recording support device comprising: An information recording support device that is mounted on a vehicle having an exterior camera that captures images of the outside of the vehicle, has an agent unit that functions as an interface for passengers, and supports information recording at an arbitrary timing using the exterior camera via the agent unit. An information recording support program for operating the
the information recording support device,
a passenger detection unit that detects the passenger's motion and/or conversation;
an object-of-interest identifier that identifies objects of interest of the occupant outside the vehicle based on the occupant's actions and/or speech;
A subject-of-interest recording unit that, when the subject of interest is identified, notifies the passenger that the subject of interest has been identified and records the subject of interest with the external camera;
Information recording support program that functions as A vehicle having an exterior camera that captures the outside of the vehicle, an information recording support device that has an agent unit that functions as an interface for passengers, and supports information recording at an arbitrary timing using the exterior camera via the agent unit; An information recording support system comprising
a passenger detection unit that detects the passenger's motion and/or conversation;
an object-of-interest identifying unit that identifies objects of interest of the passenger outside the vehicle based on the passenger's actions and/or speech;
When the target of interest is specified, the information recording support device informs the passenger that the target of interest has been specified, and the external camera records the target of interest.
Information recording support system.

Images