JP7548155B2 - Occupant Monitoring System - Google Patents

Description

The present invention relates to an occupant monitoring system.

In order to protect passengers in vehicles from falling over, etc., passenger monitoring systems have been proposed that monitor the interior of the passenger compartment and notify the driver if a passenger is in danger (see, for example, Patent Document 1).

Patent Document 1 introduces an in-vehicle monitoring device that is mounted on a vehicle that transports passengers who are seated or standing, and that includes a riding state grasping means for grasping the riding state of the passengers, a running state grasping means for grasping the running state of the vehicle, and an alarm means for issuing an alarm regarding the safety of the passengers based on the riding state and the running state.
The Patent Document 1 describes that a display device (monitor 103) can be used as the notification means 13. It also explains that as a warning means when the danger level becomes high, a warning by a voice alarm or the like is desirable from the viewpoint of not disturbing the driver's attention and concentration on the road ahead as much as possible.

JP 2016-65414 A

Here, conventionally, drivers visually check the inside and outside of the passenger compartment either directly or through a mirror or the like. When a vehicle is autonomous, image data capturing images of the inside and outside of the passenger compartment is transmitted to a controller outside the vehicle, and the controller visually checks the image data to check for safety. Hereinafter, the driver and controller may be collectively referred to as the driving controller.

Even when a warning is given by an audio alarm or the like, as in the technology introduced in Patent Document 1, it is assumed that the driving controller who perceives the warning will first visually check the level of danger. In this case, the driving controller needs to perform redundant safety checks using two different sensory organs, the visual and auditory organs, which may place an excessive burden on the driving controller.

In this case, the driving controller must visually check the area where the danger has occurred based on the information obtained by the auditory organs. In other words, in order to identify the area where the danger has occurred, the driving controller must convert the sound information obtained by the auditory organs into image information that will be obtained by the visual organs in the brain. If the driving controller cannot perform this conversion instantly, there may be a time lag between when the driving controller perceives the warning and when he or she visually checks the area where the danger has occurred, making it difficult to respond quickly.
Thus, in the conventional passenger monitoring system, there is a risk that the driving controller cannot respond quickly when a danger occurs. For this reason, there is a demand for a passenger monitoring system that can guide the driver to respond quickly when a danger occurs.

The present invention was made in consideration of the above circumstances, and aims to provide a technology that can quickly and reliably inform the driving controller of danger to occupants.

The passenger monitoring system of the present invention which solves the above problems comprises:
A plurality of monitoring elements each capturing occupant information for a different area of a passenger compartment of the vehicle;
A plurality of display elements arranged near a driver's seat of the vehicle and/or near a control seat during automatic driving, each displaying different occupant information;
a control element connected to the monitoring element and the display element;
The control element is an occupant monitoring system that calculates the risk level for each occupant information based on the occupant information of each occupant, and when it determines that the risk level has reached a predetermined risk level, displays a warning sign on the display element that displays the occupant information whose risk level has reached the risk level.

The passenger monitoring system of the present invention can quickly and reliably inform the driver or controller, who is the driving control officer, of any danger to passengers.

1 is an explanatory diagram illustrating a passenger monitoring system according to a first embodiment mounted on a vehicle; 1 is a block diagram illustrating an overview of an occupant monitoring system according to a first embodiment. 4 is a flowchart illustrating the operation of the occupant monitoring system according to the first embodiment. FIG. 11 is a block diagram illustrating an outline of an occupant monitoring system according to a second embodiment.

The following describes the embodiments of the present invention. Unless otherwise specified, the numerical range "a to b" described in this specification includes the lower limit a and the upper limit b. A numerical range can be formed by arbitrarily combining these upper and lower limit values, as well as the numerical values listed in the examples. Furthermore, numerical values arbitrarily selected from within the numerical range can be used as the upper and lower limit numerical values.

The passenger monitoring system of the present invention has multiple monitoring elements and multiple display elements.

Each monitoring element obtains occupant information for a different area in the passenger compartment of the vehicle, and each display element is disposed near the driver's seat of the vehicle and/or near the control seat during automatic driving, and displays different occupant information.
In other words, in the passenger monitoring system of the present invention, passenger information for each area is acquired individually using multiple monitoring elements. Then, the acquired individual passenger information is displayed on different display elements. Therefore, a driving controller, i.e., a driver sitting in the driver's seat or a controller sitting in the control seat, can visually check the safety of the passenger compartment for each area by visually checking the display elements.

The passenger monitoring system of the present invention also includes a control element connected to the above-mentioned monitoring element and display element.

The control element calculates the risk level for each occupant information based on the occupant information. If it determines that the risk level has reached a predetermined risk level, it displays a warning sign on a specific display element that displays the occupant information whose risk level has reached the risk level. A driving controller who perceives the warning sign will naturally look at the display element on which the warning sign is displayed, and at the same time, focus on the occupant information displayed on that display element. This enables the driving controller to accurately and instantly grasp the details of the area where the danger occurred and the location of the occupant where the danger occurred, and thus can take prompt action when a danger occurs.

Therefore, according to the occupant monitoring system of the present invention, it is possible to quickly and reliably notify the driving controller of the presence of danger to an occupant, thereby leading to the driving controller's rapid response when a danger occurs.
The passenger monitoring system of the present invention will be described below with respect to each of its components.

The passenger monitoring system of the present invention is a system that monitors passengers in a vehicle and notifies the driving controller if the passengers are in danger.

Vehicles that can be equipped with the passenger monitoring system of the present invention include various vehicles such as automobiles, ships, and aircraft, and the vehicles may be private vehicles or public transportation. In particular, preferred examples of the vehicles include public transportation that has an aisle through which passengers can pass and in which passengers are expected to move between their seats and the aisle, such as buses and trains. In this specification, the term "driver" refers to a person who drives the various vehicles described above, and is also referred to as a driver, pilot, navigator, etc.

The monitoring elements in the passenger monitoring system of the present invention individually obtain passenger information in different areas in the passenger compartment of the vehicle described above.
The area may be only a passenger compartment for passengers, or may be a cockpit, i.e., an interior space with a driver's seat, a pilot's seat, or a helm seat, or may be a semi-interior space such as a step in a bus or a deck in a ship.

One monitoring element may be provided for each area, or two or more may be provided for each area. For example, two or more monitoring elements may be provided for the same area, and occupant information may be obtained from two or more different angles for the same area.

The monitoring element may be anything that can acquire occupant information over time as material for determining the danger level of the occupant, and there are no particular limitations on the type of occupant information or the method of acquiring the occupant information.

For example, the monitoring element may be a camera that captures video or still images of the area, or a sensor that detects changes in the position or state of occupants in the area. Examples of such sensors include a human presence sensor that uses electromagnetic waves such as infrared rays, and a load sensor that uses a strain gauge or a piezoelectric element.

As the monitoring element, any of the above may be used alone, or a combination of a plurality of types may be used.
For example, a camera and a stop button on a public bus may be used together as a monitoring element. The stop button is operated by the passenger himself, but the passenger who presses the stop button may stand up or move to get off. For this reason, the stop button can function as a monitoring element for monitoring changes in the position or state of the passenger.

The occupant information acquired by each monitoring element is used by the control element to determine the degree of danger. For this reason, the monitoring elements are at least connected to the control element. In addition, since the occupant information is displayed on the corresponding display element, the monitoring elements are also connected to the display element or indirectly connected to the display element via the control element.
The term "connected" here means a state in which electronic signals can be sent and received, and the monitoring element, control element and display element may be connected wirelessly or by wires.

Each display element may display the occupant information acquired by each monitoring element as is, or may convert the information into visual information in some way.

For example, if the monitoring element is a camera, the image acquired by the monitoring element may be displayed as is, or the image may be displayed with various additional information, such as text data, added thereto. Also, if the monitoring element is a sensor of various kinds, visual information may be generated based on information other than the image acquired by the monitoring element, and the visual information may be displayed on the display element. Of course, an image may be generated as the visual information.
The converted occupant information as described above can also be essentially referred to as occupant information acquired by the monitoring element.

The display element is preferably one that can display occupant information in high resolution so that the driving controller can grasp the occupant information in a short time. Specifically, it is preferable to use a monitor or an electronic mirror. Of course, the display element may be an optical mirror (i.e., a simple mirror).

As mentioned above, each display element is placed near the driver's seat of the vehicle and/or the control seat during automatic driving. Near the driver's seat or control seat here means that the display element is placed in a position where the occupant information displayed by the display element and the caution signs described below can be seen by the driver sitting in the driver's seat or the controller sitting in the control seat.

The display elements may be separate or integrated. For example, a single monitor may be divided into multiple sections, each section being regarded as an individual display element, and different occupant information may be displayed on each of these display elements.
The passenger monitoring system of the present invention may also include display elements in a number equal to or greater than the number of monitoring elements. In some cases, passenger information derived from one monitoring element may be displayed on two or more display elements. For example, multiple pieces of passenger information may be displayed on a large monitor, and individual pieces of passenger information may be displayed on their own dedicated monitors.

The control element calculates the risk level for each piece of occupant information based on the above-mentioned information. The risk level associated with changes in the occupant's position or state varies depending on the type of vehicle. The risk level also varies depending on whether the vehicle is private or public transportation. Furthermore, the risk level also varies depending on the vehicle's travel route and the conditions of that route.

For example, if the vehicle is a public bus, the occupant may be standing while the vehicle is moving. Also, if the vehicle is a public bus, the occupant seated in the seat may slide from the window side to the aisle side, or even stand up in the aisle. On the other hand, if the vehicle is a private car, the occupant is seated while the vehicle is moving. Therefore, even if the monitoring element obtains occupant information indicating that the occupant has moved and changed position, it is preferable that the control element calculates the risk level of the occupant depending on the type of vehicle.

In other words, the control element in the passenger monitoring system of the present invention only needs to calculate the degree of danger appropriately depending on the type of vehicle in which the passenger monitoring system of the present invention is installed.

The control element may calculate the risk level collectively for each area on which the occupant information is based, or may identify each occupant included in the occupant information and calculate the risk level for each occupant. In order for the driving controller to recognize the risk to various occupants, it is preferable to calculate the risk level for each occupant.

Here, the degree of danger when an occupant performs the same action differs greatly between an infant or elderly occupant and a young occupant. Such differences in the degree of danger for each occupant can also occur depending on the occupant's body type, gender, health condition, etc. Therefore, it is preferable for the control element to calculate the degree of danger for each occupant by taking into account attribute information for each occupant, such as age, body type, gender, and health condition.

In other words, it is preferable that the occupant monitoring system of the present invention uses attribute information for each occupant when calculating the risk level by the control element. In other words, it is preferable that the control element determines the occupant's posture, occupant attribute information, seating position, and whether or not the occupant is seated based on the occupant information acquired by the monitoring element, and further calculates the risk level of the occupant based on the result of the determination.

Furthermore, for example, if the vehicle is a passenger bus, when the vehicle approaches a bus stop, it is considered that the vehicle will decelerate in order to stop. If the vehicle decelerates while an occupant is standing, the danger to the occupant increases even if there is no change in the occupant information acquired by the monitoring element. Therefore, it is preferable that the occupant monitoring system of the present invention is equipped with a second monitoring element that monitors the moving state of the vehicle and the outside of the vehicle, in addition to the monitoring element that monitors the passenger compartment.

Examples of such a second monitoring element include a speed sensor or an acceleration sensor for monitoring the moving speed of the vehicle, and a GPS (Global Positioning System) for monitoring the position of the vehicle.
The second monitoring element may utilize pre-installed map data, or may access a database outside the vehicle to obtain map information, traffic information, and the like.

When the control element determines that the danger level calculated as described above has reached a predetermined danger level, it causes a warning sign to be displayed on the display element that displays the occupant information on which the danger level is based.

Whether the degree of danger has reached the danger level can be determined by comparing the above-mentioned danger level with a predetermined danger level. The danger level varies greatly depending on the type and form of the vehicle, for example, whether the vehicle is a private car or a shared public transportation system, so it can be set appropriately depending on the type and form of the vehicle.

For example, if the vehicle is a shared public transport system, when it is detected based on the passenger information that a passenger who was seated is about to stand up, it can be determined that the danger level has reached the danger level. Alternatively, it can be determined that the danger level has reached the danger level when it is detected that a passenger who was standing is about to walk away, a passenger who was standing is about to reach for the luggage rack, a passenger is walking, etc.

In addition, it is also preferable to determine that the danger level has reached the danger level when, for example, a young occupant estimated to be 10 years old or younger, an elderly occupant estimated to be 75 years old or older, or when the vehicle is occupied only by such young occupants, elderly occupants, or a single female occupant. This is because in these cases, a subsequent occupant who boards the vehicle may pose a human-induced danger to the occupants who boarded earlier. In particular, in autonomous vehicles without a driver, it is preferable to determine that the danger level has reached the danger level in the above cases and display a warning sign for the controller.

In the passenger monitoring system of the present invention, the method of displaying the attention-attention sign is not particularly limited.
For example, a sign display unit that displays a warning sign may be provided in advance outside or at the rear of the occupant information display area of the display element. For example, if the sign display unit is a lamp, the warning sign can be displayed by turning on or blinking the lamp. In this case, even if the display element is an optical mirror, the frame of the display element can be illuminated or light can be transmitted from the rear side of the translucent display element to warn the driver. Of course, the display element in this case may be a monitor, an electronic mirror, or the like.

For example, a warning sign consisting of text, graphics, etc. may be displayed on the display element together with the occupant information. Specifically, when the display element displays a video as occupant information, the warning sign may be displayed by surrounding an area in the video that corresponds to the occupant or by coloring the entire area. Alternatively, the warning sign may be displayed by inserting text such as "Caution" or "Danger" into the video.

In order to allow the driving controller to quickly perceive the warning signs, it is preferable to display these warning signs in colors that can attract the driving controller's attention, such as colors with high saturation or brightness. In this case, the driving controller's attention can also be attracted by making the warning signs flash.

The color, shape, etc. of the warning sign may be changed in stages and emphasized to more easily attract the driver's attention when the danger level for the occupants increases or when a predetermined time has passed since the danger level reached a dangerous level.

For example, when the danger level is relatively low or when the time that has elapsed since the danger level was reached is less than a predetermined time, a yellow warning sign may be displayed. The warning sign may simply be lit or may blink at a relatively slow speed. Specifically, a blinking frequency of about 0.2 Hz is preferable.
Then, when the danger level increases or when a predetermined time has elapsed since the danger level was reached, a red attention sign may be displayed. This further emphasizes the attention sign. It is also effective to make the attention sign blink at a relatively high speed. Specifically, the blinking frequency at this time is preferably three times or more than the low speed, i.e., 0.6 Hz or more, and more preferably five times or more than the low speed, i.e., 1 Hz or more.

The following describes the passenger monitoring system of the present invention using specific examples.

Example 1
The passenger monitoring system of the first embodiment is mounted on a bus, which is a type of public transport means, and monitors passengers on board the vehicle to ensure the safety of the passengers.
FIG. 1 shows a schematic diagram of an occupant monitoring system of Example 1 mounted on a vehicle, FIG. 2 shows a block diagram outlining the occupant monitoring system of Example 1, and FIG. 3 shows a flowchart illustrating the operation of the occupant monitoring system of Example 1.
In FIG. 1, the terms front, rear, left and right refer to the front, rear, left and right in the traveling direction of the vehicle.

As shown in FIG. 2, the occupant monitoring system 1 of the first embodiment includes three monitoring elements 2, three second monitoring elements 3, three display elements 4, and a control element 5.

The monitoring element 2 in the occupant monitoring system 1 of the first embodiment is an infrared camera, and is capable of acquiring video and distance information as occupant information. Specifically, the distance information is the distance from a reference position of each element, such as the occupant or the seat 96, contained in the image information. Each infrared camera is assigned an individual ID. The infrared cameras are referred to as the first camera 21, the second camera 22, and the third camera 23.

As shown in FIG. 1, the passenger compartment 98 of the bus 9 is virtually divided into three areas (first area 91, second area 92, and third area 93), and the first camera 21, second camera 22, and third camera 23 are disposed at positions corresponding to the first area 91, second area 92, and third area 93, respectively. The first camera 21 acquires passenger information for the first area 91, the second camera 22 acquires passenger information for the second area 92, and the third camera 23 acquires passenger information for the third area 93.

The passenger monitoring system 1 of the first embodiment has three second monitoring elements 3, namely, a speed sensor 31, an acceleration sensor 32, and a GPS 33. These second monitoring elements 3 are arranged outside the passenger compartment 98 of the bus 9, and monitor the moving speed and position of the bus 9.

The control element 5 in the occupant monitoring system 1 of Example 1 is an AI analysis engine 51, which is mounted on the bus 9 and connected to the monitoring element 2.

The display element 4 in the occupant monitoring system 1 of the first embodiment is an electronic mirror, is connected to the control element 5, and displays a video of the occupant information acquired by the monitoring element 2. One of the three electronic mirrors is referred to as a first electronic mirror 41, the other as a second electronic mirror 42, and the other as a third electronic mirror 43.
1 , the first electronic mirror 41, the second electronic mirror 42, and the third electronic mirror 43 are disposed at different positions in front of the driver's seat 95 and face the driver's seat 95. The first electronic mirror 41 displays a video captured by the first camera 21, the second electronic mirror 42 displays a video captured by the second camera 22, and the third electronic mirror 43 displays a video captured by the third camera 23.

The operation of the occupant monitoring system 1 of the first embodiment will be described below with reference to the flowchart shown in FIG.

The passenger monitoring system 1 of the first embodiment starts when the bus 9 is started (START) and determines whether or not the bus has started to operate (S1). When the bus 9 starts to operate (YES in S1), the monitoring element 2 starts to acquire passenger information (S2).

The occupant information is transmitted from the monitoring element 2 to the control element 5 .
The control element 5 that has received the passenger information displays a video, which is part of the passenger information, on the display element 4 (S3). Then, it is determined whether or not the operation of the bus 9 has stopped (S10), and if the operation has stopped (YES in S10), it terminates (END) the passenger monitoring system 1. The monitoring element 2 repeats S2 to S10 and displays the video until the operation of the bus 9 has stopped.

At this time, the control element 5 also uses AI, i.e., artificial intelligence, to calculate the degree of danger based on the occupant information acquired by the monitoring element 2 (S4), and determines whether the degree of danger has reached a predetermined danger level (S5).

When the degree of danger reaches a predetermined danger level (YES in S5), the control element 5 selects a display element 4 that displays a warning sign (S6), and displays the warning sign (S7). Specifically, in the occupant monitoring system 1 of the first embodiment, a sign display unit (not shown) made of an LED lamp is provided on the frame of each of the first electronic mirror 41, the second electronic mirror 42, and the third electronic mirror 43, which are the display elements 4. The control element 5 lights up the sign display unit to display the warning sign.
If the degree of danger does not reach the predetermined danger level (NO in S5), the control element 5 repeats S2 to S5.

Here, the occupant monitoring system 1 of the first embodiment is equipped with an off switch (not shown) located near the driver's seat 95. When the driver perceives that a warning sign is being displayed, the driver can turn off the warning sign by turning on the off switch.

After displaying the attention-attention sign in S7, the control element 5 determines whether the off switch has been turned on (S8). If the off switch has been turned on (YES in S8), the control element 5 turns off the display of the attention-attention sign (S9). The control element 5 then determines whether the operation of the bus 9 has stopped (S10), and if operation has stopped (YES in S10), ends the passenger monitoring system 1 (END). The monitoring element 2 repeats S2 to S10 until the operation of the bus 9 has stopped, and displays the attention-attention sign on the sign display unit when the degree of danger reaches a predetermined danger level.

In the occupant monitoring system 1 of the first embodiment, when there are multiple occupants in a dangerous situation, the same warning sign is not displayed on the sign display sections of all the corresponding display elements 4, but the light intensity of only the sign display section with the highest level of danger is increased and the light intensity of the others is decreased. This is to avoid unnecessary confusion for the driving controller.

Even under normal circumstances, the driver visually checks the safety of the passenger compartment 98 for each area by looking at the display element 4. When a warning sign is displayed on such a display element 4, the driver quickly perceives the warning sign. In addition, the display element 4 on which the warning sign is displayed also displays an image of the occupant who is the cause of the warning sign being displayed. This allows the driver to accurately and instantly grasp the details of the area where the danger has occurred and the location of the occupant where the danger has occurred, and enables the driver to quickly respond to the danger.

In the passenger monitoring system of Example 1, a sign display unit consisting of an LED lamp is provided on the frame of the electronic mirror (first electronic mirror 41, second electronic mirror 42, and third electronic mirror 43), which is the display element 4, and a warning sign is displayed by lighting up the sign display unit, but the warning sign may be displayed inside the frame, i.e., on the electronic mirror itself. For example, when the degree of danger reaches a predetermined danger level, letters, figures, etc. may be displayed so as to be superimposed on the image of the corresponding passenger in the video displayed on the electronic mirror, and this may be used as a warning sign.

Example 2
The passenger monitoring system of Example 2 differs from the passenger monitoring system of Example 1 in that the vehicle, i.e., the bus, is driven automatically and a warning sign is displayed to the controller instead of the driver, but in other respects it is roughly the same as the passenger monitoring system of Example 1.
Therefore, the passenger monitoring system of the second embodiment will be described below, focusing on the differences from the passenger monitoring system of the first embodiment.
FIG. 4 is a block diagram for explaining an outline of an occupant monitoring system according to the second embodiment.

As shown in FIG. 4, the passenger monitoring system 1 of the first embodiment is composed of a vehicle-side system 10 mounted on a vehicle and a control-side system 11 arranged outside the vehicle.
The monitoring element 2, the second monitoring element 3 and a part of the control element 5 belong to the vehicle-side system 10, and the remaining part of the control element 5 and the display element 4 belong to the control-side system 11. The occupant monitoring system 1 of the second embodiment is equipped with a first monitor 45, a second monitor 46 and a third monitor 47 as the display element 4, instead of the first electronic mirror 41, the second electronic mirror 42 and the third electronic mirror 43. The first monitor 45, the second monitor 46 and the third monitor 47 are arranged in front of the control seat in the control room.

Of the control elements 5, the AI analysis engine 51 and the transmission unit 52 belong to the vehicle-side system 10. The reception unit 53 belongs to the control-side system 11. The control element 5 transmits a video display signal from the transmission unit 52 to the reception unit 53, and the reception unit 53, which receives the signal, displays the videos captured by the monitoring elements 2, i.e., the first camera 21, the second camera 22, and the third camera 23, individually on each of the first monitor 45, the second monitor 46, and the third monitor 47. Then, similar to the occupant monitoring system 1 of Example 1, when the danger level reaches a predetermined danger level, a sign display signal is transmitted from the transmission unit 52 to the reception unit 53, and a warning sign is displayed on the sign display unit.

As with the occupant monitoring system 1 of Example 1, the occupant monitoring system 1 of Example 2 allows the controller to accurately and instantly grasp the details of the area where the danger has occurred and the location of the occupant where the danger has occurred, and is therefore able to respond quickly to the danger.

The present invention is not limited to the embodiments described above and shown in the drawings, and can be modified as appropriate without departing from the spirit of the invention. Furthermore, each of the components shown in this specification, including the embodiments, can be arbitrarily extracted and combined for implementation.

1: Passenger monitoring system 4: Display element 5: Control element 2: Monitoring element 9: Bus (vehicle)
91: First Area (Area)
92: Second Area (Area)
93: Third Area (Area)
95: Driver's seat 98: Passenger compartment

Claims (4)

A plurality of monitoring elements each acquiring occupant information for a plurality of different areas virtually partitioned in a passenger compartment of the vehicle;
A plurality of display elements are arranged near a driver's seat of the vehicle and/or near a control seat during automatic driving, each displaying different occupant information in real time ;
a control element connected to the monitoring element and the display element;
The monitoring element acquires an image as the occupant information, or detects a change in position or a change in state of the occupant as the occupant information,
The control element calculates the degree of danger for each piece of occupant information based on the occupant information of each piece, and when it determines that the degree of danger has reached a predetermined danger level, displays a warning sign on one of the multiple display elements that displays the occupant information whose degree of danger has reached the danger level. The occupant monitoring system of claim 1, wherein the control element identifies each occupant included in the occupant information, performs the judgment for each occupant, and displays the attention warning sign for each occupant. The occupant monitoring system of claim 2, wherein the attention warning sign is displayed in a frame of the display element or is superimposed on an image of the occupant displayed on the display element. the passenger compartment includes a passageway;
When a passenger sitting in a seat moves to the aisle side on the seat, the warning sign is displayed,
4. The passenger monitoring system according to claim 2, further emphasizing the attention-calling sign when the passenger stands in the passageway.

Images