JPH11222882A - Dangerous zone monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily monitor a dangerous zone such as cliff by means of a construction machine itself, which performs a construction work, without requiring a large amount of time and labor. SOLUTION: A distance image generating unit 6 generates a distance image having distance information as three-dimensional information based on a plurality of images picked up by a multiple-lens camera 1 mounted on a loading work machine body 5 to take a picture of a subject in the direction wherein the edge E of a cliff is present as a plurality of images each having a predetermined parallax. A cliff edge recognition unit 7 converts the distance images to distance images in the case that a car is not inclined based on inclination angles calculated by a car inclination arithmetic unit 10 to recognize the edge E based on the converted distance images, whereby a zone beyond the edge E is recognized as a dangerous zone. A cliff edge approach decision unit 11 decides whether or not the distance to the edge E is equal to or less than a predetermined distance and when the distance is not more than the predetermined distance, the unit 11 automatically stops the work of a loading work machine A through a control unit 13 or reports that the machine A comes close to a remote operation device. And when the inclination angle exceeds a predetermined value, the unit 7 does not perform a recognition processing but performs it on the following distance images.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention monitors a danger area having a rapid change in terrain such as a cliff, which is mounted on a movable construction machine and monitors the danger area. TECHNICAL FIELD The present invention relates to a dangerous area monitoring apparatus for preventing a vehicle from entering a vehicle, and more particularly to a dangerous area monitoring apparatus for a remotely operated construction machine.

[0002]

2. Description of the Related Art Conventionally, in a dozing operation using a bulldozer A1 as shown in FIG. 10, since a work on a cliff is dangerous, a human is placed on a cliff E3 to remotely control a bulldozer pushing a soil. Bulldozers were reported to the police and surveillance stations by hand flags or by radio. In addition, work in such a dangerous place is usually performed by a construction machine such as a bulldozer remotely controlled.

[0003] Alternatively, a boundary between the safety area and the danger area is provided by using a laser beam at the edge of a cliff, and means for detecting the laser beam is mounted on the bulldozer side. This was communicated to the remote operator and the monitoring station by wireless means.

[0004]

However, when arranging a person near a cliff, a person for monitoring is naturally required for the arrangement, which is labor intensive and does not allow efficient work. There was a point.

On the other hand, when judging the approach to a dangerous area using a laser beam, a device for emitting a laser beam, for example, a laser power supply or a laser oscillator is required, and a device for installing this device is required. If maintenance and maintenance after preparation and installation are required, and there are multiple workplaces, after completing work in one workplace, work in another workplace cannot be performed immediately, After all, there was a problem that it took a lot of time and effort.

Accordingly, the present invention is to provide a danger area monitoring apparatus which eliminates such a problem and can monitor a danger area such as a cliff by a construction machine itself, which is a work subject, simply and without much time and effort. The purpose is to provide.

[0007]

A first aspect of the present invention is to monitor a dangerous area having a rapid terrain change, such as a cliff, which is mounted on a movable construction machine and is located around the construction machine. In a danger area monitoring device for preventing a construction machine from entering the danger area, the danger area monitoring apparatus is mounted on the construction machine and captures a subject in a direction in which the danger area exists as a plurality of images having a predetermined parallax. Eye camera, the distance information between the multi-eye camera and the subject for each pixel constituting the image based on the predetermined parallax from a plurality of images captured by the multi-view camera, A distance image generating unit configured to generate a distance image in which the distance information is added to each corresponding pixel of the captured image; and a position of the dangerous area with respect to the multi-lens camera based on three-dimensional information indicated by the distance image. Recognition Wherein the hazardous area recognition means, in that the construction machine based on the recognition result of the danger area recognition unit equipped with a control means for controlling not to close to the critical section.

[0008] Thus, a dangerous area having a rapid change in terrain such as a cliff is automatically recognized based on the distance image obtained by the multi-lens camera, and control is performed to prevent the approach to the dangerous area. Therefore, it is possible to concentrate on the original work operation of the construction machine, and since the construction machine itself monitors the entry of the construction machine into the danger area, the labor, time and equipment required for monitoring the approach to the danger area Etc. can be extremely reduced.

According to a second aspect of the present invention, in the first aspect, an inclination detecting means for detecting an inclination of the construction machine, and an inclination amount for calculating an inclination amount of the construction machine based on a detection result of the inclination detecting means. It is characterized by further comprising a calculating means and a correcting means for correcting the distance image to reference coordinates based on the tilt amount.

Thus, when the construction machine rides on a convex portion on the ground plane, the construction machine is inclined, and the multi-lens camera is also inclined according to the inclination. However, the distance image is not inclined by the correction means. Since the coordinates are converted to the reference coordinates, the recognition process of the dangerous area recognition means may be a uniform process, and the load on the recognition process is reduced.

[0013] In a third aspect based on the first aspect, the dangerous area recognizing means includes an area dividing means for dividing a display plane of the distance image into a plurality of areas, and each area divided by the area dividing means. Average distance calculation means for calculating the average distance of the; and change rate calculation means for calculating the change rate of the average distance between the adjacent areas, the area where the change rate is a predetermined value or more of the dangerous area A boundary is set, and an average distance of the area is recognized as a distance from the multi-lens camera to the dangerous area.

[0012] This makes it possible to easily and quickly recognize a dangerous area.

In a fourth aspect based on the first aspect, the dangerous area recognizing means includes a ground plane calculating means for calculating a ground plane shape based on a three-dimensional position distribution indicated by the distance image, The distance from the multi-lens camera to the dangerous area is recognized based on the shape of the ground plane.

This makes it possible to easily and reliably recognize the dangerous area.

[0015] In a fifth aspect based on the first aspect, the control means automatically stops the construction machine when a distance from the construction machine to the danger area becomes equal to or less than a predetermined value. The remote control device for remotely controlling the construction machine is notified that the machine has approached the dangerous area.

As a result, an accident caused by the construction machine entering the danger area can be prevented beforehand, and the burden on the operator for preventing the accident from occurring is greatly reduced.

[0017]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing a schematic configuration of a remote control system for a construction machine including a dangerous area monitoring apparatus according to an embodiment of the present invention.

In FIG. 1, the loading work machine A is remotely controlled by a remote control device 15. In this remote control, control data is transmitted from a remote control device 15 to an antenna 14.
b, 14a, which are sent to the loading work machine main body 5 via the transmission / reception control unit 12, and based on the control data, the control unit 13 drives the traveling unit 16 to run the loading work machine A, or to load the loader bucket 4a. The driving work is performed, and data on the state of the loading work machine A is sequentially sent from the loading work machine A to the remote control device 15, thereby contributing to the remote control of the remote control device 15.

The multi-lens camera 1 is mounted at a position where the working direction or traveling direction of the loading work machine A can be seen, and its angle of view 2 is oriented in a direction in which the ground surface can be imaged. Here, when the loading work machine A performs work freely, it suffices that the cliff edge E existing around the loading work machine A can be finally imaged.

The multi-lens camera 1 has a plurality of cameras arranged at a predetermined interval in one multi-lens camera 1 and simultaneously captures an image of the same subject.
This is realized by an OS-type solid-state imaging device.

This multi-lens camera 1 is used when the number of cameras is two, as shown in FIGS. 2 (a) and 2 (b).
This is effective when obtaining the distance between a certain image 21 of one of the images 20a and 20b captured by the multi-lens camera 1, and a window 22 around the image 21.
A pattern most closely resembling the luminance value (brightness) pattern of the pixel 21 (including the pixel 21) is searched for on the excipolar line in the other image 20b, and the corresponding pattern 22 'is obtained. The position of a pixel in the image 20b is obtained, and the distance can be obtained from the displacement of the pixel, that is, the parallax. The distance is set as the distance of the pixel 21, and the other pixels similarly have information on the distance, so that An image having distance information corresponding to the image 20a can be obtained.

The captured images corresponding to the number of cameras captured by the multi-lens camera are divided into a distance image generation unit 6 and a cliff recognition unit 7.
Is input to

The distance image generation unit 6 generates a distance image by using predetermined parallaxes of captured images of the number of cameras captured by the multi-lens camera 1 as described above (see FIG. 2) (FIG. 3).
(C) and (d)).

For example, FIG. 3A shows a state in which the loading work machine A is present on a flat ground, and the multi-lens camera 1 captures an image of the flat ground.
It is generated as a distance image shown in (c). The distance image is displayed in a color closer to white as the distance is shorter, and is displayed in a color closer to black as the distance is longer. The distance image captured in this manner becomes a three-dimensional image to which distance information from the loading work machine A is added.

This distance image is sent to the cliff side recognition unit 7,
The cliff recognition unit 7 recognizes the cliff based on the three-dimensional information indicated by the distance image. The concept of this recognition is as shown in FIG. 3A, when there is no cliff in front of the loading work machine A, a distance image shown in FIG. Therefore, the distance sequentially increases. On the other hand, as shown in FIG. 3B, when there is a cliff in front of the loading work machine A,
The distance image shown in FIG. 3D is obtained, and the distance sequentially increases from the portion close to the loading work machine A to the edge of the cliff, as in FIG.
The distance becomes infinite and the distance changes rapidly at the cliff E.
Therefore, when the distance changes abruptly on the distance image, that is, when the distance image is a black-and-white image, the boundary where the shading changes abruptly can be recognized as the edge of the cliff E, and the part farther from the recognized edge of the cliff E can be recognized. Can be recognized as a dangerous area.

Further, the recognition processing of the cliff edge recognition unit 7 will be described in detail with reference to FIG. 4. First, the distance image sent from the distance image generation unit 6 is assumed to be 500 × 500 pixels, and FIG. As shown in FIG.
Each pixel is represented by d (i, j) when represented using coordinates (i, j) of 9, j = 0 to 499. Here, the distance image d
When (i, j) is divided using a divided region of 10 × 10 pixels as one unit, the result is as shown in FIG. 4B. FIG.
In (b), the coordinates of each divided area are I = 0 to 49, J = 0.
When expressed using coordinates (I, J) of ~ 49, each divided area is DA (I, J). Each divided area DA (I,
An average value of the distance indicated by each pixel in J) is obtained for each divided region, and each average value is defined as DB (I, J). After that, for each J, that is, in the depth direction (vertical direction), the change rate H (I, J) between the vertically adjacent divided regions is calculated using the following equation. That is, H (I, J) = (DB (I + 1, J) -DB (I,
J)) / DB (I, J) and the rate of change H (I, J) for all divided regions (I, J).
J).

It is determined whether or not the obtained rate of change H (I, J) exceeds a predetermined threshold value, and the coordinates of the exceeded divided area are marked. In FIG. 4C, the corresponding divided area is indicated by oblique lines. The divided area marked in this way indicates a cliff E, and from the cliff E, it is recognized that the depth direction is a cliff. Therefore, since the average distance of the marked divided area is known, the multi-lens camera 1 can be used to detect the cliff E
, In other words, the distance from the loading machine A to the cliff E.

By the way, the ground on which the loading work machine A travels is not limited to a flat surface, but often has a convex portion 40 as shown in FIG. Are not parallel. For this reason, when the loading work machine A climbs on the convex portion 40 or enters a concave portion (not shown), the loading work machine A is inclined.
As a result, the multi-view camera 1 installed in the loading work machine A also tilts according to this tilt, so that the picked-up image picked up by the multi-view camera 1 is also tilted and the distance image is tilted similarly.

For this reason, the loading work machine A has a pitching inclinometer 8 for detecting the pitching of the vehicle body of the loading work machine A and a rolling inclinometer 9 for detecting the rolling. The pitching angle and the rolling angle are calculated based on the detection results detected by the inclinometer 8 and the rolling inclinometer 9, and output to the cliff recognition unit 7.

Before performing the above-described cliff edge recognition process, the cliff edge recognition unit 7 always converts the distance image from a coordinate system to a coordinate system when there is no inclination. That is, the conversion is performed by applying the next conversion matrix MR0 to the coordinate system of the inclined distance image. That is, Is applied to the coordinate system of the inclined distance image. Note that “RX0”, “RY0”, and “RZ0”
Rotation around the X, Y, and Z axes respectively corresponds to a rolling angle, a pitching angle, and a yawing angle. Here, since the yawing angle is not detected, "RZ0"
May be set as “0”. Of course, a yaw rate gyro or the like may be installed in the loading work machine A to measure the yawing angle.

In this case, the GPS receiver is provided in the loading work machine A, and the absolute position of the loading work machine A is detected, whereby the absolute position at the edge of the cliff can be obtained. This calculation is based on the conversion matrix MR0 described above.
And can be obtained as follows. That is, the absolute position of the final cliff E (XP0, YP0, ZP0)
Sets the relative position of the cliff E to the loading machine A as (XP1, YP1, ZP1), and the absolute position of the loading machine A by GPS as (HX0, HY0, HZ).
0) Can be determined by:

When the multi-lens camera 1 is tilted as shown in FIG. 5 with the tilting of the loading work machine A, the central axis 42 of the angle of view cannot capture the ground surface. Further, if the angle of view is inclined by half the angle of view 2, the lowermost boundary line 41 of the angle of view also becomes horizontal with respect to the ground surface, and the ground surface cannot be captured at all. Cannot be performed at all.

For this reason, the cliff edge recognition section 7 does not perform the cliff edge recognition process when the boundary line 41 is inclined beyond the angle at which it becomes horizontal. That is, (camera installation angle θ
When c− (angle of view / 2)> pitching angle θp, the cliff recognition process is not performed. The camera installation angle θc refers to an angle from the horizontal plane of the multi-lens camera 1 to the center axis 42 of the angle of view 2.

The camera installation angle θc is set so that the intersection between the center angle of the angle of view 2 and the ground surface is, for example, 5 m from the multi-lens camera 1 or the loading work machine A.

Thereafter, the cliff-side proximity determining unit 11 determines whether the loading work machine A is on the cliff-side E based on the recognition result of the cliff-side recognition unit 7.
Is determined to be closer to or more than a predetermined value, and if it is determined that there is a proximity, an instruction to immediately stop the work or traveling of the loading work machine A is issued via the control unit 13. Alternatively, a notification is provided via the transmission / reception control unit 12 that the remote control device 15 has been approached.

Here, referring to a flow chart shown in FIG. 6, the procedure of the dangerous work area proximity prevention processing of the loading work machine A will be summarized.

First, the cliff edge recognition unit 7 includes the distance image generation unit 6
Acquires the distance image generated based on the captured image from the multi-lens camera 1 (step 101). Further, the pitching angle θp and the rolling angle θr are obtained from the vehicle body inclination angle calculation unit 10 (step 103). And the pitching angle θ
Using the p and the rolling angle θr, the inclined distance image is coordinate-transformed and corrected to a non-inclined distance image (step 10).
5).

Thereafter, the cliff edge recognition unit 7 determines whether or not the value of (camera setting angle θ−angle of view / 2) is greater than the pitching angle θp (step 107). Since it cannot be recognized, the process proceeds to step 101, and the processes of steps 101 to 105 are performed again. On the other hand, if it is determined in step 107 that the distance is not large, an average distance for each divided region divided in advance is calculated as shown in FIG. 4 (step 109), and an average in the depth direction is calculated based on the average distance. The rate of change of the distance is calculated (step 111), and a divided area having the rate of change equal to or greater than a predetermined value is recognized as a cliff edge E, and a deep area from the divided area is recognized as a cliff (step 113). Then, the cliff edge determination section 11 determines whether the distance between the recognized cliff edge and the multi-lens camera 1 is equal to or less than a predetermined value (step 115). Since they are not close to each other, the process proceeds to step 101 and the above-described processing is repeated.

On the other hand, if it is determined in step 115 that the value is not more than the predetermined value, the loading work machine A is close to the edge of the cliff E and is dangerous. An automatic stop process for immediately stopping is executed (step 117), and this process ends.

By the way, the above-described cliff recognition processing by the cliff recognition unit 7 shown in FIG. 4 is an example of the recognition processing, and various recognition processing is performed by using three-dimensional information indicated by the distance image. Can be.

For example, FIG. 7 shows an example of an image taken by the multi-lens camera 1. However, FIG. 7 illustrates an example in which the subject is not the ground surface including the cliff E but the worker standing on the ground surface.

A distance image is generated from the captured images of the number of cameras of the multi-lens camera 1 as shown in FIG. 7, and from the distance information for each pixel of the generated distance image, as shown in FIG. Three-dimensional information is obtained. This three-dimensional information is calculated based on the distance information for each pixel of the distance image. From the three-dimensional information shown in FIG. 8, a front image shown in FIG. 9A and a side image shown in FIG. 9B are obtained, so that the positional relationship where the worker stands on the ground surface can be easily understood. it can. The cliff edge recognition unit 7 extracts, for example, data P1 (lowest point data) having the lowest height with respect to the width-depth plane, and performs a Hough transform or the like on the data group of the extracted lowest point data P1. By performing the image conversion processing of the above, the ground surface approximated to the plane is recognized. Although FIG. 9 shows a ground surface without cliffs, for example, if there is no data in the area E2 in FIG. 9B, this area E2 can be recognized as a dangerous area due to the presence of the cliff edge E. In this case, when the plane is divided in the depth direction and a plane is obtained for each division, the cliff E can be easily recognized depending on how far the plane exists in the depth. Further, even when the shape of each of the divided planes suddenly changes, it can be recognized that the cliff E exists. Further, the plane approximation may be performed using not only the lowest point data P1 but also the second or third data from the lowest point.

As described above, the edge E can be recognized by performing various types of image processing based on the three-dimensional information obtained from the distance information for each pixel of the distance image.
Of course, in addition to the cliff E described above, even in a state where the cliff stands and is blocked at the destination, for example, the cliff can be easily recognized as an obstacle by the same recognition processing as the cliff E. .

In the above-described embodiment, a construction machine such as a loading work machine which is remotely controlled is described. However, the present invention is not limited to this, and the present invention is also applied to a case where a pilot operates on a construction machine itself. be able to. in this case,
Since the construction machine itself stops automatically without paying attention to whether or not it is close to the cliff, the operator can concentrate on the work operation itself. Also, the operator may be notified without the automatic stop of the construction machine.

[Brief description of the drawings]

FIG. 1 is a diagram showing a schematic configuration of a remote control system for a construction machine including a dangerous area monitoring device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a principle of generating a distance image using a multi-lens camera.

FIG. 3 is a diagram illustrating the principle by which a cliff can be recognized by a distance image.

FIG. 4 is a diagram illustrating a process of recognizing a cliff edge and a dangerous area based on a distance image.

FIG. 5 is a diagram illustrating a state in which a cliff cannot be imaged when the loading work machine is inclined.

FIG. 6 is a flowchart illustrating a procedure of recognition of a cliff and control processing accompanying the recognition.

FIG. 7 is a diagram illustrating an example of a captured image.

8 is a diagram illustrating three-dimensional information of a distance image generated based on the captured image of FIG. 7;

9 is a diagram showing the three-dimensional information shown in FIG. 8 as a front image and a side image.

FIG. 10 is a diagram illustrating a work state at a cliff.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Multi-view camera 2 ... Angle of view 3 ... Subject 4a ... Loader bucket 4b ... Loader arm 4c ... Lift cylinder 5 ... Main body of loading work machine 6 ... Distance image generation unit 7 ... Cliff recognition unit 8 ... Pitching inclinometer 9 ... Rolling Inclinometer 10 ... body inclination angle calculation unit 11
… Cliff-side proximity determination unit 12… transmission / reception control unit 13… control units 14 a, 14 b…
Antenna 15 ... Remote control device 16 ... Running part A ... Loading machine E ... Cliff

Claims (5)

[Claims] 1. A danger area having a sudden change in terrain, such as a cliff, mounted on a movable construction machine and monitored around the construction machine to prevent the construction machine from entering the danger area. In the hazardous area monitoring device for preventing, a multi-view camera mounted on the construction machine and imaging a subject in a direction in which the dangerous area exists as a plurality of images having a predetermined parallax, and an image captured by the multi-view camera Obtain distance information between the multi-lens camera and the subject for each pixel constituting the image based on the predetermined parallax from a plurality of captured images, and calculate the distance information for each pixel corresponding to the captured image. A distance image generating unit configured to generate a distance image added to each of the plurality of distance images; a dangerous region recognizing unit configured to recognize a position of the dangerous region with respect to the multi-lens camera based on three-dimensional information indicated by the distance image; Recognition result dangerous area monitoring device, characterized in that the construction machine based on the and control means for performing control not to close to the danger area of the recognition means. 2. An inclination detecting means for detecting an inclination of the construction machine; an inclination calculating means for calculating an inclination amount of the construction machine based on a detection result of the inclination detecting means; And a correction means for correcting the distance image to reference coordinates.
Hazardous area monitoring device according to item 1. 3. The dangerous area recognizing means, an area dividing means for dividing a display plane of the distance image into a plurality of areas, and an average distance calculating means for calculating an average distance for each area divided by the area dividing means. And a change rate calculating means for calculating a change rate of an average distance between the adjacent areas, an area where the change rate is equal to or more than a predetermined value is set as a boundary of the dangerous area, and an average distance of the area is set as the average distance. The dangerous area monitoring apparatus according to claim 1, wherein the recognition is performed as a distance from a multi-lens camera to the dangerous area. 4. The danger area recognizing means includes a ground plane calculating means for calculating a ground plane shape based on a three-dimensional position distribution indicated by the distance image, and the ground plane calculating means based on the ground plane shape. The dangerous area monitoring apparatus according to claim 1, wherein the distance from the multi-lens camera to the dangerous area is recognized. 5. The control means automatically stops the construction machine when a distance from the construction machine to the dangerous area becomes equal to or less than a predetermined value, or determines that the construction machine has approached the dangerous area. The dangerous area monitoring device according to claim 1, wherein the remote control device that remotely controls the construction machine is notified.