JP7246641B2 - agricultural machine - Google Patents

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an agricultural work machine that travels while detecting an object existing in a field.

The agricultural work machine according to Patent Document 1 is equipped with an ultrasonic obstacle sensor for detecting an obstacle existing in a field. The engine etc. are stopped.

The combine according to US Pat. This laser scanner is configured to irradiate the laser biased to the right side of the reaping part in order to detect a person who has entered an area where most of the field is exposed due to the reaping of the material to be reaping. It is Moreover, Patent Document 2 describes that it is possible to use a millimeter wave radar instead of a laser scanner or to use image processing using a camera or the like in order to detect obstacles.

In the combine disclosed in Patent Literature 3, photographed images sequentially obtained by a photographing unit provided in the machine body are input, and an existence region in which a recognition target object shown in the photographed image exists is output together with its estimated probability. be. When the recognition target is estimated with a high estimation probability, a warning is issued and travel control is performed. Although machine learning is used for estimation of this recognition object, the detailed structure of machine learning is not disclosed.

JP-A-9-76850 JP 2017-176007 A JP 2019-004773 A

Objects, such as people and animals, may exist as traveling obstacles during work travel by agricultural implements in fields. If this object exists near the fuselage in the direction of travel of the fuselage, it is necessary to avoid interference of the object with the fuselage. For this purpose, it is necessary to detect an object that may become a traveling obstacle with high accuracy, and to calculate the distance to the aircraft with high accuracy. However, with the object detection technology installed in conventional agricultural machinery, it has been difficult to detect objects in a field quickly and with high accuracy. Under such circumstances, there is a demand for an agricultural working machine equipped with an object detection technology for quickly and accurately detecting an object in a field along with the distance from the machine.

According to the present invention, an agricultural work machine that runs in a field receives a photographing unit that photographs a specific area of the field and a photographed image from the photographing unit, recognizes a specific object in the photographed image, a neural network for outputting object area information including a specific object contour indicating the area occupied by the specific object in the captured image; and object position information indicating the spatial positions of all objects existing in the specific area by photographing the specific area. and a three-dimensional point group data developed in a three-dimensional coordinate system with the field field as a reference plane from the object position information, and the three-dimensional point cloud data included in the contour of the specific object Based on the point cloud data and the contour of the specific object, an agricultural field including a boundary between an uncut land that is one of the specific objects on which crops are present and a cut land that is one of the specific objects from which the crops have been cut. a field information creating unit that creates reaping information; a spatial position of a traveling obstacle that is one of the specific objects is determined based on the object position information, the object area information, and the field reaping information ; A distance calculation unit that calculates a specific object distance, which is a distance to a traveling obstacle, and an object detection signal generation unit that outputs an object detection signal based on the specific object distance.

According to this configuration, the neural network recognizes the specific object existing in the specific area of the field using the photographed image from the photographing unit, and outputs object area information indicating the area occupied by the specific object in the photographed image. The spatial position measurement unit generates object position information indicating the spatial positions of all objects existing in the specific area of the photographed field. The spatial position of the area occupied by the specific object detected by the neural network in the specific area is calculated by associating the object area information with the object position information, and the distance from the aircraft to the specific object is calculated. For example, if the specific object indicated by the object area information is a traveling obstacle such as a human being, the distance from the aircraft to the traveling obstacle can be calculated by associating the object area information with the object position information. It is possible to control the aircraft according to the distance. In this invention, a neural network that is excellent in the ability to detect a specific object even if it cannot calculate the exact spatial position of the specific object, and a neural network that can accurately detect some object (unknown object) even if it does not have the ability to detect the specific object. By combining with a spatial position measuring unit that excels in position calculation capability, an agricultural work machine is provided that can quickly and accurately detect a specific object in a field along with the distance from the machine.

In one preferred embodiment of the present invention, in order to recognize a specific object in a photographed image and output object region information indicating the region of the specific object in the photographed image, the neural network comprises: It is composed of a segmentation neural network (semantic segmentation) consisting of an encoder network and a decoder network, and the object region information includes the contour of the specific object. In this configuration, when a photographed image is input, objects included in the photographed image are classified into at least one or more, and images identified so that the classification can be understood, for example, in different colors according to the type of the object. A represented image (label image) is output. Therefore, data indicating the contour of the specific object in the captured image can be extracted from the object area information.

If the segmentation neural network is implemented as it is, the network structure becomes complicated, so the arithmetic processing load increases and the processing speed decreases. In order to speed up the network, it is preferable to reduce the number of hidden layer units (also called perceptrons) constituting the network by pruning. For this reason, in one preferred embodiment of the present invention, the hidden layers constituting the encoder network and the decoder network include a convolution layer, a pooling layer and a batch normalization layer, and the batch Pruning is performed based on the parameters in the normalization layer. In this configuration, units judged to have a small contribution to good output based on the parameters obtained by the batch normalization operation in the batch normalization layer are not used. This simplifies the network structure and increases the processing speed of the neural network.

Since the distance to the specific object is used for airframe control to avoid interference with the specific object, it must be calculated quickly and accurately. In addition, it is necessary to avoid, as far as possible, the use of measurement values that are erroneous due to disturbances or the like in calculating the distance to a specific object. From this point of view, in one preferred embodiment of the present invention, the specific object distance is the distance from the center of gravity of the traveling obstacle to the aircraft body calculated from the contour of the traveling obstacle , or the above. It is a statistical representation of distance measurement points contained within the contour. The position of the center of gravity is calculated from coordinates (spatial positions) of a large number of distance measurement points that indicate the contour. Statistical representative values are medians, medians and averages, which are also calculated based on a large number of distance measurement points. In this way, by using the center of gravity of the specific object and the statistical representative value to calculate the specific object distance, it is possible to prevent the error value from occupying a large proportion in the calculation of the distance to the specific object.

In one preferred embodiment of the present invention, the spatial localization section generates the object position information using a stereo matching technique for the specific region. In this configuration, three-dimensional point cloud data is generated by stereo matching using a photographed image of a specific area of a field. Since each point of the point cloud data has a three-dimensional position, that is, a spatial position, object position information is generated based on the three-dimensional point cloud data. The captured image used for stereo matching can be the captured image used for the neural network, but in order to obtain a captured image with a resolution suitable for stereo matching, it is preferable to use a dedicated camera. should be used. Of course, at that time, a stereo matching photographic image having a photographic field of view equal to or wider than that of the photographic image used for the neural network is used.

Among various types of agricultural machinery, in the field where the harvester for harvesting crops runs, there are uncut land where there are crops growing high above the field, uncut land left after harvesting these crops, and ridges surrounding them. exist. In some areas, agricultural crops such as rice and wheat are in a lodging posture, and the height varies depending on the location. Furthermore, the areas of already-cut and un-cut land fluctuate as the cutting work progresses. In detecting a specific object such as a person or an animal, it is important to distinguish between already cut land and uncut land and to detect the height of crops growing on the uncut land. Therefore, in one preferred embodiment of the present invention, in the case where the agricultural work machine is a harvester having a harvesting unit for harvesting crops, the already harvested land is harvested by the harvesting unit. An area, and the uncut land is an area before the crop is harvested by the harvesting unit . By using this field harvesting information, specific objects detected in areas submerged in crops and specific objects detected in areas above already harvested fields can be regarded as erroneous detections. , and those regions can be masked. Also, objects detected in areas far from the ridge do not become obstacles to travel, so such areas can also be masked. As a result, it is possible to quickly and accurately detect running obstacles such as people and animals and avoid the aircraft from interfering with them.

In one of the preferred embodiments of the present invention, a caution area setting unit for setting an obstacle caution area defined based on the position from the aircraft, a vehicle speed change command output unit for changing the vehicle speed (object detection signal generation (an example of part) , and when the traveling obstacle exists in the obstacle warning area, the vehicle speed change command output part changes the vehicle speed based on the specific object distance. According to this configuration, when it is determined that the detected specific object, that is, the traveling obstacle exists in the obstacle warning area, the vehicle speed is changed according to the positional relationship between the traveling obstacle and the aircraft body. Therefore, it is possible to change the degree of vehicle speed change depending on whether there is a high possibility of interference within a predetermined period of time due to the positional relationship between the machine body and the traveling obstacle, and when the possibility of interference is low. At that time, for example, if there is a high possibility of interference within a relatively short time due to the positional relationship between the aircraft and the traveling obstacle, changing the vehicle speed to zero, that is, stopping the aircraft. Become. As a result, even when the possibility of interference within a relatively short period of time is small due to the positional relationship between the machine body and the traveling obstacle, it is possible to reduce the possibility of stopping the machine body and lowering workability.

It is the whole combine side view. FIG. 4 is a schematic diagram showing a specific object detection range by a camera mounted on the front part of the body; FIG. 3 is a schematic diagram showing the structure of a segmentation neural network; 4 is a schematic flow chart of processing for detecting a specific object and determining the spatial position of the specific object; Fig. 5 is a flow diagram showing a variant of the flow diagram according to Fig. 4; FIG. 4 is a plan view showing a first caution area and a second caution area used for caution determination of a detected traveling obstacle; It is a functional block diagram of a control system in a combine. 4 is a flowchart showing the flow of traveling obstacle avoidance control;

Hereinafter, an embodiment of a combine as a harvesting machine, which is an example of an agricultural working machine according to the present invention, will be described with reference to the drawings. Agricultural crops to be harvested are planted culms such as wheat and rice. In this embodiment, when defining the longitudinal direction of the machine body 1, it is defined along the traveling direction of the machine body in the working state. The direction indicated by symbol (F) in FIG. 1 is the front side of the fuselage, and the direction indicated by symbol (B) in FIG. 1 is the rear side of the fuselage. When defining the left-right direction of the airframe 1, the left-right is defined as viewed in the traveling direction of the airframe. “Upper (above)” or “lower (lower)” is a positional relationship in the vertical direction (vertical direction) of the airframe 1, and indicates a relationship at ground level.

As shown in FIG. 1, the combine has a harvesting section 2, which is a harvesting section, connected to a front portion of a machine body 1 having a pair of left and right crawler traveling devices 10 so as to be vertically movable around a horizontal axis. The body 1 can turn left and right due to the speed difference between the left and right crawler traveling devices 10 . A rear portion of the machine body 1 is provided with a threshing device 11 and a grain tank 12 for storing grains arranged side by side in the width direction of the machine body. A boarding/operating section 14 is provided on the right front portion of the body 1, and an engine (not shown) is provided below the boarding/operating section 14. As shown in FIG.

As shown in FIG. 1, the threshing device 11 receives inside the harvested grain culm that has been harvested by the harvesting unit 2 and conveyed backward, and the base of the grain culm is clamped by the feed chain 111 and the clamping rail 112. The tip side is threshed by a threshing cylinder 113 while being conveyed. Grain sorting processing for the threshed material is executed in the sorting unit provided below the threshing drum 113, and the sorted grains are conveyed to the grain tank 12 and stored. Although not described in detail, a grain discharging device 13 for discharging grains stored in the grain tank 12 to the outside is provided.

The reaping unit 2 is equipped with a clipper-type cutting device 22 for cutting the base of the erected planted stalk, a stalk conveying device 23, and the like. The culm conveying device 23 conveys the harvested culms of which the base has been cut from the vertical posture toward the starting end of the feed chain 111 while gradually changing the posture to the lying posture.

A satellite positioning module 80 is also provided on the ceiling of the boarding operation section 14 . Satellite positioning module 80 includes a satellite antenna for receiving global navigation satellite system (GNSS) signals (including GPS signals). To complement the satellite navigation provided by the satellite positioning module 80, an inertial navigation unit consisting of gyroscopic accelerometers and magnetic heading sensors may be incorporated into the satellite positioning module 80. FIG. Of course, the inertial navigation unit can be placed elsewhere. An optical sensor unit 9 used for obstacle detection is provided at the front of the combine. The optical sensor unit 9 includes a photographing unit 9A which is an RGB camera, a stereo vision camera 9B capable of depth measurement, and an illuminator 9C.

As shown in FIG. 2, each camera of the optical sensor unit 9 is set to have a field of view of a specific area in front of the traveling direction of the machine body 1 in a field. In this embodiment, the horizontal viewing angle, vertical viewing angle, and oblique viewing angle of the stereo vision camera 9B are approximately 100°, 70°, and 100°, respectively.

As shown in FIG. 3, the photographed image output from the photographing unit 9A is used as input data for the neural network 4. As shown in FIG. The neural network 4 receives a captured image, recognizes a specific object appearing in the captured image, and outputs object area information indicating the area occupied by the specific object in the captured image.

In this embodiment, the neural network 4 is configured as a segmentation neural network consisting of an encoder network 4A and a decoder network 4B. The encoder network 4A consists of an input layer 41 and a plurality of hidden layers 42, and the decoder network 4B consists of a plurality of hidden layers 42 and an output layer 43. FIG. Hidden layer 42 includes a convolutional layer and a pooling layer. The output data output from the segmentation type neural network 4 is a label image for the input photographed image, and each pixel thereof is assigned a label identifying the detected (separated) specific object. . In FIG. 3, dark and light colors are used as the labels, and planted grain culms (unharvested land) and harvested traces (harvested land) as specific objects are distinguished by the dark and light colors.

Furthermore, in this embodiment, to speed up the neural network 4, the hidden layer 42 is interposed with a batchNormalizationLayer 44 for pruning based on the parameters in the batchnormalization layer 44. ) is performed. This pruning reduces the number of units in the hidden layer 42 based on the parameters in the batch normalization layer 44, as described below.

とし、
出力を、

とすると、

となる。
ここで用いられたγがユニット削減のパラメータとして用いられる。つまり、学習処理において、パラメータ：γが小さいユニットが削除される。 For the discussion of pruning here, let the input vector to the batch normalization layer be

year,
the output as

and

becomes.
γ used here is used as a parameter for unit reduction. That is, in the learning process, units with a small parameter γ are deleted.

FIG. 4 is a schematic flowchart of the process of detecting a specific object and determining the spatial position of the specific object using the captured image from the capturing unit 9A of the optical sensor unit 9 and the stereo vision image from the stereo vision camera 9B. shown in

First, on the one hand, when the photographed image from the photographing unit 9A is input to the learned neural network 4, it is estimated whether or not a specific object exists in the photographed image, and when the specific object is estimated, the estimation is performed. Object area information indicating the area occupied by the specified object is output (#a). In FIG. 4, the object region information is shown as a label image in which people, planted culms (unharvested land), harvested trails (harvested land), and ridges are classified. The contour of the person, which is the area occupied by the detected person, can be obtained from this label image.

On the other hand, based on the stereo vision image from the stereo vision camera 9B, the spatial position measurement unit 52 uses stereo matching technology to determine the spatial positions of all objects existing in the specific area (depth data consisting of many distance measurement points group) is generated. The object position information is output as an image called a depth image in which the depth of each pixel is represented by a grayscale color in FIG. 4 (#b). Each pixel corresponds to a stereo matching point, and since these point groups have three-dimensional coordinate values, they are called three-dimensional point group data and can be developed in a three-dimensional coordinate space (specific area).

Next, based on the object area information output from the neural network 4 and the object position information output from the spatial position measuring unit 52, the distance from the aircraft 1 to the person or animal (human in FIG. 4) as a traveling obstacle is detected. is calculated by the distance calculator 53 (#c). In this embodiment, the specific object distance is calculated from the center of gravity of the human contour (indicated by P1 in FIG. 4) calculated using a large number of distance measurement points included in the detected human contour. distance to 1. As the reference point for the specific object distance, instead of the center of gravity of the contour, a statistical representative value (mean value, median value, mean value, etc.) of a large number of distance measurement points included in the contour may be used.

In the above description of FIG. 4, the object position information is directly given to the distance calculator 53 and used to calculate the specific object distance. As another embodiment, instead of the method shown in FIG. 4, a method as shown in FIG. 5 may be adopted. In the method shown in FIG. 5, the field information creating unit 54 creates three-dimensional point cloud data developed in a three-dimensional coordinate system with the field field as one reference plane (bottom plane) from the object position information (# c1). Furthermore, the farm field information creation unit 54 creates farm field harvest information including ridges, planted culms, harvest marks, and boundaries between already harvested and uncut land from the three-dimensional point cloud data and object region information ( #c2 and #c3). Based on the object position information, the object area information, and the field reaping information, the distance calculation unit 53 determines a specific object that is a traveling obstacle, and calculates the center of gravity of the determined specific object (indicated by P1 in FIG. 5). (#c).

If there is a possibility that the detected traveling obstacle will interfere with the airframe 1, control is performed to avoid interference with the traveling obstacle. The obstacle warning area used for this purpose is defined on a reference plane (XY plane) corresponding to a field spreading forward in the traveling direction of the machine body 1, as shown in FIG. Here, the obstacle caution area is divided into a first caution area (shown darkly in FIG. 6 and denoted by symbol Z1) and a second caution area (shown lightly in FIG. is given the reference Z2).

The first warning area is a rectangle defined by the working width of the reaping unit 2 and the warning distance from the machine body 1 to the front of the machine body 1 in the direction of travel (indicated by symbol L1 in FIG. 6). and a wedge-shaped area obtained by cutting out a blind spot area determined by the angle of view of the stereo vision camera 9B. In other words, the first warning area is defined by a rectangle having one side of the left and right width of the reaping unit 2 which is substantially the width of the running path of the combine harvester, and the other side of which is a predetermined distance (warning distance) forward in the traveling direction from the machine body 1. , is a shape obtained by cutting out the blind area determined by the angle of view of the camera. A traveling obstacle existing in the first caution area comes into contact with the combine harvester as the combine harvester advances. Therefore, the caution distance is preferably changed according to the vehicle speed of the combine harvester. In this embodiment, the work traveling speed is substantially constant, so it is set to several meters.

The second warning area has a shape obtained by cutting out the first warning area from a tip wedge-shaped area obtained by cutting out the blind spot area determined by the angle of view of the camera on the plane extending forward in the traveling direction of the airframe 1 . It has a width about three times that of the first caution area in the horizontal direction of the combine harvester, and a length slightly less than two times that of the first caution area in the forward direction of movement of the machine body 1 (indicated by L2 in FIG. 6). ). The width of the second caution area in the left-right direction is set in consideration of the turn of the combine harvester, and the width of the traveling direction of the second caution area is set so that there is enough time before the machine body 1 interferes with a traveling obstacle. It is set to be larger than the warning area.

FIG. 7 shows a functional block diagram of the control system of the combine. The control system of this embodiment is composed of a large number of electronic control units called ECUs, various operating devices, a group of sensors, a group of switches, and a wiring network such as an in-vehicle LAN for data transmission between them. The notification device 84 is a device for notifying a driver, a manager in the vicinity of the field, and the like of warnings such as the detection result of traveling obstacles and the state of work traveling, and includes a buzzer, a lamp, a speaker, and a display. When the detected traveling obstacle exists in the first warning area, a warning sound, a warning light, or a warning message indicating an imminent state is issued as emergency warning information, and the obstacle exists in the first warning area. If it is, a warning sound, warning light, or warning message that is milder than the emergency alert notification is issued as the warning notification.

A communication unit 85 is used to exchange data with an external communication device. A control unit 6 is a core element of this control system and is shown as a collection of multiple ECUs. Positioning data from the satellite positioning module 80 and image data from the optical sensor unit 9 are input to the control unit 6 through a wiring network.

The control unit 6 has an output processing section 6B and an input processing section 6A as input/output interfaces. The output processing unit 6B is connected to the vehicle running equipment group 7A and the working device equipment group 7B. The vehicle running device group 7A includes control devices related to vehicle running, such as an engine control device, a shift control device, a braking control device, a steering control device, and the like. The working device device group 7B includes the power control device for the reaping unit 2, the threshing device 11, the grain discharging device 13, the grain culm conveying device 23, and the like.

A travel system detection sensor group 8A, a work system detection sensor group 8B, and the like are connected to the input processing unit 6A. The traveling system detection sensor group 8A includes sensors for detecting states of an engine speed adjuster, an accelerator pedal, a brake pedal, a shift operation tool, and the like. The work system detection sensor group 8B includes sensors for detecting the device state of the reaping unit 2, the threshing device 11, the grain discharging device 13, and the grain culm conveying device 23, and the state of the culms and grains.

The control unit 6 includes a work travel control module 60, an obstacle detection unit 50, a vehicle speed change command output section 65, a body position calculation section 66, and a notification section 67 as control function sections. It should be noted that these control function units are separately configured and connected via an in-vehicle LAN or the like.

The notification section 67 generates notification data based on requests from the respective functional sections of the control unit 6 and provides it to the notification device 84 . The body position calculation unit 66 calculates the body position, which is the map coordinates (or field coordinates) of the body 1, based on the positioning data sequentially sent from the satellite positioning module 80. FIG.

The combine of this embodiment can travel both automatically (automatic steering) and manually (manual steering). The work travel control module 60 includes a work travel instruction unit 63 and a travel route setting unit 64 in addition to the travel control unit 61 and the work control unit 62 . A travel mode switch (not shown) for selecting either an automatic travel mode in which the vehicle travels by automatic steering or a manual steering mode in which the vehicle travels by manual steering is provided in the boarding operation section 14 . By operating this travel mode switch, it is possible to shift from manual steering travel to automatic steering travel, or from automatic steering travel to manual steering travel.

The travel control unit 61 has an engine control function, a steering control function, a vehicle speed control function, and the like, and provides a travel control signal to the vehicle travel equipment group 7A. The work control unit 62 gives a work control signal to the work device equipment group 7B in order to control the movements of the reaping unit 2, the threshing device 11, the grain discharging device 13, the grain culm conveying device 23, and the like.

The travel route setting unit 64 develops a travel route for automatic travel in a memory. The travel routes developed in the memory are sequentially used as target travel routes in automatic travel. This travel route can also be used for guidance for the combine to travel along the travel route, even during manual travel.

The work travel command unit 63 generates an automatic steering command and a vehicle speed command as automatic travel commands and gives them to the travel control unit 61 . The automatic steering command is generated by the travel route setting unit 64 so as to eliminate azimuth and position deviations between the travel route and the position of the aircraft itself calculated by the aircraft position calculation unit 66 . During automatic driving, the vehicle speed command is generated based on a preset vehicle speed value. During manual running, the vehicle speed command is generated based on the manual vehicle speed operation. However, in the event of an emergency such as detection of a running obstacle, the vehicle speed is changed, including a forced stop, and the engine is automatically stopped.

When the automatic traveling mode is selected, based on the automatic traveling command given by the work traveling command section 63, the traveling control section 61 controls the vehicle traveling device group 7A related to steering and the vehicle traveling device group 7A related to vehicle speed. When the manual travel mode is selected, the travel control unit 61 generates a control signal based on the operation by the driver to control the vehicle travel device group 7A.

The obstacle detection unit 50 performs the object detection processing described using FIG. 4 or FIG. Therefore, the obstacle detection unit 50 includes an AI section 51 , a spatial position measurement section 52 , a distance calculation section 53 , a field information generation section 54 , an object detection signal generation section 55 and a caution area setting section 56 . The above-described components constituting the obstacle detection unit 50 are separated mainly for the purpose of explanation, and the integration of the components and the division of the components may be freely performed.

The AI unit 51 is equipped with a trained model of the segmentation neural network 4 described with reference to FIG. 3, and detects traveling obstacles (specific object detection) based on the principle described with reference to FIGS.

The spatial position measuring unit 52, the distance calculating unit 53, and the field information creating unit 54 have the functions described with reference to FIGS. 4 and 5. FIG. The caution area setting unit 56 sets an obstacle caution area defined based on the position with respect to the aircraft 1 . In this embodiment, the obstacle caution area is divided into the first caution area and the second caution area as described with reference to FIG.

The distance calculation unit 53 calculates the position of the detected traveling obstacle, especially the distance from the aircraft 1 to the traveling obstacle (specific object distance) in cooperation with the AI unit 51, the spatial position measurement unit 52, and the like. . The object detection signal generator 55 determines in which of the first warning area and the second warning area the detected traveling obstacle exists based on the distance from the aircraft 1 to the traveling obstacle. and generate an object detection signal for each warning area. This object detection signal includes a traveling obstacle notification signal given to the notification section 67 and an obstacle travel signal given to the vehicle speed change command output section 65. Further, the obstacle travel signal indicates that there is a traveling obstacle. Signals by alert area are included.

Vehicle speed change command output unit 65 outputs a first vehicle speed change command including a vehicle speed change value to work travel command unit 63 when the received obstacle travel signal indicates that a travel obstacle exists in the first warning area. If the traveling obstacle exists in the second warning area, the second vehicle speed change command including the vehicle speed change value is output to the work travel command section 63 . The first vehicle speed change command is a command to set the vehicle speed to zero, that is, a command to stop the machine body 1 . Of course, the first vehicle speed change command may be a command to bring the vehicle speed to a value close to zero, or a command to gradually bring the vehicle speed to zero. The second vehicle speed change command is a deceleration value for decelerating the vehicle speed to a very slow speed, for example, about 1 km/h. The second vehicle speed change command may be a deceleration value determined by a ratio to the immediately preceding vehicle speed, such as decelerating to a fraction of the immediately preceding vehicle speed. In any case, the first vehicle speed change value by the first vehicle speed change command is larger than the second vehicle speed change value by the second vehicle speed change command, and is a command for emergency evacuation. Based on the vehicle speed change command given from the vehicle speed change command output unit 65, the work travel command unit 63 gives a control command to the travel control unit 61 to decelerate the machine body 1 urgently. When the presence of the traveling obstacle shifts from the first caution area to the second caution area, the machine body 1 that has stopped moves forward at a slow speed defined by the second vehicle speed change command. Furthermore, when the traveling obstacle is no longer detected in the warning area, the vehicle returns to traveling at the normal working vehicle speed.

Next, FIG. 8 shows an example of a control routine for decelerating the vehicle speed when an obstacle is detected in the control system described above. When this control routine starts, initialization such as resetting of flags and variables is performed (#01). A caution area is set, that is, a first caution area and a second caution area are set (#02). When the first caution area is changed depending on the vehicle speed, the current vehicle speed is acquired and the first caution area is set based on the vehicle speed.

Image data (photographed image) is taken in from the optical sensor unit 9 (#03). A specific object detection process (running obstacle detection process) is performed by the obstacle detection unit 50 using the captured image data (#04).

It is determined whether or not a running obstacle is detected, and if a running obstacle is detected (#05 Yes branch), the position of the running obstacle in the obstacle warning area is calculated (#11).

It is checked whether a running obstacle is in the first warning area (#12). If the traveling obstacle is in the first warning area (#12 Yes branch), the vehicle speed change command output unit 65 gives a stop command to the work travel command unit 63 as the first vehicle speed change command (#13). Furthermore, the stop flag is turned ON (#14). In step #14, when the deceleration flag is ON (when the traveling obstacle moves from the second warning area to the first boundary area), the deceleration flag is turned OFF and the stop flag is turned ON. As a result, the machine body 1 stops and the control returns to step #02.

If it is determined in step #12 that the obstacle is not in the first caution area (#12, No branch), it is further checked whether it is in the second caution area (#20). If the traveling obstacle is in the second warning area (#20 Yes branch), the vehicle speed change command output unit 65 gives a deceleration command to the work travel command unit 63 as the second vehicle speed change command (#21). Further, the deceleration flag is turned ON (#22). In step #22, when the stop flag is ON (when the traveling obstacle moves from the first warning area to the second boundary area), the stop flag is turned OFF and the deceleration flag is turned ON. As a result, the aircraft 1 runs at a slow speed (approximately 1 km/h), and the control returns to step #02.

If it is determined in step #20 that the obstacle is not in the second warning area (#20, No branch), it means that there is no obstacle in the warning area. In this way, when there is no traveling obstacle in the warning area, or when no traveling obstacle is detected in the first place (#08 No branch), the states of the stop flag and the deceleration flag are checked, and the states are checked. The corresponding flag contents are rewritten. Specifically, at that time, if the stop flag is ON (#30 Yes branch), the stop flag is rewritten to OFF (#31), the travel command becomes effective, and the body 1 performs normal travel ( #32). Control then returns to step #02. At that time, if the stop flag is not ON (#30 No branch), the content of the deceleration flag is checked (#40). If the deceleration flag is ON (#40 Yes branch), the deceleration flag is rewritten to OFF (#41), the travel command becomes effective, and the body 1 travels normally (#42). Normal travel is work travel when no travel obstacle or the like is detected, and the vehicle speed is higher than the vehicle speed when the travel obstacle exists in the second warning area. Control then returns to step #02. If the deceleration flag is OFF (#40, No branch), of course the travel command is effective and normal travel is performed (#43). The control then returns to step #02.

By controlling the detection of traveling obstacles and the change of vehicle speed in this manner, for example, when a field worker enters a warning area in front of the machine 1 in the traveling direction, the machine 1 stops or decelerates, and the field worker leaves the warning area. Then, the combine returns to work running at normal vehicle speed.

This control of vehicle speed deceleration based on the detection of traveling obstacles can be executed whether the combine is traveling automatically or traveling manually.

In the above-described embodiment, the first warning area and the second warning area are set, and when a traveling obstacle exists in each of the warning areas, the machine body 1 is stopped or the vehicle speed is changed to slower than normal driving. rice field. Alternatively, three or more warning areas may be provided, and the aircraft 1 may be stopped or slowed down according to each area. Furthermore, the caution area may be set steplessly, and the vehicle body 1 may be stopped or the vehicle speed may be changed steplessly to a speed lower than that of normal running, depending on the distance between the vehicle body 1 and the traveling obstacle.

It should be noted that the configurations disclosed in the above embodiments (including other embodiments, the same shall apply hereinafter) can be applied in combination with configurations disclosed in other embodiments as long as there is no contradiction. The embodiments disclosed in this specification are exemplifications, and the embodiments of the present invention are not limited thereto, and can be modified as appropriate without departing from the object of the present invention.

INDUSTRIAL APPLICABILITY The present invention is applied to agricultural working machines such as harvesters, rice transplanters, and totactors that travel for work in fields.

1: Airframe 2: Reaping unit 4: Segmentation neural network (neural network)
4A: Encoder network 4B: Decoder network 41: Input layer 42: Hidden layer 43: Output layer 44: Batch normalization layer 50: Obstacle detection unit 51: AI unit 52: Spatial position measurement unit 53: Distance calculation unit 54: Field Information creation unit 55 : Object detection signal generation unit 56 : Warning area setting unit 6 : Control unit 61 : Travel control unit 63 : Work travel command unit 65 : Vehicle speed change command output unit 67 : Notification unit 9 : Optical sensor unit 9A : Photographing Unit 9B: Stereo vision camera

Claims (7)

An agricultural work machine for working and traveling in a field,
a photographing unit for photographing a specific area of the field;
A neural network that receives a photographed image from the photographing unit, recognizes a specific object appearing in the photographed image, and outputs object area information including a specific object contour indicating an area occupied by the specific object in the photographed image. and,
a spatial position measuring unit that captures an image of the specific area and generates object position information indicating the spatial positions of all objects existing in the specific area;
3D point cloud data developed in a 3D coordinate system with the field surface as a reference plane is created from the object position information, and the 3D point cloud data included inside the specific object contour and the specific object contour are generated. a field information creating unit for creating field harvesting information including a boundary between an uncut land that is one of the specific objects on which the crops are present and a harvested land that is one of the specific objects from which the crops have been cut. and,
determining a spatial position of a traveling obstacle, which is one of the specific objects , based on the object position information, the object area information, and the field reaping information; a distance calculation unit that calculates
and an object detection signal generator that outputs an object detection signal based on the specific object distance. 2. The agricultural working machine according to claim 1, wherein said neural network is composed of a segmentation neural network consisting of an encoder network and a decoder network, and said object area information includes an outline of said specific object. 3. The hidden layers of the encoder network and the decoder network of claim 2, comprising a convolutional layer, a pooling layer and a batch normalization layer, wherein pruning is performed based on parameters in the batch normalization layer. farm machine. 3. The specific object distance is a distance from the center of gravity of the traveling obstacle to the airframe calculated from the contour of the traveling obstacle , or a statistical representative value of distance measurement points included in the contour. 3. The farm working machine according to 3. The agricultural work machine according to any one of claims 1 to 4, wherein the spatial position measurement unit generates the object position information using a stereo matching technique for the specific area. The agricultural machine is a harvester having a harvesting unit for harvesting crops,
6. The harvesting unit according to any one of claims 1 to 5, wherein the harvested land is an area where the crops have been harvested, and the uncut land is an area before the crops have been harvested by the harvesting unit. agricultural machinery. A warning area setting unit that sets an obstacle warning area defined based on the position from the aircraft, and a vehicle speed change command output unit that changes the vehicle speed,
The agricultural work machine according to any one of claims 1 to 6, wherein the vehicle speed change command output unit changes the vehicle speed based on the specific object distance when the traveling obstacle exists in the obstacle warning area.

Images