JP7535442B2 - Vehicle assistance method and vehicle assistance device - Google Patents

Description

The present invention relates to a vehicle assistance method and a vehicle assistance device.

Road sign recognition technology using camera images has been known for some time (Patent Document 1). The invention described in Patent Document 1 performs template matching on camera images to recognize road signs, which are road structures.

JP 2015-191281 A

However, in areas with low image resolution, such as areas far from the vehicle, template matching does not produce the desired results.

The present invention has been made in consideration of the above problems, and its purpose is to provide a vehicle assistance method and vehicle assistance device that can improve the recognition accuracy of road structures in areas where the image resolution is low.

A vehicle assistance method according to one aspect of the present invention extracts a repeating pattern area, which is an area in which a repeating pattern having spatial frequency components equal to or higher than a predetermined frequency exists, from each of a plurality of images captured at different times; for the entire area of the plurality of images captured at a predetermined time, a region obtained by excluding the area above the center of the image from the repeating pattern area, which is at infinity in the image, is set as a reference area; for the entire area of the plurality of images captured after the predetermined time, a region obtained by excluding the area above the center of the image from the repeating pattern area, which is at infinity in the image, is set as a comparison area; an amount of movement on the image is calculated using an overlapping area between the reference area and the comparison area, the plurality of images are aligned based on the amount of movement, super-resolution processing is performed using the aligned plurality of images, a super-resolution image that exceeds the resolution of the image captured by the camera is generated, and road structures are recognized based on the super-resolution image.

The present invention makes it possible to improve the recognition accuracy of road structures in areas where the image resolution is low.

FIG. 1 is a configuration diagram of a vehicle assistance device 1 according to a first embodiment of the present invention. FIG. 2 is a diagram for explaining an example of a method for extracting a repeated pattern area. FIG. 3 is a diagram for explaining an example of a method for dividing an image. FIG. 4 is a diagram illustrating an example of a frequency analysis result. FIG. 5 is a diagram for explaining an example of a method for extracting a repeated pattern area. FIG. 6 is a diagram for explaining an example of a method for extracting a repeated pattern area. FIG. 7 is a diagram illustrating an example of the super-resolution processing. FIG. 8 is a flowchart illustrating an example of the operation of the vehicle assistance device 1. FIG. 9 is a flowchart illustrating an example of the operation of the vehicle assistance device 2. FIG. 10 is a configuration diagram of a vehicle assistance device 3 according to a third embodiment of the present invention. FIG. 11 is a diagram illustrating an example of a region of interest 70. As shown in FIG. FIG. 12 is a flowchart illustrating an example of the operation of the vehicle assistance device 3.

Embodiments of the present invention will be described below with reference to the drawings. In the description of the drawings, identical parts are given the same reference numerals and the description will be omitted.

An example of the configuration of the vehicle assistance device 1 will be described with reference to FIG. 1. As shown in FIG. 1, the vehicle assistance device 1 includes a camera 10, a storage device 11, a controller 20, a steering actuator 12, an accelerator pedal actuator 13, and a brake actuator 14.

The vehicle assistance device 1 may be mounted on a vehicle that has an automatic driving function, or may be mounted on a vehicle that does not have an automatic driving function. The vehicle assistance device 1 may also be mounted on a vehicle that is capable of switching between automatic driving and manual driving. The automatic driving function may also be a driving assistance function that automatically controls only some of the vehicle control functions, such as steering control, braking force control, and driving force control, to assist the driver in driving. In this embodiment, the vehicle assistance device 1 will be described as being mounted on a vehicle that has an automatic driving function.

The camera 10 has an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The location of the camera 10 is not particularly limited, but as an example, the camera 10 is installed in front, on the side, or behind the vehicle. The camera 10 continuously captures images of the surroundings of the vehicle at a predetermined cycle. In other words, the camera 10 acquires multiple images captured at different times. The camera 10 detects objects (pedestrians, bicycles, motorcycles, other vehicles, etc.) present around the vehicle, and information in front of the vehicle (demarcation lines, traffic lights, signs, crosswalks, intersections, etc.). The images captured by the camera 10 are stored in the storage device 11. Hereinafter, the images captured at different times T1, T2, ..., Tn are referred to as images I1, I2, ..., In.

The storage device 11 mainly stores images captured by the camera 10. As an example, the storage device 11 is composed of a HDD (Hard Disk Drive) and an SSD (Solid State Drive).

The controller 20 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memory, and input/output units. A computer program for functioning as the vehicle assistance device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as multiple information processing circuits equipped in the vehicle assistance device 1. Note that here, an example is shown in which multiple information processing circuits equipped in the vehicle assistance device 1 are realized by software, but it is of course possible to configure the information processing circuits by preparing dedicated hardware for executing each information process shown below. The multiple information processing circuits may also be configured by individual hardware. The controller 20 includes a repeating pattern area extraction unit 21, an image shift unit 22, a super-resolution processing unit 23, a road structure recognition unit 24, and a vehicle control unit 25 as examples of multiple information processing circuits.

The repeating pattern area extraction unit 21 divides the images I1, I2, ..., In captured by the camera 10 into small areas of arbitrary size, and if a repeating pattern exists in each small area, extracts that small area as a repeating pattern area. When calculating the amount of movement of the image, the repeating pattern area extraction unit 21 sets maximum values for the top, bottom, left, and right of the image. The maximum value is the largest value that is expected to move during one frame. If it is assumed that there are L pixels in all four directions, top, bottom, left, and right, the size of the small area must be at least (2L + 1) x (2L + 1). This is to prevent erroneous calculation of the amount of movement.

When the small regions thus divided are R1, R2, ..., Rn, frequency analysis of each small region can determine whether a repeating pattern exists. When a two-dimensional FFT (Fast Fourier Transform) is performed on the small region, the frequency intensity in the horizontal direction (hereafter x direction) and vertical direction (hereafter y direction) is detected two-dimensionally. Here, it is confirmed whether a repeating component exists in the x direction or y direction. In the x direction, all rows of the FFT result are added together for each x-direction frequency component. In the y direction, all columns of the FFT result are added together for each y-direction frequency component. From the frequency components in one direction obtained as a result, the maximum value H2 of the frequency intensity is compared with the previous minimum value H1 in a specified period (maximum value L or less), and if it is equal to or greater than an arbitrary ratio, the small region is extracted as a repeating pattern region.

Furthermore, after performing frequency analysis on multiple small regions in the image, if the repeating pattern region exceeds a certain ratio α of the entire region of image I1, the repeating pattern region extraction unit 21 sets the entire image I1 as the reference image, and sets the region excluding the repeating pattern region as the reference region. On the other hand, if the repeating pattern region is equal to or less than the certain ratio α of the entire region of the image, the repeating pattern region extraction unit 21 sets the entire image I1 as the reference image, and sets the entire region of image I1 as the reference region. The certain ratio α is the minimum ratio that affects the accuracy when performing super-resolution processing. When the movement amount is calculated for each small region, and the average movement amount for each small region is calculated as the movement amount for the entire region, and a maximum value L, which is the maximum error, is calculated only for the repeating pattern region, αL is obtained as the overall movement amount.

When the desired enlargement factor in super-resolution processing is N, measurements must be made with a resolution of 1/N, so an accuracy of 1/2N is required when rounding is taken into account. In order for αL to be smaller than 1/2N, α must be smaller than 1/2NL. For this reason, when performing super-resolution processing in an area of 1/2NL or more for the desired enlargement factor N and the maximum value L for the movement amount calculation, the accuracy of the super-resolution processing is likely to decrease. In order to calculate the movement amount in 1/N units for the desired enlargement factor N, at least N pixels are required when using the average movement amount of each pixel in the vertical direction, so if there are not N or more pixels in the vertical or horizontal directions of the reference area excluding the repeating pattern area, super-resolution processing is not performed.

The image shifting unit 22 calculates the amount of movement between the images, and aligns the images by moving them by the calculated amount of movement. The image movement amount is the amount of movement between two images. For example, SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), ZNCC (Zero-mean Normalized Cross-Correlation), etc. are used to calculate the amount of movement. Specifically, the image shifting unit 22 calculates the similarity by shifting one pixel at a time in the up, down, left, and right directions within the search range, and calculates the amount of movement to the place with the highest similarity as the amount of movement. A reference image is required to compare two images. Therefore, image I1 captured at time T1 is used as the reference image. If image I1 is called reference image R1, reference image R1 has A pixels in the horizontal direction and B pixels in the vertical direction. The representative coordinates of the center of reference image R1 are (X1, Y1).

Next, in image I2 captured at time T2, a predetermined search range is set based on coordinates (X1, Y1). For example, if S pixels are set up above, below, left, and right, the search range will be from (X1-S, Y1-S) to (X1+S, Y1+S). The coordinates of the center of image I2 are (a, b). Reference image R1 and image I2 are compared, and the center coordinates (aM, bM) with the highest pixel similarity become the destination of (X1, Y1). The destination of (X1, Y1) is called (X2, Y2).

Similarly, in image I3 captured at time T3, a search range is set based on (X2, Y2). Reference image R1 and image I3 are compared, and (X3, Y3), which is the destination of (X2, Y2), is set. At this time, image shifting unit 22 may use a predetermined number of frames to update the image at that time as reference image R1. In calculating the similarity, A x B pixels, which are all pixels of reference image R1, are used, and the luminance values of the same coordinates in reference image R1 and the comparison image are compared. The comparison images refer to images I2 and I3.

In the case of SAD, the sum of the absolute values of the luminance differences is calculated, and in the case of SSD, the sum of the squares of the luminance differences is calculated. Also, if the similarity is lower than the threshold value at any central pixel in the search range, the movement amount calculation is stopped. In this embodiment, in order to calculate the movement amount of the road structure without being influenced by buildings, guardrails, etc. that have repeating patterns around the vehicle, the repeating pattern area is excluded from the similarity calculation. Therefore, the area excluding the repeating pattern area extracted by the repeating pattern area extraction unit 21 in the reference image R1 is used as the reference area, and the area excluding the repeating pattern area extracted by the repeating pattern area extraction unit 21 in each comparison image is used as the comparison area, and the pixels where both areas overlap are used for the similarity calculation. As a result, the movement amount obtained is in pixel units. In order to calculate the movement amount of the pixel after the decimal point, an approximation calculation is performed using the similarity calculation result. For example, the method of non-patent document 1 is used for such an approximation calculation. Non-Patent Document 1 "Arai Motoki, Sumi Kazuhiko, and Matsuyama Takashi. Optimization of correlation functions and subpixel estimation methods in image block matching. Information Processing Society of Japan Research Report Computer Vision and Image Media (CVIM) 2004.40 (2004-CVIM-144) (2004): 33-40."

By using the method of Non-Patent Document 1, the amount of movement of the decimal point of the pixel is calculated. When the desired enlargement ratio in the super-resolution processing is N, the image shift unit 22 rounds the amount of movement to 1/N pixels. The image shift unit 22 sets the amount of movement (MX2, MY2) ... (MXn, MYn) from the center coordinates (X1, Y1) of time T1 at each time T2 ... Tn. After that, the image shift unit 22 approximately enlarges the images J1, J2 ... Jn, which have a size of A pixels in the horizontal direction and B pixels in the vertical direction and are centered on the center coordinates (X1, Y1), (X2, Y2) ... (Xn, Yn) at each time, by N times. This generates images K1, K2 ... Kn. Examples of the method of approximate enlargement include nearest neighbor interpolation (Nearest Neighbor), bilinear interpolation (Bilinear), and bicubic interpolation (Bicubic). The approximately enlarged image is returned by multiplying the amount of movement, which is in 1/N pixel units, by N. The image shifter 22 moves the comparison image relative to the reference image R1 so that the center coordinates match, thereby aligning them.

The super-resolution processing unit 23 performs super-resolution processing using the images aligned by the image shift unit 22 to generate an image with high resolution. Here, the number of images required for super-resolution processing is Z. The super-resolution processing unit 23 calculates the luminance value of each pixel by averaging the images K1, K2, ..., Kz that are enlarged N times by the image shift unit 22 and have the same center coordinates. However, if the super-resolution processing unit 23 cannot track from the reference image R1 when Z images are not collected, or if the area to be subjected to super-resolution processing is filled with moving objects, the super-resolution processing unit 23 stops the super-resolution processing. The super-resolution processing is a technique for generating an image with high resolution by referring to multiple images. In this embodiment, the reference image R1 is aligned so that the comparison image is superimposed on the reference image R1, and the resolution of the reference image R1 is increased. In other words, an image with a resolution exceeding that of the image captured by the camera 10 is generated. The image generated by the super-resolution processing unit 23 is output to the road structure recognition unit 24.

The road structure recognition unit 24 recognizes road structures using the images input from the super-resolution processing unit 23. In this embodiment, road structures are defined as stationary objects. Specifically, road structures include structures such as lanes, stop lines, pedestrian crossings, road markings such as arrows, shoulders, curbs, signs, and traffic lights. One example of a method for recognizing road structures is semantic segmentation, which recognizes what type of object each pixel is. The road structure recognition unit 24 outputs the recognition results to the vehicle control unit 25.

The vehicle control unit 25 controls the steering actuator 12, accelerator pedal actuator 13, and brake actuator 14 using the road structures recognized by the road structure recognition unit 24. Stationary object areas become clearer through super-resolution processing, thereby improving the recognition accuracy of road structures.

Next, an example of a method for extracting a repeating pattern area will be explained with reference to Figures 2 to 7.

Image 40 shown in FIG. 2 is an image captured by camera 10 while the host vehicle is traveling. For convenience of explanation, an example of a method for extracting a repeating pattern area will be explained using image 50, which is a portion of image 40 shown in FIG. 2. Of course, the repeating pattern area may be extracted from the entire image 40 by the method described below. Image 40 is assumed to be an image captured at time T1.

As shown in FIG. 3, the repeating pattern area extraction unit 21 divides the image 50 captured by the camera 10 into small areas of an arbitrary size. In FIG. 3, the small areas are indicated by the symbols 50a to 50p. In FIG. 3, the image 50 is divided into 16 small areas. Next, the repeating pattern area extraction unit 21 determines whether or not a repeating pattern exists in each of the small areas 50a to 50p. Specifically, the repeating pattern area extraction unit 21 performs a two-dimensional FFT on each of the small areas 50a to 50p. An example of the two-dimensional FFT result is shown in FIG. 4. The result shown in FIG. 4 is the result for the x direction of an arbitrary small area. The vertical axis of FIG. 4 indicates the magnitude of the frequency component, and the horizontal axis indicates the frequency. As shown in FIG. 4, from the frequency component in one direction (here, the x direction), a maximum value H2 indicating the frequency intensity is compared with the previous minimum value H1 in a predetermined period (maximum value L or less), and if the ratio is equal to or greater than an arbitrary ratio, the small area is extracted as a repeating pattern area.

An example of the results of extracting a repeating pattern area is shown in Figure 5. The area surrounded by reference numeral 51 in Figure 5 is an area in which a repeating pattern having spatial frequency components equal to or higher than a predetermined frequency exists. In Figure 5, nine small areas, 50b, 50c, 50d, 50f, 50g, 50h, 50j, 50k, and 50i, have been extracted as repeating pattern areas. Note that a stationary building exists in the area surrounded by reference numeral 51.

In image 50 shown in FIG. 5, the repeating pattern area exceeds a certain ratio α of the entire area of image 50, so the repeating pattern area extraction unit 21 sets the entire image 50 as a reference image. In other words, the repeating pattern area extraction unit 21 sets image 50 itself as a reference image.

The repeating pattern area extraction unit 21 also sets the area excluding the repeating pattern area as the reference area. The area excluding the repeating pattern area is the range surrounded by the reference symbol 52 in FIG. 6. In other words, the seven small areas 50a, 50e, 50i, 50m, 50n, 50o, and 50p are the area excluding the repeating pattern area.

The same processing as above is performed on images captured after time T2. Images captured after time T2 are images used for alignment with image 40 captured at time T1, and correspond to the comparison image described above. The area in this comparison image excluding the repeating pattern area is defined as the comparison area. If the comparison area of the image captured at time T2 is comparison area 2, and the comparison area of the image captured at time T3 is comparison area 3, the image shifting unit 22 calculates the similarity between the reference area and comparison area 2, and calculates the similarity between the reference area and comparison area 3. If there are images captured after time T4, the similarity is calculated in a similar manner.

If the similarity is equal to or greater than the threshold, the image shifter 22 calculates the amount of movement on the image based on the similarity, and moves each comparison image relative to the reference image to align them. The super-resolution processor 23 then performs super-resolution processing using the images aligned by the image shifter 22. In this way, in this embodiment, the area excluding the repeating pattern area is used to calculate the similarity. In the super-resolution processing, the entire area including the repeating pattern area is used. As shown in FIG. 7, the image 50 becomes clear by such super-resolution processing. The cleared image is indicated by the reference symbol 53. In this way, according to this embodiment, the amount of movement between images can be calculated by excluding the repeating pattern area in the image, so that a high-resolution image can be generated. This improves the recognition accuracy of road markings, traffic lights, signs, etc. that are far from the vehicle. This allows the vehicle control unit 25 to quickly perform control appropriate for road markings, traffic lights, and signs, thereby realizing high-precision automatic driving.

Next, an example of the operation of the vehicle assistance device 1 will be described with reference to the flowchart in FIG.

In step S101, the surroundings of the vehicle are captured by the camera 10 mounted on the vehicle. The process proceeds to step S103, and if a reference image has been set (YES in step S103), the process proceeds to step S115. On the other hand, if a reference image has not been set (NO in step S103), the process proceeds to step S105.

In step S105, the repeating pattern area extraction unit 21 divides the image 50 captured by the camera 10 into small areas of any size (see Figure 3). The process proceeds to step S107, where the repeating pattern area extraction unit 21 performs a two-dimensional FFT on each of the small areas 50a to 50p (see Figures 3 and 4). If a repeating pattern area is present (YES in step S109), the repeating pattern area extraction unit 21 sets the entire image 50 as a reference image. The repeating pattern area extraction unit 21 also sets the area excluding the repeating pattern area as a reference area (step S111).

On the other hand, if no repeating pattern area exists (NO in step S109), the repeating pattern area extraction unit 21 sets the entire image 50 as the reference image. The repeating pattern area extraction unit 21 also sets the entire area of the image 50 as the reference area (step S113).

In step S115, the repeating pattern area extraction unit 21 performs a two-dimensional FFT on the image captured after time T2 in the same manner as the image captured at time T1. The process proceeds to step S117, where the repeating pattern area extraction unit 21 sets the area excluding the repeating pattern area in the image captured after time T2 as a comparison area. The process proceeds to step S119, where the image shifting unit 22 calculates the similarity between the reference area and the comparison area. As described above, the image shifting unit 22 calculates the similarity of the area where the reference area and the comparison area overlap within a predetermined range. The luminance values of the same coordinates of the reference area and the comparison area are compared as the similarity. If the similarity is equal to or greater than the threshold (YES in step S121), the process proceeds to step S123. On the other hand, if the similarity is less than the threshold (NO in step S121), the process ends.

In step S123, the image shifting unit 22 calculates the amount of movement between the images as described above, and aligns the comparison image by the calculated amount of movement. In step S125, the super-resolution processing unit 23 generates an image with high resolution using the image aligned by the image shifting unit 22. In step S127, the image generated by the super-resolution processing unit 23 is output to the road structure recognition unit 24. In step S129, the road structure recognition unit 24 recognizes road structures using the image input from the super-resolution processing unit 23. In step S131, the vehicle control unit 25 controls the steering actuator 12, accelerator pedal actuator 13, and brake actuator 14 using the road structures recognized by the road structure recognition unit 24. The super-resolution processing makes the image clearer. This improves the recognition accuracy of road structures.

(Action and Effect)
As described above, the vehicle assistance device 1 according to the first embodiment provides the following advantageous effects.

The vehicle assistance device 1 includes a camera 10 that captures images of the surroundings of the vehicle multiple times at different times, and a controller 20 that processes the multiple images captured by the camera 10 at different times. The controller 20 extracts a repeating pattern area, which is an area in which a repeating pattern having a spatial frequency component equal to or higher than a predetermined frequency exists, from each of the multiple images, and calculates the amount of movement on the image based on the image of the area excluding the repeating pattern area in each of the multiple images. The controller 20 aligns the multiple images based on the amount of movement, and performs super-resolution processing using the aligned multiple images. The controller 20 generates a super-resolution image that exceeds the resolution of the image captured by the camera, and recognizes road structures based on the super-resolution image. In this way, the vehicle assistance device 1 can calculate the amount of movement between images by excluding the repeating pattern area in the image, making it possible to generate a high-resolution image. This improves the recognition accuracy of road markings, traffic lights, signs, etc. that are far from the vehicle.

In the above-mentioned conventional technology, consider performing super-resolution processing to detect distant road structures that cannot be detected by template matching. If a repeating pattern is present in the area where super-resolution processing is performed, the accuracy of calculating the amount of movement decreases, and the amount of movement used for alignment in order to perform addition processing shifts. This makes it impossible to perform super-resolution processing on the road structure to be recognized, and there is a risk of the background becoming blurred. According to the first embodiment, since repeating pattern areas in the image are excluded, the amount of movement between images can be calculated without being influenced by buildings, guardrails, etc. that have repeating patterns. This makes it possible to generate high-resolution images.

The controller 20 also divides each of the multiple images into regions (small regions) of a predetermined size, and performs frequency analysis on the divided regions. The controller 20 determines whether or not a repeating pattern exists in the divided regions based on the frequency analysis results. This makes it possible to determine which small regions contain repeating patterns. In addition, because only periods below the maximum value L are processed, processing can be omitted for low frequencies that have little effect.

The controller 20 also sets the number of images Z to be used for super-resolution processing. This ensures the number of images required for the desired enlargement ratio. The controller 20 performs super-resolution processing when the number of images captured by the camera 10 is equal to or greater than the set number Z.

Second Embodiment
Next, a vehicle assistance device 2 (not shown) according to a second embodiment of the present invention will be described. The second embodiment differs from the first embodiment in the function of a repeated pattern area extraction unit 21. Configurations that overlap with those in the first embodiment will be referred to by reference numerals and their description will be omitted. The following description will focus on the differences.

The function of the repeat pattern area extraction unit 21 to extract a repeat pattern area is the same as in the first embodiment. The repeat pattern area extraction unit 21 extracts an area having a predetermined height in real space within the extracted repeat pattern area. An area having a predetermined height in real space is an area above the center of the image, which is at infinity in the image. By extracting such an area as an area having a predetermined height in real space, it is possible to reduce the influence of buildings and the like that are far away from the vehicle. As an alternative method, there is also a method of extracting a predetermined height in real space by measuring the distance using the camera 10.

The repeating pattern area extraction unit 21 sets the repeating pattern area excluding an area having a predetermined height in real space as the reference area. Note that the entire image I1 is set as the reference image in the same manner as in the first embodiment. The same processing is performed on images captured after time T2.

Next, an example of the operation of the vehicle assistance device 2 will be described with reference to the flowchart shown in FIG. 9. However, since the processes in steps S201 to S209 and S213 to S231 are similar to the processes in steps S101 to S109 and S113 to S131 shown in FIG. 8, their description will be omitted.

In step S211, the repeating pattern area extraction unit 21 sets an area obtained by excluding an area having a predetermined height in real space from the repeating pattern area as a reference area.

(Action and Effect)
As described above, according to the vehicle assistance device 2 according to the second embodiment, the following advantageous effects can be obtained.

The controller 20 sets, as a reference region, an area obtained by excluding areas having a predetermined height in real space from the repeating pattern area for all areas of the images captured at a predetermined time among the multiple images. The controller 20 sets, as a comparison region, an area obtained by excluding areas having a predetermined height in real space from the repeating pattern area for all areas of the images captured after the predetermined time among the multiple images. The controller 20 calculates the amount of movement on the image using the overlapping area between the reference region and the comparison region. This allows the amount of movement to be calculated excluding areas having a predetermined height in real space (areas other than the road surface, such as buildings, poles, and guardrails), improving the accuracy of calculating the amount of movement. The predetermined time is, for example, the above-mentioned time T1. After the predetermined time is, for example, the above-mentioned time T2 or later.

Third Embodiment
Next, a vehicle assistance device 3 according to a third embodiment of the present invention will be described with reference to Fig. 10. The third embodiment differs from the first embodiment in that the controller 20 further includes a region of interest setting unit 26. Configurations that overlap with those in the first embodiment will be denoted by reference numerals and will not be described in detail below. The following description will focus on the differences.

The region of interest setting unit 26 sets a region of interest. The region of interest is the area in which super-resolution processing is performed.

Next, an example of a region of interest will be described with reference to FIG. 11. Image 40 shown in FIG. 11 is the same as image 40 shown in FIG. 2. Reference numeral 70 in FIG. 11 indicates a region of interest. As can be seen from FIG. 11, region of interest 70 is smaller than image 40. In the first embodiment, super-resolution processing is performed on the entire image 40, but in the third embodiment, super-resolution processing is performed on region of interest 70. This reduces the computational load compared to the first embodiment.

The method of setting the region of interest 70 is not particularly limited, but for example, a region in which road markings, traffic lights, signs, etc. exist far from the vehicle is set as the region of interest 70. As described in FIG. 7, such distant regions are made clear by super-resolution processing. Super-resolution processing improves the recognition accuracy of road markings, traffic lights, signs, etc. far from the vehicle. This enables the vehicle control unit 25 to quickly perform control appropriate for road markings, traffic lights, and signs. This realizes highly accurate automated driving. The region of interest 70 may be set in the upper half of the image 40. The upper half of the image 40 means a region farther from the vehicle than the lower half. The region of interest 70 is also set in the same way for images captured after time T2.

Next, an example of the operation of the vehicle assistance device 3 will be described with reference to the flowchart shown in FIG. 12. However, since the processes in steps S301 to S303, S305 to S309, and S315 to S331 are similar to the processes in steps S101 to S103, S105 to S109, and S115 to S131 shown in FIG. 8, their description will be omitted.

In step S304, the region of interest setting unit 26 sets an area far from the vehicle where road markings, traffic lights, signs, etc. are present as the region of interest 70.

In step S311, the repeating pattern area extraction unit 21 sets the entire region of interest 70 as a reference image. The repeating pattern area extraction unit 21 also sets the region of interest 70 excluding the repeating pattern area as a reference region. In step S313, the repeating pattern area extraction unit 21 sets the entire region of interest 70 as a reference image. The repeating pattern area extraction unit 21 also sets the entire region of image 50 as a reference region.

(Action and Effect)
As described above, the vehicle assistance device 3 according to the third embodiment has the following advantages.

The controller 20 sets the area to be subjected to super-resolution processing for each of the multiple images as the region of interest 70. The controller 20 sets the region of interest 70 of an image captured at a specified time among the multiple images, excluding the repeating pattern region, as the reference region, and sets the region of interest 70 of an image captured after the specified time among the multiple images, excluding the repeating pattern region, as the comparison region. The controller 20 calculates the amount of movement on the image using the overlapping area between the reference region and the comparison region. This makes it possible to increase the resolution of only the region of interest (e.g., an area far from the vehicle) that is of interest in the image.

The desired magnification is defined as N, and the maximum amount of movement between one frame is defined as L. When the size of the repeating pattern area within the region of interest 70 is 1/(2N×L) or more, the controller 20 sets the area excluding the repeating pattern area as a reference area for the region of interest 70 in an image captured at a specified time. The controller 20 also sets the area excluding the repeating pattern area as a comparison area for the region of interest 70 in an image captured after the specified time. The controller 20 calculates the amount of movement on the image using the overlapping area between the reference area and the comparison area. As a result, when the size of the repeating pattern area is less than 1/(2N×L), super-resolution processing is omitted.

In addition, the controller 20 stops generating the super-resolution image if the number of pixels that are not in the repeating pattern area within the region of interest 70 is less than the square of N. This ensures the accuracy of the super-resolution image.

Each of the functions described in the above embodiments may be implemented by one or more processing circuits. Processing circuits include programmed processing devices, such as processors that include electrical circuitry. Processing circuits also include devices, such as application specific integrated circuits (ASICs) or circuit components, arranged to perform the described functions.

As described above, an embodiment of the present invention has been described, but the descriptions and drawings that form part of this disclosure should not be understood as limiting this invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

When the number of images required to perform super-resolution processing is set to Z as described above, images with center pixels aligned are stored for Z frames, and super-resolution processing is performed. However, if the number of images captured by camera 10 does not reach the set number Z, and if there is no area with a similarity equal to or greater than the threshold within the search range for calculating the amount of movement in images captured after a specified time, super-resolution processing unit 23 may perform super-resolution processing using images that have been aligned up to that time. This can improve the recognition accuracy of road structures.

In addition, if the number of images captured by the camera 10 is less than the set number Z, and if there is no area with a similarity equal to or greater than the threshold within the search range for calculating the amount of movement in images captured after the specified time, the super-resolution processor 23 may wait a specified time until the set number Z is collected before performing super-resolution processing. This allows super-resolution processing to be performed without interruption even if a moving object passes through the entire region of interest 70.

Reference Signs List 1 to 3 Vehicle assistance device 10 Camera 11 Storage device 12 Steering actuator 13 Accelerator pedal actuator 14 Brake actuator 20 Controller 21 Pattern area extraction unit 22 Image shift unit 23 Super-resolution processing unit 24 Road structure recognition unit 25 Vehicle control unit 26 Region of interest setting unit

Claims (6)

A vehicle assistance method comprising: a camera that captures an image of an area around a vehicle at different times; and a controller that processes the images captured by the camera at different times,
The controller:
extracting a repeating pattern area, which is an area in which a repeating pattern having a spatial frequency component equal to or higher than a predetermined frequency exists, from each of the plurality of images;
setting, as a reference region, an area obtained by excluding an area above a center of the image that is at infinity in the repeating pattern area from an entire area of an image captured at a predetermined time among the plurality of images;
setting, as a comparison region, an area obtained by excluding an area above a center of the image that is at infinity in the repeating pattern area from an entire area of the images captured after the predetermined time among the plurality of images;
Calculating a movement amount on the image using an overlapping area between the reference area and the comparison area;
aligning the images based on the amount of movement;
performing super-resolution processing using the aligned images to generate a super-resolution image having a resolution exceeding that of the image captured by the camera;
A vehicle assistance method comprising: recognizing road structures based on the super-resolution image. The controller:
Dividing each of the plurality of images into regions having a predetermined size;
Frequency analysis is performed on the divided regions.
The vehicle assistance method according to claim 1 , further comprising the step of determining whether or not a repetitive pattern exists in the divided regions based on a result of the frequency analysis. The controller:
Set the number of images to be used in the super-resolution processing;
The vehicle assistance method according to claim 1 , wherein the super-resolution processing is performed when the number of images captured by the camera is equal to or greater than a set number. The controller:
Set the number of images to be used in the super-resolution processing;
The vehicle assistance method described in claim 1, characterized in that if the number of images captured by the camera does not reach a set number, and if there is no area having a similarity greater than or equal to a threshold within the search range for calculating the amount of movement in images captured after a specified time, the super-resolution processing is performed using images that have been aligned up to that time. The controller:
Set the number of images to be used in the super-resolution processing;
The vehicle assistance method according to claim 1, characterized in that if the number of images captured by the camera is less than the set number, and if there is no area having a similarity greater than or equal to a threshold within the search range for calculating the amount of movement in images captured after a specified time, the super-resolution processing is delayed for a specified time until the set number of images is collected. A camera that captures images of the surroundings of the vehicle multiple times at different times;
a controller for processing a plurality of images captured by the camera at different times;
The controller:
extracting a repeating pattern area, which is an area in which a repeating pattern having a spatial frequency component equal to or higher than a predetermined frequency exists, from each of the plurality of images;
setting, as a reference region, an area obtained by excluding an area above a center of the image that is at infinity in the repeating pattern area from an entire area of an image captured at a predetermined time among the plurality of images;
a comparison region is set for an entire region of the images captured after the predetermined time among the plurality of images, the comparison region being obtained by excluding a region above a center of the image that is at infinity in the image from the repeating pattern region;
Calculating a movement amount on the image using an overlapping area between the reference area and the comparison area;
aligning the images based on the amount of movement;
performing super-resolution processing using the aligned images to generate a super-resolution image having a resolution exceeding that of the image captured by the camera;
A vehicle assistance device that recognizes road structures based on the super-resolution image.

Images