JP5872171B2 - Camera system - Google Patents

Description

  The present invention relates to a camera system that transmits an image captured by a camera device using a fisheye lens to a display device for display.

  The fish-eye lens has an angle of view (viewing angle) of about 180 °, and an almost horizontal field of view can be captured at once in an image sensor or the like. Therefore, since a wide range can be photographed with a small number of cameras (low cost), it is expected to be applied to surveillance cameras and video conference cameras. More recently, fisheye lenses have begun to be used in cameras that monitor the surroundings of vehicles. In a camera system using a wide-angle lens such as a fisheye lens, the following technology has been proposed.

  In the method of compressing imaging data disclosed in Patent Document 1, a plurality of pixel data within a predetermined circular range are allocated based on a rectangular imaging image including a circular image captured by a wide-angle lens, and rectangular imaging is performed. The method includes a step of generating a square reduced image having a smaller number of pixels than the image and a step of compressing the square reduced image. As a result, a rectangular captured image captured using a wide-angle lens is compressed after being converted into a reduced image having a smaller number of pixels than that, so the amount of data after compression is obtained by compressing the captured image itself. It is stated that it will be reduced compared to the case.

  The image presentation method disclosed in Patent Document 2 includes a step of inputting visual field information including a visual line direction and an angle of view in real time, and a plane having a width that is orthogonal to the visual line direction from the input visual field information and is determined by the angle of view. A step of calculating a drawing area, and a step of forming a normalized image in which the distortion is removed from the fisheye image in the drawing area by obtaining each pixel position in the fisheye image corresponding to each pixel position of the calculated drawing area Have As a result, it is stated that the target pixels to be normalized are limited, and high-speed moving image rendering becomes possible.

JP 2007-318597 A JP 2005-339313 A

  In general, a camera system includes a camera device (imaging device) and a display device that displays the camera device. However, when a fisheye lens is used for the camera device, it is necessary to display the image after correcting the distortion of the captured fisheye image. There is. In this case, image processing such as distortion correction is often executed in the display device. On the other hand, when the camera device and the display device are installed apart from each other, the image data must be transmitted from the camera device to the display device with a cable or the like. At this time, when the number of cameras increases, the amount of data to be transmitted also increases, and the load on the cable increases. In this regard, it is necessary to efficiently reduce and transmit image data by image compression or the like.

  In Patent Document 1, in order to convert to a reduced image with a small number of pixels, for example, an operation of moving pixels protruding from the left and right and below from a predetermined circular area in the captured image to the four corner areas of the quadrangular reduced image is performed. Is what you do. As a result, the amount of image data is reduced, and after that, compression processing is performed, but the distortion of the captured image by the fisheye lens remains. Further, Patent Document 1 is premised on compression using a still image compression technique, and does not consider the relevance to motion compensation used when compressing a moving image.

  The above Patent Document 2 describes an algorithm for forming a normalized image by removing image distortion from a fisheye image. Such processing is performed by an image presentation device that displays an image. That is, it is estimated that the image is transmitted from the fish-eye imaging device to the image presentation device in an uncompressed state, and no consideration is given to reducing the amount of transmitted data.

  As described above, in the conventional technique, it has been difficult to achieve both the image distortion correction accompanying the use of the fisheye lens and the reduction of the amount of transmitted data.

  An object of the present invention is to provide a camera system that reduces the amount of transmitted data by correcting image distortion caused by using a fisheye lens and efficiently compressing a moving image.

  The present invention is a camera system that transmits an image captured by a camera device to a display device for display, and the camera device has a lens unit that uses a fisheye lens and a distortion aberration image captured by the fisheye lens. A distortion correction unit that performs distortion correction processing so as to be small and an encoding unit that compresses and encodes an image output from the distortion correction unit. The display device includes a decoding unit that decompresses and decodes an image that has been compression-encoded by the encoding unit of the camera device, and a display unit that displays the image output from the decoding unit.

  ADVANTAGE OF THE INVENTION According to this invention, while correcting the image distortion of the image image | photographed with the fisheye lens, the load of the transmission line which transmits image data can be reduced.

The schematic diagram which shows one Example of the vehicle-mounted camera system to which the camera system by this invention is applied. The vehicle top view which shows the attachment position and viewing angle of the camera apparatus 2. FIG. 1 is a block diagram of an in-vehicle camera system (Example 1). The figure which shows the case where a compression process is performed without the distortion correction process by the distortion correction part 24 (comparative example). The figure which shows the case where a compression process is performed after the distortion correction by the distortion correction part 24. FIG. Block diagram of an in-vehicle camera system (Example 2).

  FIG. 1 is a schematic diagram showing an embodiment of an in-vehicle camera system to which a camera system according to the present invention is applied. The in-vehicle camera system has a configuration in which a camera device (imaging device) 2 and a display device 4 are attached to a vehicle 1 and both are connected by a cable 3. The camera device 2 uses a fish-eye lens, and creates an image that looks down on the entire vehicle from the top of the vehicle, with distortion correction, viewpoint conversion, pop-up detection, obstacle detection, white line recognition, sign recognition, and the like. Perform image processing such as The moving image data after image processing is transmitted to the display device 4 near the driver's seat via the cable 3 and displayed as driving support information for the driver. As will be described later, the camera device 2 includes an encoding unit, and the display device 4 includes a decoding unit, thereby compressing and encoding image data transmitted through the cable 3 to reduce its transmission bandwidth, and a relatively inexpensive Ethernet. (Registered trademark) cable can be adopted. The cable 3 may be another wired cable or may be transmitted wirelessly like WiFi (registered trademark).

  Next, using the in-vehicle camera system of FIG. 1, an example of correcting the rear side image distortion and displaying it on a car navigation monitor or the like so that the driver can confirm without any discomfort when changing the lane to the right side. explain.

FIG. 2 is a vehicle plan view showing the attachment position and the viewing angle of the camera device 2. The left direction of the drawing is the vehicle traveling direction. In this embodiment, a camera device 2 equipped with a fisheye lens with a viewing angle of approximately 180 ° is installed on the vehicle body side (near the side mirror) on the right side of the driver's seat.
Hereinafter, the internal configuration and operation of the in-vehicle camera system will be described in two embodiments.

  FIG. 3 is a block diagram of the in-vehicle camera system of FIG. The camera device 2 includes a lens unit 21, a sensor unit 22, a signal processing unit 23, a distortion correction unit 24, an encoding unit 25, and a transmission unit 26. The display device 4 includes a receiving unit 41, a decoding unit 42, and a display unit (monitor) 43. The cable 3 connects the transmission unit 26 of the camera device 2 and the reception unit 41 of the display device 4.

  First, the configuration of the camera device 2 will be described. The lens unit 21 condenses a wide range of external light using a fisheye lens, and converts the external light collected by the light receiving element in the sensor unit 22 into an electrical signal (image). The signal processing unit 23 performs signal processing for improving image quality such as removal of imaging noise, contrast correction, and white balance adjustment of the captured image.

  The distortion correction unit 24 performs distortion correction processing on the signal-processed image. Here, the distortion correction processing is a method of correcting an image with a specific distortion aberration photographed with a fisheye lens so that the distortion aberration is reduced. As an example, a method described in Patent Document 2 may be used. .

  The encoding unit 25 performs data compression and encoding processing (encoding processing) of the image subjected to distortion correction processing. Here, as a compression method, MPEG2 or H.264 is used. A known compression method such as H.264 can be used. The transmission unit 26 packetizes the compressed and encoded image data suitable for the transmission method (for example, an RTP packet in the case of an Ethernet (registered trademark) cable) and outputs the packet data to the cable 3.

  Next, the configuration of the display device 4 will be described. The receiving unit 41 performs depacketization processing on the packetized image data received from the cable 3. The decoding unit 42 performs decompression decoding (decoding processing) of the image data by a method corresponding to the compression method of the encoding unit 25. The display unit 43 corresponds to an image display device such as a car navigation system, a monitor, or a TV, and displays an image from the decoding unit 25.

  In the present embodiment, the camera device 2 is characterized in that after the distortion correction processing of the fisheye image is performed by the distortion correction unit 24, the compression coding processing of the image is performed by the encoding unit 25. As a result, the amount of data after the compression encoding process can be greatly reduced, and the load on the cable 3 can be reduced. The reason will be described below.

  The encoding unit 25 performs image compression using motion compensation. Motion compensation is a technique (interframe prediction) that predicts the current image from the previous image that precedes the time axis. It is adopted in a moving image compression method such as H.264. In compression using inter-frame prediction, only the difference (error) between the current image and the previous image generated by moving the current image needs to be encoded, so the amount of code is greatly reduced and high compression efficiency is achieved. Can be achieved. Specifically, the current image is divided into rectangular blocks such as 16 × 16 pixels, a location with the least error is searched from the previous image for each rectangular block, and the amount of motion is encoded as a motion vector. . In general, since motion compensation is a compression method that assumes parallel movement, when the subject moves in parallel between frames, the difference is small, so that the required data amount is small and the compression efficiency is improved. On the other hand, when the subject rotates or enlarges / reduces between frames, the difference increases, so that the necessary data amount is large and the compression efficiency is deteriorated.

  In the present embodiment, the distortion correction unit 24 is disposed in front of the encoding unit 25, and the distortion correction processing of the fisheye image is performed first, thereby reducing the difference between frames and improving the compression efficiency.

FIG. 4 is a diagram illustrating a case where the compression process is performed without the distortion correction process by the distortion correction unit 24 for comparison.
The upper arrow represents the time axis, 101 and 102 are images taken with a fisheye lens, 101 is the previous image preceding the time axis, and 102 is the current image. Further, the ranges displayed on the display device 4 for confirmation of the rear side are indicated by dotted lines 111 and 112, respectively. Here, the display ranges 111 and 112 have a fan shape, but are converted into a rectangular shape by the distortion correction processing in the display device 4. Reference numerals 121 and 122 denote image ranges to be captured when the images in the display ranges 111 and 112 are compressed.

  Here, the efficiency of motion compensation when compression processing is performed between the images 121 and 122 will be described. Assuming that the rectangular block to be encoded of the current image 122 is 132, the block with the least error in the previous image 121 is 131, and the motion vector between them is 140. When the patterns inside the rectangular blocks 132 and 131 are compared, the pattern is rotationally moved, a large difference value (error) is generated, and the efficiency of motion compensation is reduced. This is because the subject is rotationally moved in the photographed image because of the distortion aberration peculiar to the fisheye lens even though the vehicle is moving in parallel. As a result, the amount of code after compression increases and the compression efficiency decreases.

FIG. 5 is a diagram illustrating a case where compression processing is performed after distortion correction by the distortion correction unit 24 in the present embodiment.
The upper arrow represents the time axis, 201 and 202 are images taken with a fisheye lens, 201 is the previous image preceding the time axis, and 202 is the current image. In addition, the ranges displayed on the display unit 43 for confirmation of the rear side are indicated by dotted lines 211 and 212, respectively. Note that the images 201 and 202 and the dotted lines 211 and 212 are in substantially the same range as the images 101 and 102 and the dotted lines 111 and 112 in FIG. The distortion correction unit 24 performs distortion correction processing of the captured image, and the image in the range of the dotted line 211 is converted into the image 221 and the image in the range of the dotted line 212 is converted into the image 222. The range of the images 221 and 222 has a rectangular shape substantially equal to the range displayed on the display unit 43.

Here, the efficiency of motion compensation when compression processing is performed between the images 221 and 222 after distortion correction processing will be described. If the rectangular block to be encoded of the current image 222 is 232, the block with the least error in the previous image 221 is 231 and the motion vector between them is 240. When the patterns inside the rectangular blocks 232 and 231 are compared, the change is small and the difference value (error) is small. That is, the difference value is significantly reduced compared to the case of FIG. 4 and the efficiency of motion compensation is improved. This is because the movement of the vehicle (subject) is corrected to parallel movement by the distortion correction processing, and the amount of code after compression can be reduced, so that the compression efficiency can be improved.
Further, according to the present embodiment, there is an effect that unnecessary image data not used for display can be deleted.

  In the case of the comparative example of FIG. 4, the image ranges on which the compression processing is performed on the display ranges 111 and 112 are 121 and 122. This is because MPEG2 and H.264. This is because, in a moving image compression method such as H.264, it is assumed that an image in a rectangular area is compressed, and therefore, the minimum rectangular areas 121 and 122 including an image to be displayed are used as a compression processing range. Here, the rectangular images 121 and 122 include images (indicated by diagonal lines) of areas 151 and 152 that are not displayed on the display device 4. That is, since a code amount is assigned to a region (image) that is not used for display, unnecessary image data is included, and image display efficiency is reduced.

  On the other hand, in the present embodiment shown in FIG. 5, the image ranges in which the compression processing is performed on the display ranges 211 and 212 are 221 and 222, and the two ranges substantially coincide. This is because the captured image is converted into an image of a rectangular area displayed on the display device 4 by the distortion correction processing. Therefore, since only the necessary image area can be encoded by the display device 4 during the compression process, there is an advantage that the image display efficiency is improved.

  As described above, according to the present embodiment, since the compression processing is performed after the distortion correction processing of the captured image using the fisheye lens, the motion compensation efficiency, that is, the compression efficiency can be improved, and the display can be used for display. Since it is not necessary to encode unnecessary data, display efficiency can be improved. As a result, if the image quality is the same, the data size becomes smaller and the transmission bandwidth is reduced, or if the data size is the same, a higher quality image can be transmitted. In addition, the image recognition accuracy in the camera system can be improved by increasing the image quality.

  In the second embodiment, a case where an existing image processing unit on the premise of an image having distortion characteristic of a fisheye lens on the display device side will be described. In other words, the display device has an image processing algorithm including distortion correction (for example, viewpoint conversion, pop-out detection, obstacle detection, white line recognition, sign recognition, and creation of an image that looks down on the entire vehicle from above). In that case, if the distortion correction processing is performed by the camera device 2 shown in the first embodiment, and further the image processing is performed by the existing image processing unit of the display device 3, the effect of the original image processing may not be obtained or the problem may occur. There is a problem that occurs. Therefore, in this embodiment, the display device 3 solves this problem by using an inverse conversion formula of the distortion correction process to return to an image having distortion aberration peculiar to the fisheye lens before performing the image process. .

  FIG. 6 is a block diagram of the in-vehicle camera system of the present embodiment. Blocks having the same functions as those of the camera system of Embodiment 1 (FIG. 3) are denoted by the same reference numerals, and description thereof is omitted. A difference from the camera system of FIG. 3 is that a reverse distortion correction unit 44 and an image processing unit 45 are added between the decoding unit 42 and the display unit 43 of the display device 4. Note that the image processing unit 45 is an existing one provided in the display device 4.

  The inverse distortion correction unit 44 returns the image subjected to the distortion correction processing by the distortion correction unit 24 of the camera device 2 to an image having distortion aberration peculiar to the fisheye lens, using a conversion formula opposite to the conversion formula of the distortion correction process. Processing (inverse distortion correction processing) is performed. Further, the image processing unit 45 performs image processing (including distortion correction processing) on the premise of an image having distortion aberration peculiar to a fisheye lens, on the image subjected to the reverse distortion correction processing by the reverse distortion correction unit 44. As a result, the image processing unit 45 can obtain an image with the original image processing effect.

  According to the configuration of the present embodiment, it is not necessary to change the image processing algorithm of the existing image processing unit 45, and the compression efficiency can be improved and the display efficiency can be improved as in the first embodiment. As a result, if the image quality is the same, the data size becomes smaller and the transmission bandwidth is reduced, or if the data size is the same, a higher quality image can be transmitted.

The embodiment described above can be modified as follows.
In the above embodiment, an example of an in-vehicle camera system has been specifically described as an example of a camera system. Applicable. Also in this case, since the compression efficiency can be improved, if the image quality is the same, the data size is reduced and the transmission bandwidth is reduced. If the data size is the same, a higher quality image can be transmitted.

  In addition, in the lens unit 21 of the camera device 2, an example of an image having a specific distortion aberration photographed with a fisheye lens has been described, but it may be replaced with a wide-angle lens. Since an image photographed with a wide-angle lens also has distortion aberration, compression efficiency can be improved by compressing the image after performing distortion correction processing.

  In the camera device 2, the signal processing unit 23 and the distortion correction unit 24 are separated from each other. However, these functions may be integrated into the signal processing unit 23 and the distortion correction process may be performed in the signal processing unit 23. . Alternatively, the order of the signal processing unit 23 and the distortion correction unit 24 may be switched, and the signal processing unit 23 may be disposed after the distortion correction unit 24.

  Further, there may be a storage medium such as an HDD or an SD card between the camera device 2 and the display device 4, and the recording once recorded may be reproduced. Also in this case, since the compression efficiency can be improved, it is possible to record on the recording medium for a longer time.

2 ... Camera device,
3 ... Cable
4 ... display device,
21 ... Lens part,
22 ... sensor part,
23 ... Signal processing unit,
24 ... distortion correction unit,
25. Encoding part,
26: Transmitter,
41. Reception unit,
42. Decoding part,
43 ... display part,
44 ... reverse distortion correction unit,
45. Image processing unit.

Claims (2)

In a camera system that transmits and displays an image captured by a camera device to a display device,
The camera device is
A lens part using a fisheye lens,
A distortion correction unit that performs distortion correction processing on the image with distortion aberration captured by the fisheye lens so that the distortion aberration is reduced;
An encoding unit that compresses and encodes an image output from the distortion correction unit;
The display device
A decoding unit that decompresses and decodes an image compressed and encoded by the encoding unit of the camera device;
A first image processing unit that performs image processing on the image output from the decoding unit;
A second image processing unit that performs image processing including distortion correction processing on the image output from the first image processing unit;
A display unit for displaying an image output from the second image processing unit;
Equipped with a,
The first image processing unit performs reverse distortion correction for generating an image with distortion using a conversion formula opposite to a conversion formula of a distortion correction process performed by the distortion correction unit of the camera device. Camera system. The camera system according to claim 1,
The range of an image to be compression-encoded by the encoding unit and the range of an image to be displayed by the display unit are substantially equal .

Images