JP4561649B2 - Image compression apparatus, image compression program and image compression method, HDR image generation apparatus, HDR image generation program and HDR image generation method, image processing system, image processing program and image processing method - Google Patents

Description

  The present invention relates to an image compression apparatus, an image compression program, and an image compression method suitable for compressing image data of images respectively captured at a plurality of types of exposure times for generating HDR (High Dynamic Range) image data, An HDR image generation apparatus, an HDR image generation program and an HDR image generation method suitable for generating HDR image data based on image data compressed by any of these, and restoring image data compressed by the image compression method The present invention relates to an image processing system, an image processing program, and an image processing method that can generate HDR image data based on the restored image data.

In recent years, it has been desired to increase the dynamic range (HDR) of surveillance cameras used for security applications. For security applications, logarithmic compression type CMOS sensors (research stage) and surveillance cameras using two-stage exposure of CCD image sensors are being put into practical use.
On the other hand, with the shift to a large-capacity digital network society, a monitoring system using a network is being put into practical use. In addition to networking, the practical use of a monitoring system that combines machine vision that recognizes suspicious persons and entering objects by computers is awaited.

As described above, HDR is expected for the function of the surveillance camera, and networking and machine vision are expected for the surveillance camera system.
As a system combining a wide dynamic range camera and a network, for example, an on-site monitoring system described in Patent Document 1, a video signal transmission / reception system described in Patent Document 2, a video information adjustment device described in Patent Document 3, and video information There are a transmitting device and a video information receiving device.

The prior art of Patent Document 1 has logarithmic input / output characteristics and a wide dynamic range camera with a wide viewing angle that outputs image data obtained by imaging a wide area in the field, and image data transmitted from the wide dynamic range camera. And a host computer having display means for displaying the image, and monitors the site based on the image displayed on the display means.
In the prior art of Patent Document 2, the long-exposure video and the short-exposure video are separately compressed and transmitted. On the receiving terminal side, the transmitted data is decompressed separately and synthesized at an arbitrary ratio. Wide dynamic range video.

Further, the prior art of Patent Document 3 includes an illuminance information acquisition unit that separately acquires image information corresponding to at least two different illuminance ranges from the same shot based on predetermined imaging conditions, and the at least two types. The image information corresponding to the different illuminance ranges is separately transmitted to the outside, and the transmission unit that receives the image information corresponding to at least two different illuminance ranges from the outside, and the at least two different types of the illuminance ranges. An illuminance information adjustment unit that generates image information that has been subjected to luminance adjustment based on a predetermined illuminance range using all or part of the image information corresponding to the illuminance range. Thereby, the image information transmitted to the outside can be adjusted on the receiving side.
Japanese Patent Laid-Open No. 2002-204443 Japanese Patent Laid-Open No. 2003-299067 JP-A-8-340523

  However, in the conventional techniques disclosed in Patent Documents 1 to 3, an image having a very large contrast (HDR image) or a plurality of images for generating the HDR image is captured. It becomes. Since such image data needs to be transmitted to a receiving device via a network, in particular, monitoring is performed such that the captured image data is transmitted via the Internet and the captured image is monitored in real time in a remote place. When applied to a system, it is necessary to perform high-speed and stable data transmission processing in consideration of network delay and the like. Therefore, it is desirable to reduce the transmission amount of image data of each frame as much as possible in order to increase the data transmission speed. However, in the conventional techniques of Patent Documents 1 and 3, the image data amount is compressed on the transmission side. Nothing is said about the process of reducing the amount of transmitted data.

  On the other hand, in the prior art of Patent Document 2 described above, it is described that long-exposure image data and short-exposure image data captured by a plurality of cameras are compressed by JPEG and transmitted to reduce the data transmission amount. However, due to the nature of the system, the amount of image data to be transmitted increases in proportion to the increase in the number of cameras and the type of exposure time, etc. In order to realize a real-time monitoring system, It is necessary to reduce the amount of image data to be transmitted as much as possible by performing compression processing with a high compression rate and transmit the image data to the receiving side at high speed. However, if the compression rate is too high, the image quality will be affected. Therefore, there is a concern that the visibility of captured images that are important in the monitoring system may deteriorate.

  Therefore, the present invention has been made paying attention to such an unsolved problem of the conventional technology, and each of the images is picked up at a plurality of types of exposure times for generating HDR (High Dynamic Range) image data. Image compression apparatus, image compression program, and image compression method suitable for compressing image data of a recorded image, and HDR image generation suitable for generating HDR image data based on the image data compressed by any of these Apparatus, HDR image generation program and HDR image generation method, and image processing system, image processing program and image capable of restoring image data compressed by the image compression method and generating HDR image data based on the restored image data It aims to provide a processing method.

[Mode 1] In order to achieve the above object, an image compression apparatus according to mode 1 includes:
Image data storage means for storing image data of an image captured at a standard exposure time and image data of each image captured at an exposure time of N types (N is a natural number of 1 or more) shorter than the standard exposure time; ,
Determining means for determining whether or not the luminance indicated by each pixel data constituting each image data corresponding to each exposure time is saturated;
Based on the determination result of the determination means, pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time is replaced with a fixed value for each exposure time. Effective image data generating means for generating image data;
Image compression means for compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
Exposure time information generating means for generating exposure time information in which information related to each exposure time is associated with each compressed effective image data.

  With such a configuration, the image data storage means captures image data of an image captured with the standard exposure time and N types of exposure time (N is a natural number of 1 or more) shorter than the standard exposure time. It is possible to store the image data of each image, and it is possible to determine whether or not the luminance indicated by each pixel data constituting each image data corresponding to each exposure time is saturated by the determination unit. The pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time is fixed by the effective image data generation unit based on the determination result of the determination unit. It is possible to generate effective image data for each exposure time replaced with a value, and the effective image data corresponding to each exposure time by an image compression means The exposure time information can be compressed using a predetermined image compression method, and the exposure time information generation means associates information related to each exposure time with each compressed effective image data. Can be generated.

  Therefore, among the pixel data constituting the image data corresponding to each exposure time, the saturated pixel data is converted into a constant value (for example, 0) while the non-saturated pixel data is left as it is, and effective image data is generated. Since the effective image data is compressed by using a predetermined image compression method, it is possible to efficiently compress a portion having a constant value rather than compressing the image data of each exposure time as it is. can get.

  In addition, effective image data corresponding to an exposure time shorter than the standard exposure time requires correction of the luminance level when generating HDR image data, but is shorter than the standard exposure time when compressing effective image data. The effective image data corresponding to each exposure time is compressed without correcting the luminance level, and the exposure time corresponding to each exposure time is associated with the effective image data corresponding to each exposure time. Since the information is generated, when the HDR image data is generated by synthesizing the effective image data corresponding to each exposure time, the brightness level of the image data shorter than the standard exposure time is simplified by the exposure time information. It is possible to correct the luminance level necessary for the image. In other words, the effective image data corresponding to each exposure time shorter than the standard exposure time can be compressed in a state of low (small) entropy (information amount), thereby further reducing the total data amount after compression. The effect that it can do is acquired.

  Here, the HDR image data is generally non-saturated among pixel data constituting the same imaging target with image data captured with a standard exposure time and image data captured with a shorter exposure time shorter than this. It is generated by combining pixel data. Specifically, for example, HDR image data is generated by replacing saturated pixel data in the image data captured with the standard exposure time with non-saturated pixel data in the image data captured with the short exposure time. Hereinafter, an aspect relating to an image compression program, an aspect relating to an image compression method, an aspect relating to an HDR image generation apparatus, an aspect relating to an HDR image generation program, an aspect relating to an HDR image generation method, an aspect relating to an image processing system, an aspect relating to an image processing program, an image processing method And the like in the column of the best mode for carrying out the invention.

  In addition, as the constant value, a value that cannot be taken as a normal pixel value, a value that does not change even after multiplication (for example, “0”), and the like in a subsequent multiplication process when generating HDR image data are set. Hereinafter, an aspect relating to an image compression program, an aspect relating to an image compression method, an aspect relating to an HDR image generation apparatus, an aspect relating to an HDR image generation program, an aspect relating to an HDR image generation method, an aspect relating to an image processing system, an aspect relating to an image processing program, an image processing method And the like in the column of the best mode for carrying out the invention.

  The predetermined compression processing method includes various encoding processing methods such as JPEG (Joing Picture Expert Group), PNG (Portable Network Graphic), TIFF (Tagged Image File Format), etc., and includes lossless compression and lossy compression. Either of these is acceptable. Hereinafter, an aspect relating to an image compression program, an aspect relating to an image compression method, an aspect relating to an HDR image generation apparatus, an aspect relating to an HDR image generation program, an aspect relating to an HDR image generation method, an aspect relating to an image processing system, an aspect relating to an image processing program, an image processing method And the like in the column of the best mode for carrying out the invention.

[Mode 2] Furthermore, the image compression apparatus according to mode 2 is the image compression apparatus according to mode 1,
The effective image data generation means generates the effective image data in order from the longest exposure time, and in the generation process of the effective image data corresponding to each exposure time other than the standard exposure time, Based on the pixel data information replaced with a constant value at the time of image data generation, among the pixel data replaced with the constant value in the previous effective image data in the image data corresponding to each exposure time Non-saturated pixel data is selected, and pixel data other than the selected pixel data is replaced with the constant value to generate the effective image data.

  With such a configuration, among the pixel data constituting the image data corresponding to each exposure time, in order from the longest exposure time to the shortest one, the saturated pixel data is left unchanged as the non-saturated pixel data. In the generation process of the effective image data corresponding to each exposure time other than the standard exposure time, the effective image data is generated by converting to a constant value (for example, 0). Based on the pixel data information converted into the pixel data, non-saturated pixel data is selected from the pixel data at the same position as the pixel position converted to a constant value before each image data, and the selected pixel data is selected. It is possible to generate effective image data by converting other pixel data into a constant value as it is.

  Therefore, rather than simply selecting saturated pixel data and non-saturated pixel data and converting the saturated pixel data to a constant value to generate effective image data, it corresponds to exposure times other than the standard exposure time. Since the ratio of the constant value in the effective image data can be increased, it is possible to compress efficiently, and the effect that the total data amount after compression can be further reduced is obtained.

[Mode 3] Furthermore, the imaging device of mode 3 is the imaging device of mode 2,
The exposure time information generating means corresponds to each pixel data for each non-saturated pixel data based on position information of the non-saturated pixel data in each effective image data corresponding to each exposure time. It is characterized in that exposure time information is generated by associating information related to the exposure time to be performed.

With such a configuration, it is possible to obtain exposure time information including information related to the exposure time corresponding to the pixel position of the unsaturated pixel data in the image data corresponding to each exposure time.
Therefore, the exposure time information can be handled as one piece of image data, and the luminance level of each pixel data can be easily corrected even after synthesizing effective image data corresponding to each exposure time in order to generate HDR image data. The effect that it can be performed is obtained.

[Mode 4] Furthermore, the imaging device of mode 4 is the imaging device of any one of modes 1 to 3,
An exposure time information compression unit is provided for reversibly compressing the exposure time information using a predetermined lossless compression method.
With such a configuration, the data amount of the exposure time information can be reduced, and the total data amount can be further reduced.
Here, the lossless compression method is a compression method capable of completely reproducing information before compression such as run-length compression.

[Embodiment 5] Furthermore, the imaging device of Embodiment 5 is the imaging device of any one of Embodiments 1 to 4,
Binary image data obtained by converting each pixel data constituting each image data corresponding to each exposure time into one of binary values indicating saturation and non-saturation based on the determination result of the determination means. Binary image data generating means for generating
Noise removal means for performing isolated point removal processing on data indicating saturation in binary image data corresponding to each exposure time generated by the binary image data generation means,
The effective image data generating means generates the effective image data based on the binary image data processed by the noise removing means.

  With such a configuration, the binary image data generation means indicates saturation and desaturation for each pixel data constituting each image data corresponding to each exposure time based on the determination result of the determination means. It is possible to generate binary image data that is converted into any one of binary values, and a binary image corresponding to each exposure time generated by the binary image data generating means by the noise removing means. Isolated data generated due to noise or the like can be removed from data indicating saturation in the data. Further, the effective image data generating means can generate the effective image data based on the binary image data processed by the noise removing means.

  That is, based on the binary image data corresponding to each exposure time after noise removal, the isolated pixel data portion is converted from each effective image data by converting the saturated pixel data in the effective image data corresponding to each exposure time to a constant value. Therefore, the image quality of the HDR image data generated from these effective image data can be improved.

Here, as the noise removing means, there are, for example, n times (n is a natural number equal to or greater than 1) expansion processing and binary image processing that performs n times of shrinkage processing after the n times of expansion processing. The dilation processing is processing for setting a pixel to 1 if there is at least 1 (white) in the vicinity of a certain pixel (near 4 or 8). The contraction process is a process for setting a pixel to 0 if there is any 0 in the vicinity of a certain pixel (near 4 or 8). In other words, by performing the contraction process after the expansion process, if the selected pixel is an isolated pixel, this pixel is removed (converted to “0”).
Further, the binary image processing performs n times of shrinkage processing after n times of dilation processing, and by performing multiple times of dilation processing, not only the removal of isolated pixels but also two or more pixels can be obtained. It is also possible to restore a clean image by connecting the faint images that have been cut off and separated.

[Mode 6] On the other hand, in order to achieve the above object, an image compression program of mode 6
Whether or not the brightness indicated by each pixel data constituting the image data of the image captured at the standard exposure time and the image data of each image captured at a plurality of types of exposure times shorter than the standard exposure time is saturated A determination step for determining whether or not
Based on the determination result of the determination step, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. An image data generation step;
An image compression step of compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
A program for causing a computer to execute processing including an exposure time information generation step of generating exposure time information in which information related to each exposure time is associated with each compressed effective image data; It is characterized by.

  With such a configuration, when the program is read by the computer and the computer executes processing in accordance with the read program, the same operation and effect as those of the image compression apparatus of mode 1 can be obtained.

[Mode 7] In order to achieve the above object, an image compression method according to mode 7 includes:
Whether or not the brightness indicated by each pixel data constituting the image data of the image captured at the standard exposure time and the image data of each image captured at a plurality of types of exposure times shorter than the standard exposure time is saturated A determination step for determining whether or not
Based on the determination result of the determination step, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. An image data generation step;
An image compression step of compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
An exposure time information generating step of generating exposure time information in which information related to each exposure time is associated with each compressed effective image data.
Thereby, the same operation and effect as those of the image compression apparatus of aspect 1 can be obtained.

[Mode 8] On the other hand, in order to achieve the above object, an HDR image generation apparatus according to mode 8
The effective image data compressed by the image compression apparatus according to claim 1 is restored to the effective image data before compression, and based on the restored effective image data and the exposure time information. An HDR image generation device that generates HDR (High Dynamic Range) image data,
Effective image data restoring means for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, multiplication means for multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
HDR image data generation means for generating HDR image data based on the effective image data corresponding to each of the exposure times after multiplication by the multiplication means.

  If it is such a structure, in the image compression apparatus of any one of Claim 1 thru | or 5 by the effective image data decompression | restoration means, the effective image data compressed using the said predetermined compression method will be carried out. Each effective image data can be reconstructed and based on each effective image data corresponding to each of the reconstructed exposure times and the exposure time information corresponding to each of the effective image data by a multiplication unit. Can be multiplied by a constant determined from the ratio between the standard exposure time and each exposure time, and the HDR image data generating means can multiply the pixel data that has not been replaced with the constant value. It is possible to generate HDR image data based on the effective image data corresponding to each of the exposure times after multiplication by.

  Therefore, for example, when the image compression apparatus according to any one of claims 1 to 5 and the HDR image generation apparatus of the present embodiment are connected via a network, the image compression apparatus has high efficiency. The effective image data compressed in step 1 and the exposure time information are transmitted to the HDR image generation device via the network, and the HDR image generation device receives the HDR image based on the effective image data and the exposure time information received via the network. Since data can be generated, a system that easily transmits various data necessary for generating HDR image data, such as a monitoring system that monitors captured images of remote locations in real time, can be easily constructed. The effect of being able to be obtained.

[Mode 9] In order to achieve the above object, an HDR image generation program according to mode 9
The effective image data compressed by the image compression apparatus according to claim 1 is restored to the effective image data before compression, and based on the restored effective image data and the exposure time information. An HDR image data generation program for generating HDR (High Dynamic Range) image data,
An effective image data restoring step for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, a multiplication step of multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
And a program for causing a computer to execute processing including HDR image data generation step for generating HDR image data based on the effective image data corresponding to each of the exposure times after multiplication in the multiplication step.

  With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the HDR image generation device according to mode 8 can be obtained.

[Mode 10] In order to achieve the above object, an HDR image generation method according to mode 10 includes:
The effective image data compressed by the image compression apparatus according to claim 1 is restored to the effective image data before compression, and based on the restored effective image data and the exposure time information. An HDR image data generation method for generating HDR (High Dynamic Range) image data,
An effective image data restoring step for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, a multiplication step of multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
And a HDR image data generation step of generating HDR image data based on the effective image data corresponding to each of the exposure times after multiplication in the multiplication step.
Thereby, the same operation and effect as those of the HDR image generation device according to mode 8 can be obtained.

[Mode 11] On the other hand, in order to achieve the above object, an image processing system according to mode 11
Imaging means for imaging an imaging object with a standard exposure time and the N types of exposure times shorter than the standard exposure time (N is a natural number of 1 or more);
Image data output means for separately outputting image data of an image taken by the imaging means and taken by the standard exposure time and image data taken by each of the N types of exposure times;
Image data storage means for storing the image data of the image captured at the standard exposure time and the image data of each image captured at the N types of exposure times output by the output means;
Determining means for determining whether or not the luminance indicated by each pixel data constituting each image data corresponding to each exposure time is saturated;
Based on the determination result of the determination means, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. Image data generating means;
Image compression means for compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
Exposure time information generating means for generating exposure time information in which information related to each exposure time is associated with each compressed effective image data;
Effective image data restoring means for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, multiplication means for multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
HDR image data generation means for generating HDR image data based on the effective image data corresponding to each of the exposure times after multiplication by the multiplication means.

  With such a configuration, an imaging target can be imaged by the imaging means with a standard exposure time and the N types of exposure times (N is a natural number of 1 or more) shorter than the standard exposure time. The data output means can output separately the image data of the image taken by the imaging means and taken by the standard exposure time and the image data of each image taken by the N types of exposure times. The image data storage means stores the image data of the image picked up at the standard exposure time and the image data of the images picked up at the N types of exposure time outputted by the image data output means. It is possible to determine whether the luminance indicated by each pixel data constituting each image data corresponding to each exposure time is saturated by the determination means. Pixels other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time based on the determination result of the determination unit by the effective image data generation unit It is possible to generate effective image data in which the data is replaced with a constant value, and the exposure time information generating means associates the information related to each exposure time with the compressed effective image data. It is possible to generate exposure time information.

  Further, it is possible to restore the effective image data compressed using the predetermined compression method by the effective image data restoring means, and each effective image corresponding to each of the restored exposure times by the multiplying means. Based on the data and the exposure time information corresponding to each effective image data, the ratio between the standard exposure time and each exposure time for the pixel data not replaced with the constant value in each effective image data The HDR image data generating unit can generate HDR image data based on the effective image data corresponding to each exposure time after multiplication by the multiplying unit. is there.

  Therefore, among the pixel data constituting the image data corresponding to each exposure time, the saturated pixel data is converted into a constant value (for example, 0) while the non-saturated pixel data is left as it is, and effective image data is generated. Since the effective image data is compressed by using a predetermined image compression method, it is possible to efficiently compress a constant value portion rather than compressing the image data of each exposure time as it is. The total amount of data can be reduced. Further, it is possible to restore the effective image data compressed with high efficiency in this way, and to easily generate the HDR image data based on the restored effective image data and the exposure time information.

  For example, an image pickup apparatus and an image data output means constitute an image pickup apparatus, and an image data storage means, a determination means, an effective image data generation means, and an exposure time information generation means constitute an image compression apparatus, and an effective image When an HDR image generation apparatus is configured by data restoration means, multiplication means, and HDR image data generation means, and these are connected via a network, an image processing system is configured. An image can be picked up and the picked-up image data can be output separately for each exposure time, and the effective image data compressed with high efficiency in the image compression device and the exposure time information are The effective image data and the exposure time information transmitted to the HDR image generation device via the network and received via the network in the HDR image generation device Since HDR image data can be generated based on this, a system that needs to transmit various data necessary for generating HDR image data at high speed, such as a monitoring system that monitors a captured image of a remote place in real time, can be simplified. The effect that it can be constructed is obtained.

  Here, this system may be realized as a single device, terminal, or other device, or may be realized as a network system in which a plurality of devices, terminals, or other devices are communicably connected. . In the latter case, each component may belong to any one of a plurality of devices and the like as long as they are connected so as to communicate with each other.

[Mode 12] In order to achieve the above object, an image processing program according to mode 12
Image data of an image captured at the standard exposure time and each pixel data constituting the image data of each image captured at the N types of exposure times (N is a natural number of 1 or more) shorter than the standard exposure time. A determination step for determining whether or not the indicated luminance is saturated;
Based on the determination result of the determination step, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. An image data generation step;
An image compression step of compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
An exposure time information generating step for generating exposure time information in which information relating to each exposure time is associated with each compressed effective image data;
An effective image data restoring step for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, a multiplication step of multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
And a program for causing a computer to execute processing including HDR image data generation step for generating HDR image data based on the effective image data corresponding to each of the exposure times after multiplication in the multiplication step.

  With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operations and effects as those of the image processing system according to mode 11 are obtained.

[Mode 13] In order to achieve the above object, an image processing method according to mode 13 includes:
Image data of an image captured at the standard exposure time and each pixel data constituting the image data of each image captured at the N types of exposure times (N is a natural number of 1 or more) shorter than the standard exposure time. A determination step for determining whether or not the indicated luminance is saturated;
Based on the determination result of the determination step, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. An image data generation step;
An image compression step of compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
An exposure time information generating step for generating exposure time information in which information relating to each exposure time is associated with each compressed effective image data;
An effective image data restoring step for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, a multiplication step of multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
And a HDR image data generation step of generating HDR image data based on the effective image data corresponding to each of the exposure times after multiplication in the multiplication step.
Thereby, the same operation and effect as those of the image processing system according to the eleventh aspect can be obtained.

  Embodiments of an image compression apparatus, an image compression program and an image compression method, an HDR image generation apparatus, an HDR image generation program and an HDR image generation method, an image processing system, an image processing program and an image processing method according to the present invention are described below. This will be described with reference to the drawings. 1 to 13 show an image compression apparatus, an image compression program and an image compression method, an HDR image generation apparatus, an HDR image generation program and an HDR image generation method, an image processing system, an image processing program and an image processing method according to the present invention. It is a figure which shows this embodiment.

Hereinafter, an image compression apparatus, an image compression program and an image compression method, an HDR image generation apparatus, an HDR image generation program and an HDR image generation method, an image processing system, an image processing program and an image processing method according to the present invention are applied to a camera system. An embodiment in the case of such a case will be described.
The overall configuration of the camera system 1 will be described below with reference to FIG. Here, FIG. 1 is a block diagram showing the overall configuration of the camera system 1.

  As shown in FIG. 1, the camera system 1 includes a multi-stage exposure camera 100 that can capture an imaging target with a plurality of types of exposure times, and each image captured with a plurality of types of exposure times in the multi-stage exposure camera 100. An image compression apparatus 200 that generates HDR image compression data based on image data, an HDR image data generation apparatus 300 that generates HDR image data based on image compression data transmitted from the image compression apparatus 200 via a network, and HDR Based on the HDR image data generated by the image data generation device 300, the computer 400 that executes a predetermined control process, the control means 500 that controls the computer 400, and the HDR image data output from the HDR image data generation device 300 Display apparatus 600 for displaying an HDR image based on the image compression apparatus 2 0 and a and a network 700 connecting the HDR image generation apparatus 300 to each other to enable data communication.

  The multi-stage exposure camera 100 captures an image exposed with a standard exposure time in one frame period (one exposure period) and the same one frame as the exposure for the standard exposure time (which can be arbitrarily set by the user). Function to capture images exposed at each exposure time of short exposure time, ultra short exposure time and ultra ultra short exposure time (standard exposure time> short exposure time> long and short exposure time> ultra ultra short exposure time) Have. Furthermore, the image data of each image picked up with these four types of exposure times is output to the image compression apparatus 200 independently and in parallel using the four output channels (CH1 to CH4). . In the present embodiment, it is assumed that image data of images captured from CH1 to standard exposure time, CH2 to short exposure time, CH3 to ultrashort exposure time, and CH4 to ultrashort exposure time are output.

  The image compression apparatus 200 stores an image storage unit 20 that stores image data respectively corresponding to the four types of exposure times output from CH1 to CH4 of the multistage exposure camera 100, and the image storage unit 20 stores the image data. An image compression unit 22 that generates HDR image compression data based on image data corresponding to each of the four types of exposure times, and an HDR image generation device 300 that converts the HDR image compression data generated by the image compression unit 22 via the network 700. And a data transmission unit 24 that transmits the data. Hereinafter, the image data corresponding to the standard exposure time is referred to as standard exposure image data, the image data corresponding to the short exposure time is referred to as short exposure image data, and the image data corresponding to the ultra-short exposure time is referred to as long / short exposure image data. The image data corresponding to the ultra-short exposure time will be referred to as long and short exposure image data.

  The image storage unit 20 stores, for each line, standard exposure image data, short exposure image data, ultrashort exposure image data, and ultra ultrashort exposure image data, which are input for each pixel line from the multistage exposure camera 100. In order to synchronize the four types of stored image data in subsequent processing, each image data is buffered (accumulated). Alternatively, it has a function of storing four types of image data for one frame and then outputting them independently or in parallel to the image compression unit 22.

  The image compression unit 22 generates mask data, valid area image data, and invalid binary image data corresponding to each exposure time based on the image data corresponding to each exposure time input from the image storage unit 20, and Exposure code image data is generated based on the effective area image data corresponding to each exposure time. Further, the effective area image data corresponding to each exposure time and the generated exposure code image data are compressed using a predetermined compression method, and these five data are combined into one file to generate HDR image compression data. This is output to the data transmission unit 24.

  The data transmission unit 24 transmits the HDR image compression data input from the image compression unit 22 to the HDR image generation device 300 via the network 700, and also from the external device including the HDR image generation device 300 or the like to the image compression device 200. It also has a function of receiving various data sent to the address.

  The HDR image generating device 300 receives the HDR image compressed data sent from the image compressing device 200 via the network 700, and the HDR image data based on the HDR image compressed data received by the data transmitting unit 30. The HDR image generation unit 32 for generating the data, the recording device 34 for recording the HDR image compressed data received by the data transmission unit 30, and the HDR image data generated by the HDR image generation unit 32 are transmitted to the computer 400 or displayed. And an HDR image processing unit 36 that performs processing such as transmission to the apparatus 600.

  The data transmission unit 30 receives the HDR image compression data transmitted from the image compression device 200 via the network 700, and outputs the received HDR image compression data to the HDR image generation unit 32 and the recording device 34, respectively. Or transmitting various data to the external device including the image compression apparatus 200 via the network 700.

  The HDR image generation unit 32 decomposes the HDR image compressed data input from the data transmission unit 30 into four compressed effective area image data and one compressed exposure code image data, and these five compressed image data. Has a function of decompressing (decoding). Further, the expanded four effective area image data is synthesized, and based on the exposure code image data, the brightness level of each pixel data of the synthesized image data is corrected to generate HDR image data. Yes.

The recording device 34 records and holds the HDR image compressed data sent from the image compression device 200. The recorded HDR image compressed data is used when the computer 400 or the like analyzes past captured image data again. Used for etc.
The HDR image processing unit 36 transmits the HDR image data generated by the HDR image generation unit 32 to the computer 400 and the display device 600, or displays an image to be displayed on the display device 600 in accordance with a control command from the computer 400. It has a function of performing various processes on HDR image data, such as changing the gradation.

For example, when the camera system 1 is used for monitoring purposes, the computer 400 analyzes the HDR image data to determine whether or not a suspicious person appears in the captured image, and further, based on the determination result. Perform an action (eg, output a signal to activate an alarm).
The control means 500 is a means for an operator to give an instruction to the computer 400, and is composed of, for example, an input device such as a keyboard and a mouse.

The display device 600 has a function of displaying a captured image based on display image data (HDR image data or image data obtained by correcting HDR image data to a low gradation) from the HDR image processing unit 34.
The network 700 is configured by any one of known LANs, WANs, the Internet, and the like, or a combination thereof. The connection between the image compression apparatus 200 and the HDR image generation apparatus 300 is wired and wireless, or a combination thereof. Either of them is acceptable. The network 700 is selected and used according to the purpose of this system. For example, when the monitoring target and the monitoring place (monitoring person) are far away, WAN or the Internet is used.

  Furthermore, the internal structure of the image compression part 22 is demonstrated based on FIGS. Here, FIG. 2 is a diagram illustrating an internal configuration of the image compression unit 22. FIG. 3 is a diagram showing an internal configuration of the effective image forming unit 22a. FIG. 4 is a diagram illustrating an internal configuration of the mask data generation unit 220a.

  As shown in FIG. 2, the image compression unit 22 includes an effective image forming unit 22a that generates effective area image data corresponding to each exposure time and exposure code image data indicating the exposure time of each effective area image data, and the effective image. The effective image compression unit 22b that compresses the effective area image data and the exposure code image data generated by the forming unit 22a by a predetermined compression method, and the effective area image data and the exposure code image data that are compressed by the effective image compression unit 22b. And a file conversion unit 22c that generates HDR image compression data by converting the file into one file.

As shown in FIG. 3, the effective image forming unit 22a includes first to fourth mask data generation units 220a to 220d, first to fourth effective / invalid image generation units 224a to 224d, and an exposure code image generation unit 228. It is comprised including.
As shown in FIG. 4A, the first mask data generation unit 220a includes a saturation region detection unit 220a ₁ and a noise removal filter 220a ₂ .

As shown in FIG. 4B, the saturation region detection unit 220a ₁ has a comparator, and the comparator inputs each pixel data (signal) of the standard exposure image data and is set in advance. Compared with the threshold YHigh, “0” is output when the input pixel data is less than or equal to the threshold YHigh, and “1” is output when the input pixel data is greater than the threshold YHigh. This output is input to the noise removal filter 220a _2. That is, the saturated region detection unit 220a ₁ becomes binary image data obtained by converting each pixel data of non-saturation into “0” and each pixel data saturated into “1” in the standard exposure image data.

The noise removal filter 220a ₂ performs binary image processing on the binary image data from the saturation region detection unit 220a ₁ to remove isolated pixels and the like generated due to noise. For example, two expansion processes are performed, and two contraction processes are performed after the two expansion processes.
Then, the binary image data subjected to noise removal by the binary image processing is output as first mask data to the first valid / invalid image generation unit 224a.

  Note that the second to fourth mask data generation units 220b to 220d are configured so that the input mask data is only the short-exposure image data, the ultra-short exposure image data, and the ultra-short exposure image data, respectively. Since it becomes the structure similar to the production | generation part 220a, description is abbreviate | omitted. The second to fourth mask data generation units 220b to 220d output the generated second to fourth mask data to the second to fourth valid / invalid image generation units 224b to 224d, respectively.

  Returning to FIG. 3, the first valid / invalid image generation unit 224 a performs first mask data based on the standard exposure image data from the image storage unit 20 and the first mask data from the first mask data generation unit 220 a. The standard exposure image data at the same position as the pixel position of each pixel data whose value is “1” (saturated) is invalidated, and the values of the other pixel data are validated. Standard exposure effective area image data is generated based on the validity / invalidity. That is, in the standard exposure effective area image data, the value of the invalid pixel is converted to “0”, and the valid pixel is the value of the standard exposure image data as it is. That is, in the standard exposure effective area image data, the saturated pixel data in the standard exposure image data is converted to “0”. Further, first invalid binary image data indicating invalid / valid of the standard exposure image data is generated. The first invalid binary image data indicates that the standard exposure image data is valid when the value is “1”, and indicates that the standard exposure image data is invalid when the value is “0”. Then, the standard exposure effective area image data is output to the effective image compression unit 22b, and the first invalid binary image data is output to the second valid / invalid image generation unit 224b and the exposure code image generation unit 228, respectively.

  The second valid / invalid image generation unit 224b includes the short exposure image data from the image storage unit 20, the second mask data from the second mask data generation unit 220b, and the first valid / invalid image generation unit 224a. 1 invalid binary image data is input, and among the pixel data of the short exposure image data, the first invalid binary image data has a value of “0” (saturated) and the second mask data has a value of “0”. The value of pixel data that is (unsaturated) is validated, and the values of other pixel data are invalidated. Based on the validity / invalidity, short exposure effective area image data is generated. That is, in the short exposure effective area image data, the effective pixel is converted to “0” while the effective pixel is left as it is, and the invalid pixel is converted to “0”. In the short exposure effective area image data, the pixel data values at the same pixel positions as the effective pixels in the standard exposure effective area image data are all converted to “0” (invalid). Since the effective pixel data does not overlap at the same pixel position between the short exposure effective area image data and the short exposure effective area image data, the compression efficiency of the image is improved as will be described later.

  Further, second invalid binary image data indicating invalid / valid of the short exposure valid area image data is generated. In the second invalid binary image data, the value of the pixel data at the same pixel position as the pixel data of which the value of the first invalid binary image data is “1” (valid) is “1”. Pixels for which data is valid are set to “1”, and other pixels are set to “0”. Then, the short exposure effective area image data is output to the effective image compression unit 22b, and the second invalid binary image data is output to the third valid / invalid image generation unit 224c and the exposure code image generation unit 228, respectively.

  The third valid / invalid image generation unit 224c includes the ultrashort exposure image data from the image storage unit 20, the third mask data from the third mask data generation unit 220c, and the second valid / invalid image generation unit 224a. The second invalid binary image data is input, and among the pixel data of the ultrashort exposure image data, the second invalid binary image data has a value “0” (saturated) and the third mask data has a value “ “0” (unsaturated) pixels are validated and other pixels are invalidated. Based on this validity / invalidity, ultrashort exposure effective area image data is generated. That is, in the ultra-short exposure effective area image data, the effective pixel is converted to “0” while the effective pixel is left as it is, and the invalid pixel is converted to “0”. In the ultra-short exposure effective area image data, all pixels in which the standard exposure effective area image data and the short exposure effective area image data are valid are converted to “0”. The non-saturated pixel data does not overlap at the same pixel position between the area image data and the ultrashort exposure effective area image data.

  Further, third invalid binary image data indicating invalidity / validity of the ultrashort exposure valid area image data is generated. In the third invalid binary image data, the value of the pixel data at the same pixel position as the pixel data of which the value of the second invalid binary image data is “1” (valid) is “1”. Pixels for which image data is valid are set to “1”, and other pixels are set to “0”. Then, the ultrashort exposure effective area image data is output to the effective image compression unit 22b, and the third invalid binary image data is output to the fourth effective / invalid image generation unit 224d and the exposure code image generation unit 228, respectively.

  The fourth valid / invalid image generation unit 224d receives the ultra-short exposure image data from the image storage unit 20, the fourth mask data from the fourth mask data generation unit 220c, and the third valid / invalid image generation unit 224a. Of the third invalid binary image data, and among the pixel data of the ultra-short exposure image data, the value is “0” (saturated) in the third invalid binary image data and the value in the fourth mask data. “0” (unsaturated) pixels are valid, and other pixels are invalidated. Based on the validity / invalidity, ultra-short exposure effective area image data is generated for the image. That is, in the ultra-short exposure effective area image data, the effective pixel is converted to “0” while the effective pixel is left as it is, and the invalid pixel is converted to “0”. In the ultra-short exposure effective area image data, all the valid pixels of the standard exposure effective area image data, the short exposure effective area image data, and the ultra-short exposure effective area image data are converted to “0” (invalid). Therefore, the non-saturated pixel data between the standard exposure effective area image data, the short exposure effective area image data, the ultra short exposure image data, and the ultra ultra short exposure effective area image data is not overlapped at the same pixel position. Become.

  Further, fourth invalid binary image data indicating invalid / valid of the ultra-short exposure effective area image data is generated. In the fourth invalid binary image data, the value of the pixel data at the same pixel position as the pixel data of which the value of the third invalid binary image data is “1” is “1”, and the ultra-short exposure effective area image data “1” is an effective pixel and “0” is otherwise. Then, the ultra-short exposure effective area image data is output to the effective image compression unit 22b, and the fourth invalid binary image data is output to the exposure code image generation unit 228.

  The exposure code image generation unit 228 uses the 4-bit pixel data of the first invalid binary image data, the second invalid binary image data, the third invalid binary image data, and the fourth invalid binary image data for each pixel. An exposure code is generated. Here, the exposure code is a code that gives an exposure amount. Now consider the vector of equation (1) below.

Vector = (first invalid binary image data, second invalid binary image data, third invalid binary image data, fourth invalid binary image data) (1)
Using the above generation method, the vectors shown in the above equation (1) are (1, 1, 1, 1), (0, 1, 1, 1), (0, 0, 1, 1), (0 , 0, 0, 1) are generated. Then, an exposure code is assigned according to these values. In the present invention, an exposure code is assigned in accordance with the method described in Japanese Patent Laid-Open No. Hei 6-292202. The method counts the occurrence frequency (total number of pixels) of four vectors. At this time, a 2-bit exposure code “00” is assigned to the most frequent vector, “01” is assigned to the next most frequent vector, “10” is assigned to the next most frequent vector, and “10” is assigned to the least frequently occurring vector. 11 ”is allocated. This makes it possible to allocate a code suitable for information compression (low entropy). Then, exposure code image data composed of these 2-bit exposure codes is generated. The exposure code image data is specifically data in which the pixel positions of the non-saturated pixel data constituting the effective area image data corresponding to the exposure times and the exposure codes are associated with each other on a one-to-one basis. It becomes. The exposure code is associated with information on the exposure time of each corresponding effective area image data. This exposure time information is used when the HDR image generating apparatus 300 corrects the luminance level for the effective area image data corresponding to the exposure time other than the standard exposure time. Further, the exposure code image generation unit 228 outputs the generated exposure code image data to the effective image compression unit 22b.

  In the effective image compression unit 22b, the standard exposure effective area image data, the short exposure effective area image data, the ultrashort exposure effective area image data, the ultra ultrashort exposure effective area image data, and the exposure code image data are used to form an effective image. Input from the unit 22a. Then, the effective image compression unit 22b performs compression processing on each effective area image data in consideration of network delay, communication speed, and the like. Therefore, although not shown, each effective area image data is buffered (accumulated). Here, when it is desired to increase the compression rate, compression is performed using an irreversible compression method such as JPEG, and when priority is given to image quality, compression is performed using a reversible compression method such as TIFF. In addition, you may compress independently for every four image data, and you may compress four images as one image. This is possible because there is only one effective image at the corresponding pixel position of the four images. On the other hand, the exposure code image data is compressed by another method. That is, since it is necessary to restore the information as accurate information in the transmission destination HDR image data generation device 300, compression is performed by a lossless compression method such as run-length compression. Then, four effective area image data and one exposure code image data corresponding to each exposure time after compression are output to the file conversion unit 22c.

The file conversion unit 22c generates HDR image compression data obtained by converting these five image data into one file from the four effective area image data and one exposure code image data input from the effective image compression unit 22b. This is output to the data transmission unit 24.
Furthermore, the internal configuration of the HDR image generation unit 32 will be described with reference to FIG. Here, FIG. 5 is a block diagram showing an internal configuration of the HDR image generation unit 32.

As shown in FIG. 5, the HDR image generation unit 32 includes a file division unit 32a, an effective image expansion unit 32b, and an image composition unit 32c.
The file dividing unit 32a converts one HDR image compressed data input from the data transmission unit 30 into standard exposure effective area image data, short exposure effective area image data, ultrashort exposure effective area image data, and ultra ultrashort exposure effective area. The image data is divided into four effective area image data and one exposure code image data, and each divided image data is output to the effective image expansion unit 32b.

  The effective image expansion unit 32b corresponds to the compression method of the image compression apparatus 200, and expands (decodes) each effective area image data compressed by lossless compression or lossy compression, and exposure compressed by lossless compression. The code image data is decompressed (decoded), and each decompressed image data is output to the image composition unit 32c.

  The image composition unit 32c receives the standard exposure effective area image data, the short exposure effective area image data, the ultra short exposure effective area image data, and the ultra ultra short exposure effective area image data that are input from the effective image expansion section 32b. Are multiplied by a constant specified by each exposure code of the exposure code image data corresponding to the pixel position of each of the pixel data, and each of the pixel data of the four effective area image data is HDR. Convert to pixel data. In the present embodiment, the constant is, for example, 1/10 of the short exposure time, 1/100 of the ultra short exposure time, 1/100 of the ultra short exposure time, with the standard exposure time (long exposure time) as a reference. Is 1/1000 of the standard, the standard exposure time is “1”, the short exposure time is “10”, the ultrashort exposure time is “100”, and the ultrashort exposure time is “1000”. I will do it.

Accordingly, the pixel value of each pixel data of the effective area image data corresponding to the standard exposure time is multiplied by 1, and the pixel value of each pixel data of the effective area image data corresponding to the short exposure time is multiplied by 10 to obtain the ultra short exposure time. The pixel value of each pixel data of the effective area image data corresponding to is multiplied by 100, and the pixel value of each pixel data of the effective area image data corresponding to the ultra-short exposure time is multiplied by 1000.
And HDR image data is produced | generated by synthesize | combining each unsaturated pixel data of four effective area | region image data after multiplication by each said constant. Further, the image composition unit 32 c outputs the generated HDR image data to the HDR image processing unit 34.

Further, the flow of operation processing of the image compression apparatus 200 will be described with reference to FIG. Here, FIG. 6 is a flowchart showing an operation process of the image compression apparatus 200.
As shown in FIG. 6, first, the process proceeds to step S <b> 100, and the standard exposure corresponding to each of the four types of exposure times of the standard exposure time, the short exposure time, the ultrashort exposure time, and the ultrashort exposure time in the image storage unit 20. When it is determined that the image data, the short exposure image data, the ultra-short exposure image data, and the ultra-short exposure image data have been stored with an amount of data that can be processed later, and it is determined that the data has been stored (Yes) Outputs each stored image data to the image compression unit 22 and proceeds to step S102. Otherwise (No), it waits until it is stored.

When the process proceeds to step S102, the image compression unit 22 generates mask data of image data corresponding to each exposure time input from the image storage unit 20 in step S100, and the process proceeds to step S104.
In step S104, in the image compression unit 22, based on the mask data generated in step S102, standard exposure effective area image data, short exposure effective area image data, ultrashort exposure effective area image data, and ultra ultrashort exposure effective area image data. The effective area image data corresponding to each of the four types of exposure times and the first to fourth invalid binary image data corresponding to the exposure times are generated, and the process proceeds to step S106.

In step S106, the image compression unit 22 generates exposure code image data based on the first to fourth invalid binary image data generated in step S104, and proceeds to step S108.
In addition, when performing hardware processing from step S102 to S106, parallel processing is possible and can be made into one step.

In step S108, the image compression unit 22 generates the four types of standard exposure effective area image data, short exposure effective area image data, ultrashort exposure effective area image data, and ultra ultrashort exposure effective area image data generated in step S104. The effective area image data is compressed using a predetermined compression method (JPEG, TIFF, etc.), and the process proceeds to step S110.
In step S110, the image compression unit 22 compresses the exposure code image data generated in step S106 using a predetermined lossless compression method (run-length compression or the like), and the process proceeds to step S112.

Note that steps S108 and S110 can be performed by dedicated hardware processing, in which case high-speed processing is possible.
In step S112, the image compression unit 22 generates HDR image compressed data in which the four types of effective area image data compressed in step S108 and one exposure code image data compressed in step S110 are combined into one file. Then, the generated HDR image compressed data is output to the data transmission unit 24, and the process proceeds to step S114.

In step S114, the data transmission unit 24 transmits the HDR image compressed data input from the image compression unit 22 to the HDR image generation device 300 via the network 700, and the process proceeds to step S100.
Furthermore, the flow of the mask data generation process in the image compression unit 22 in step S102 will be described based on FIG. Here, FIG. 7 is a flowchart showing mask data generation processing of the image compression unit 22. Note that this processing can be performed by hardware as shown in FIG.

  As shown in FIG. 7, in the mask data generation process, first, the process proceeds to step S200. In each of the first to fourth mask data generation units 220a to 220d of the effective image forming unit 22a, the standard exposure image data and the short exposure image are displayed. It is determined whether image data corresponding to each of the data, the ultra-short exposure image data, and the ultra-short exposure image data is input. If it is determined that the image data is input (Yes), the process proceeds to step S202. If not (No), it waits until it is input.

  When the process proceeds to step S202, each of the first to fourth mask data generation units 220a to 220d compares the pixel value of each pixel data of the input image data with a preset threshold value YHigh, and the pixel value is First to fourth binary image data in which “0” is associated with the pixel value when the threshold value is lower than YHigh and “1” is associated with the pixel value when the pixel value is greater than threshold value YHigh are generated, and the process proceeds to step S204. Transition.

In step S204, each of the first to fourth mask data generation units 220a to 220d performs expansion processing twice on the corresponding binary image data generated in step S202, and the process proceeds to step S206. .
In step S206, in each of the first to fourth mask data generating units 220a to 220d, the mask data is obtained by subjecting the corresponding binary image data after the expansion process in step S204 to the contraction process twice in succession. The process proceeds to step S208.

  In step S208, in each of the first to fourth mask data generation units 220a to 220d, the corresponding mask data generated in step S206 corresponds to the first to fourth valid / invalid image generation units 224a to 224d. The result is output to the valid / invalid image generation unit, and the process proceeds to step S200.

  Further, based on FIG. 8, the flow of processing for generating the standard exposure effective area image data and the first invalid binary image data in the image compression unit 22 in step S104 will be described. Here, FIG. 8 is a flowchart showing the generation processing of the standard exposure effective area image data and the first invalid binary image data of the image compression unit 22. Note that this processing can also be performed by dedicated hardware processing using a multiplexer or a gate circuit.

  As shown in FIG. 8, the standard exposure effective area image data and the first invalid binary image data are generated by first proceeding to step S300, and the first valid / invalid image generation unit 224a receives the standard data from the image storage unit 20. It is determined whether exposure image data has been input and first mask data has been input from the first mask data generation unit 220a. If it is determined that it has been input (Yes), the process proceeds to step S302. (No) waits until input.

  When the process proceeds to step S302, the first valid / invalid image generation unit 224a validates pixel data at the same pixel position as the pixel position that is not saturated (0) in the first mask data in the standard exposure image data. It is determined that the other pixel data is invalid, and standard exposure effective area image data obtained by converting those values into “0” is generated, and the process proceeds to step S304.

In step S304, the first valid / invalid image generation unit 224a generates first invalid binary image data in which the valid (pixel value itself) value in the standard exposure valid area image data generated in step S302 is converted to 1. Then, the process proceeds to step S306.
In step S306, the first valid / invalid image generation unit 224a outputs the standard exposure effective image data generated in step S302 to the effective image compression unit 22b, and the process proceeds to step S308.

  In step S308, the first valid / invalid image generation unit 224a outputs the first invalid binary image data generated in step S304 to the second valid / invalid image generation unit 224b and the exposure code image generation unit 228, respectively. The process ends.

  Further, based on FIG. 9, the flow of processing for generating effective area image data and invalid binary image data corresponding to each exposure time other than the standard exposure time in the image compression unit 22 in step S <b> 104 will be described. Here, FIG. 9 is a flowchart showing the generation processing of the effective area image data and invalid binary image data corresponding to each effective exposure time other than the standard exposure time of the image compression unit 22. Note that this processing can also be performed by dedicated hardware processing using a multiplexer or a gate circuit.

  As shown in FIG. 9, the generation processing of the valid area image data and invalid binary image data first proceeds to step S400, and any one of the second to fourth valid / invalid image generation units 224b to 224d performs self It is determined whether or not corresponding image data, mask data, and invalid binary image data have been input. If it is determined that they have been input (Yes), the process proceeds to step S402. If not (No), input is performed. Wait until

  When the process proceeds to step S402, any one of the second to fourth valid / invalid image generation units 224b to 224d performs invalid binary in the input image data based on the mask data and invalid binary image data corresponding to itself. Among the pixel data at the same pixel position as the pixel data of the pixel data that is invalid “0” in the image data, the pixel data at the same pixel position as the pixel data of the pixel data that is not saturated (0) in the mask data Is determined to be valid, the value is left as it is, other pixel data is determined to be invalid, and the effective area image data obtained by converting those values to “0” is generated, and the process proceeds to step S404.

  In step S404, in any one of the second to fourth valid / invalid image generation units 224b to 224d, based on the valid area image data generated in step S402 and the invalid binary image data input in step S400, the valid area The value of valid (pixel value itself) in the image data is converted to 1, and the value of the pixel data at the same pixel position as the pixel data that is valid “1” in the invalid binary image data in the previous stage is set to “1”. Conversion is performed to generate invalid binary image data, and the process proceeds to step S406.

  In step S406, in any one of the second to fourth valid / invalid image generation units 224b to 224d, the effective area image data generated in step S402 is output to the effective image compression unit 22b, and the process proceeds to step S408.

  In step S408, in any one of the second to fourth valid / invalid image generation units 224b to 224d, the invalid binary image data generated in step S404 is used as the next-stage valid / invalid image generation unit and exposure code image data generation. The data is output to the unit 228, and the process is terminated. Note that the fourth valid / invalid image generation unit 224d outputs the generated fourth invalid binary image data only to the exposure code image data generation unit 228 because there is no subsequent valid / invalid image generation unit.

  Further, based on FIG. 10, the flow of the process of generating the exposure code image data in the image compression unit 22 in step S110 will be described. Here, FIG. 10 is a flowchart showing the process of generating the exposure code image data of the image compression unit 22. Note that this processing can also be performed by a dedicated hardware process using a counter, a lookup table, and a gate circuit.

  As shown in FIG. 10, the process of generating the exposure code image data first proceeds to step S500, and the exposure code image data generation unit 228 performs the first processing from the first to fourth valid / invalid image generation units 224a to 224d. It is determined whether or not the fourth invalid binary image data has been input. If it is determined that the fourth invalid binary image data has been input (Yes), the process proceeds to step S502. If not (No), the process waits until it is input. .

  In step S502, in the exposure code image data generation unit 228, the four types of vectors (1, 1, 1, 1), (0, 1, 1, 1), (0, 0, 1, 1) The occurrence frequency (total number of pixels) of (0, 0, 0, 1) is counted, and the process proceeds to step S504.

  In step S504, in the exposure code image data generation unit 228, based on the counting result in step S502, the 2-bit exposure codes “00”, “01”, “10” and “11” are associated with each other and exposure code image data including these vector values is generated, and the process proceeds to step S506.

In step S506, the exposure code image data generation unit 228 outputs the exposure code image data generated in step S504 to the effective image compression unit 22b, and the process ends.
Furthermore, the flow of HDR image data generation processing in the HDR image generation apparatus 300 will be described with reference to FIG. Here, FIG. 11 is a flowchart showing processing for generating HDR image data.

  As shown in FIG. 11, the HDR image data generation processing first proceeds to step S600, where the data transmission unit 30 determines whether or not the HDR image compression data from the image compression apparatus 200 has been received. (Yes), the received HDR image compressed data is output to the HDR image generation unit 32 and the recording device 34, respectively, and the process proceeds to step S602. Otherwise (No), the received HDR image compressed data is received. stand by.

  When the process proceeds to step S602, the HDR image data (file) input from the data transmission unit 30 is converted into four compressed effective area images corresponding to each exposure time in the file division unit 32a of the HDR image generation unit 32. The data is divided into one piece of compressed exposure code image data, and these five pieces of image data are output to the effective image expansion unit 32b, and the process proceeds to step S604.

  In step S604, the effective image expansion unit 32b expands each of the four compressed effective area image data, expands the compressed exposure code image data, and combines these five expanded image data into an image composition unit. It outputs to 32c, and it transfers to step S606.

  In step S606, the image composition unit 32c corrects the luminance level of each pixel data of the four effective area image data expanded in step S604 based on the exposure code image data expanded in step S604, and the process proceeds to step S608. To do. Specifically, as described above, the luminance level is corrected by multiplying each pixel data of the effective area image data corresponding to each exposure code by a constant corresponding to each exposure code of the exposure code image data.

In step S608, the image combining unit 32c combines the four effective area image data whose luminance levels have been corrected in step S606 to generate HDR image data, and proceeds to step S610.
In step S610, the image composition unit 32c outputs the HDR image data generated in step S608 to the HDR image processing unit 34, and the process proceeds to step S600.

Next, the actual operation of the present embodiment will be described with reference to FIGS.
Here, FIG. 12A is a diagram illustrating an example of an image to be imaged, and FIGS. 12B to 12E are diagrams illustrating an example of an effective area image corresponding to each exposure time. These are figures which show the exposure code image corresponding to the effective area | region image of (b)-(e). FIGS. 13A to 13E are diagrams showing first to fourth invalid binary image data corresponding to the effective area images of FIGS. 12B to 12E, respectively.

  First, in the multi-stage exposure camera 100, an imaging target is imaged with four types of exposure times, that is, a standard exposure time, a short exposure time, an ultrashort exposure time, and an ultraultra short exposure time. Standard exposure image data, short exposure image data, ultra-short exposure image data, and ultra-short exposure image data are independently compressed through four output channels (CH1 to CH4) for each pixel line. The image is output to the image storage unit 20 of the apparatus 200.

When image data corresponding to each exposure time is input from the multistage exposure camera 100, the image storage unit 20 stores the input image data in a corresponding storage area for each exposure time. The obtained image data is independently output to the HDR image compression unit 20b for each exposure time (step S100).
When the image data corresponding to each exposure time is input from the image storage unit 20, the image compression unit 22 generates mask data in the first to fourth mask data generation units 220a to 220d (step S102).

  In the mask data generation process, when image data is input (step S200), first, the comparator compares the value of each pixel data in the captured image data with the threshold value YHigh, and the pixel in the input image data. Binary image data is generated in which the value of the data is equal to or less than the threshold YHigh, “0”, and the value greater than the threshold YHigh is “1” (step S202).

  Next, two expansion processes are continuously performed on the generated binary image data (step S204), and the binary image data after the two expansion processes are further processed twice. Shrinkage processing is continuously performed to generate mask data in which isolated pixel data is removed from binary image data (isolated pixel non-saturation value “0” is converted to saturation “1”) (step S206). The mask data generation processing is performed in the first to fourth mask data generation units 220a to 220d, respectively, and the first mask data generated by the first mask data generation unit 220a is used as the first valid / invalid image generation unit. The second mask data generated by the second mask data generation unit 220b is displayed in 224a, the third mask data generated by the third mask data generation unit 220c is displayed in the second valid / invalid image generation unit 224b, The fourth mask data generated by the fourth mask data generation unit 220d is output to the fourth valid / invalid image generation unit 224d to the valid / invalid image generation unit 224c (step S208).

  On the other hand, in the first to fourth valid / invalid image generation units 224a to 224d, when mask data and image data corresponding to each of the first to fourth valid / invalid image generation units 224a to 224d are input, valid region image data and invalid binary image data are generated based on these. (Step S104).

  Hereinafter, the generation process of the effective area image data and the invalid binary image data is specifically assumed on the assumption that the imaging target image captured by the multistage exposure camera 100 is the one shown in the example of FIG. explain. Here, in the example of FIG. 12A, for example, it is assumed that “mountain”, “cloud”, “sky = lake”, and “sun” are in order from the lowest luminance. Further, in the first mask data corresponding to the standard exposure image data having the longest exposure time, the value of the pixel data corresponding to the “mountain” portion in FIG. 12A becomes “unsaturated (0)”. It is assumed that the other pixel data values are “saturated (1)”. Similarly, in the second mask data corresponding to the short exposure time image data, the value of the pixel data corresponding to the “mountain” and “cloud” portions in FIG. 12A is “unsaturated (0)”. In the third mask data corresponding to the ultrashort exposure time image data, it is assumed that the value of the other pixel data is “saturation (1)”. , “Cloud”, “sky”, and “lake” pixel data values are “unsaturated (0)”, and “sun” portion pixel data values are “saturated ( 1) ”, and in the fourth mask data corresponding to the ultra-short exposure time image data, the pixel data values of all the images in FIG. 12A become“ unsaturated (0) ”. Suppose that

In the above example of conditions, the generation processing of the valid area image data and invalid binary image data in the first to fourth valid / invalid image generation units 224a to 224d will be described.
First, when the standard exposure image data and the first mask data are input (step S300), the first valid / invalid image generation unit 224a uses the first mask data in the standard exposure image data based on the first mask data. Standard exposure effective area image data is generated by leaving the value of the pixel data at the pixel position that is not saturated (0) as it is and converting the values of the other pixel data to “0”. That is, as shown in FIG. 12B, the pixel data corresponding to the “mountain” portion of the captured image is left valid and the pixel data corresponding to the portion other than the “mountain” portion is invalidated, and the value is set to “0”. The standard exposure effective area image data converted into "" is generated (step S302). The standard exposure effective area image data is output to the effective image compression unit 22b.

  Furthermore, when the first valid / invalid image generation unit 224a generates the standard exposure valid area image data, next, as shown in FIG. 13A, it is determined that the standard valid valid area image data is valid. First invalid binary image data having a configuration in which the value of the pixel data corresponding to the “mountain” portion is converted to “1” and the values of the other pixel data are kept “0” (step S304). . Further, the generated first invalid image data is output to the second valid / invalid image generation unit 224b and the exposure code image data generation unit 228, respectively (step S308).

  On the other hand, when the short exposure image data, the second mask data, and the first invalid binary image data are input (step S400), the second valid / invalid image generation unit 224b receives the first invalid 2 in the short exposure image data. Among the pixel data of the same pixel position as the invalid “0” pixel position in the value image data, the pixel data of the same pixel position as the pixel position “unsaturated (0)” in the second mask data is obtained. It is determined to be valid, and the value is left as it is, and the other pixel data is converted to “0”. That is, as shown in FIG. 12C, the pixel data corresponding to the “cloud” portion of the captured image is left valid, and the pixel data corresponding to the portion other than the “cloud” portion is invalidated and the value is set to “0”. The short-exposure effective area image data converted into "" is generated (step S402). The short exposure effective area image data is output to the effective image compression unit 22b.

  Further, when the second valid / invalid image generation unit 224b generates the short exposure valid area image data, the second valid / invalid image generation unit 224b next generates the short exposure valid area image data and the first invalid binary image data as shown in FIG. As shown, the pixel data at the pixel position of the pixel data having the value corresponding to the “mountain” portion of the first invalid binary image data in the short exposure effective area image data and the effective in the short exposure effective area image data. The second invalid binary image data having a configuration in which the pixel data value corresponding to the “cloud” portion determined as “1” is converted to “1” and the other pixel data values are left as “0” is generated. (Step S404). Then, the generated second invalid image data is output to the third valid / invalid image generation unit 224c and the exposure code image data generation unit 228, respectively (step S408).

  On the other hand, the third valid / invalid image generation unit 224c is shown in FIG. 12D based on the third mask data and the second invalid binary image data in the same procedure as the second valid / invalid image generation unit 224b. As described above, the pixel data corresponding to the “sky” and “lake” portions of the captured image are left valid, and the pixel data corresponding to other than the “sky” and “lake” portions are invalidated and the value is set to “0”. The ultra-short exposure effective area image data converted into the image is generated, and the generated ultra-short exposure effective area image data is output to the effective image compression unit 22b (step S402).

  Furthermore, when the third valid / invalid image generation unit 224c generates the ultrashort exposure effective area image data, the third effective / invalid image generation unit 224c then generates the ultrashort exposure effective area image data and the second invalid binary image data, as shown in FIG. ), The pixel data at the same pixel position as the pixel data whose values corresponding to the “mountain” and “cloud” portions of the second invalid binary image data in the ultrashort exposure effective area image data are “1”, and The pixel data values corresponding to the “sky” and “lake” portions determined to be valid in the ultrashort exposure effective area image data are converted to “1”, and the other pixel data values remain “0”. The third invalid binary image data having the structure as described above is generated (step S404). Then, the generated third invalid image data is output to the fourth valid / invalid image generation unit 224d and the exposure code image data generation unit 228, respectively (step S408).

  On the other hand, the fourth valid / invalid image generation unit 224d performs the same procedure as the second and third valid / invalid image generation units 224b and 224c based on the fourth mask data and the third invalid binary image data. As shown in (e), the pixel data corresponding to the “sun” portion of the captured image is left valid, the pixel data corresponding to the portion other than the “sun” portion is invalidated, and the value is converted to “0”. The ultra-short exposure effective area image data is generated, and the generated ultra-short exposure effective area image data is output to the effective image compression unit 22b (step S402).

  Further, after the fourth valid / invalid image generation unit 224d generates the ultra-short exposure effective area image data, next, based on the ultra-short exposure effective area image data and the third invalid binary image data, FIG. As shown in (d), the value corresponding to the “mountain”, “cloud”, “sky” and “lake” portions of the third invalid binary image data in the ultra-short exposure effective area image data is “1”. The pixel data corresponding to the “sun” portion determined to be valid in the pixel data at the same pixel position as the pixel data and the ultra-short exposure effective area image data is converted to “1”, and the other pixel data The fourth invalid binary image data having a configuration in which the value of “0” is kept “0” is generated (step S404). Then, the generated fourth invalid binary image data is output to the exposure code image data generation unit 228 (step S408).

  On the other hand, the exposure code image data generation unit 228 receives the first to fourth invalid binary image data from the first to fourth valid / invalid image generation units 224a to 224d (step S500). 1, 1, 1, 1), (0, 1, 1, 1), (0, 0, 1, 1), (0, 0, 0, 1), the number of pixels in each of the four types of vectors. Count (step S502). In the example of FIG. 12, as a result of counting, “total number of pixels of (0, 0, 1, 1)> total number of pixels of (1, 1, 1, 1)> pixels of (0, 1, 1, 1)” The total number of pixels of each vector can be obtained by the relationship of “total number of pixels> total number of pixels of (0, 0, 0, 1)”. Therefore, as shown in FIG. 12 (f), the pixel data (corresponding to “sky” and “lake”) of the vector (0, 0, 1, 1) is assigned a 2-bit exposure code “00” to the vector (1 , 1, 1, 1) pixel code (corresponding to “mountain”) with an exposure code “01”, and pixel data (vector corresponding to “0, 1, 1, 1”) (corresponding to “cloud”) with an exposure code “10”. ”Is associated with the pixel data (corresponding to“ sun ”) of the vector (0, 0, 0, 1) with the exposure code“ 11 ”, and these are synthesized as shown in FIG. The exposure code image data having the configuration is generated (step S504). Then, the generated exposure code image data is output to the effective image compression unit 22b (step S506).

  The effective image compression unit 22b includes four effective area image data of standard exposure effective area image data, short exposure effective area image data, ultrashort exposure effective area image data, and ultra ultrashort exposure effective area image data, and one exposure code. When image data is input, for example, the four effective area image data are compressed using JPEG compression, which is one of irreversible compression methods with a high compression rate, and these four compressed data are converted into a file conversion unit 22c. (Step S108).

  As described above, the effective area image data in which the values of the pixel data of the invalid portion are all set to “0” while the value of the pixel data of the valid portion remains unchanged is compressed, so that the effective portion is the same as normal image data. The invalid portion can be compressed at a compression rate higher than the compression rate of normal image data. For example, the amount of data is significantly larger than that of compressing four image data corresponding to each exposure time as it is. The amount of data can be reduced, and the amount of data can be made smaller than that obtained by compressing one piece of HDR image data as it is.

  Furthermore, the exposure code image data that needs to be accurately restored to the data before compression is compressed using run-length compression, which is one of the lossless compression methods, and this compressed data is output to the file conversion unit 22c ( Step S110).

  Since the exposure code image data is also an enumeration of four types of 2-bit codes, compression can be performed at a high compression rate even in lossless run-length compression. The exposure code image data is used to correct the brightness level when generating the HDR image data, and each effective area image is compared with the HDR image data after the brightness level is corrected. The data is in a state where the amount of information (entropy) is compressed. Therefore, as described above, since the entropy is compressed, the total data amount of the effective area image data can be made smaller than the data amount of the HDR exposure image data.

The file conversion unit 22c generates HDR image compressed data having a configuration in which the four effective area image data after compression corresponding to each exposure time and the exposure code image data after compression are combined into one file, and this is converted into data The data is output to the transmission unit 24 (step 112).
The data transmission unit 24 transmits the generated HDR image compressed data to the HDR image generation device 300 via the network 700 (step S114).

  When the data transmission unit 30 receives the HDR image compression data from the image compression device 200, the HDR image generation device 300 outputs the received HDR image compression data to the file division unit 32a of the HDR image generation unit 32 (step). S600). The file dividing unit 32a divides the input HDR image compressed data into four compressed effective area image data and one compressed exposure code image data, and the divided five image data are effective images. The data is output to the decompression unit 32b (step S602). When the five image data are input to the effective image expansion unit 32b, the four effective area image data are expanded by the expansion method for JPEG compression, and the exposure code image data is expanded by the expansion method for run-length compression. The expanded five image data are output to the image composition unit 32c (step S604). When the expanded effective area image data and the exposure code image data are input, the image composition unit 32c converts the pixel data at the same pixel position as the pixel position of each exposure code of the exposure code image data in each effective area image data. On the other hand, the brightness level of each effective area image data is corrected by multiplying each exposure code by a preset constant (step S606). Specifically, since the exposure code is as shown in FIG. 12F, for example, as described above, the short exposure time is 1/10 of the standard with the standard exposure time (long exposure time) as a reference. If the ultra-short exposure time is 1/100 of the standard and the ultra-short exposure time is 1/1000 of the standard, the constant of the exposure code “01” corresponding to the standard exposure time is “1”, which corresponds to the short exposure time. The constant of the exposure code “10” is “10”, the constant of the exposure code “00” corresponding to the ultrashort exposure time is “100”, and the constant of the exposure code “11” corresponding to the ultrashort exposure time is “1000”. Is set.

  Therefore, the luminance value indicated by the pixel data of the “mountain” portion in the standard exposure effective area image data after expansion is multiplied by 1, and the luminance indicated by the pixel data of the “cloud” portion in the short exposure effective area image data after expansion. The luminance value indicated by the pixel data of the “sky” and “lake” portions in the extended ultrashort exposure effective area image data is multiplied by 100, and the extended ultrashort exposure effective area image data The luminance level is corrected by multiplying the luminance value indicated by the pixel data of the “sun” portion by 1000.

  Furthermore, the image composition unit 32c synthesizes the pixel data of the portions determined to be valid in the four effective area image data whose luminance levels have been corrected to generate HDR image data. Thus, for example, assuming that the dynamic range of only the standard exposure time of the multistage exposure camera 100 is 60 dB, the dynamics of the HDR image data generated from the effective area image data corresponding to the above four types of exposure times. The range is 120 dB, and the dynamic range can be expanded as much as 60 dB.

  The HDR image generation unit 32 outputs the HDR image data generated as described above to the HDR image processing unit 36, and the HDR image processing unit 36 outputs the HDR image data to the computer 400 and the display device 600. The computer 400 performs processing according to the application such as image analysis based on the HDR image data, while the display device 600 displays the HDR image based on the HDR image data. Since the HDR image compressed data is held in the recording device 34, it is possible to reuse past image data or analyze it again.

  As described above, in the camera system 1 according to the present embodiment, the image compression apparatus 200 leaves the unsaturated pixel data in the image data corresponding to each of the four types of exposure times as valid and leaves the saturated pixel data. The effective area image data corresponding to each exposure time in which the value is set to “0” as invalid is generated, and the exposure for correcting the luminance level when generating the HDR image data for the effective portion of each effective area image data Generates exposure code image data with a configuration that associates codes and combines these exposure codes, and compresses four effective area image data and one exposure code image data corresponding to each exposure time to generate compressed HDR image data Then, it can be transmitted to the HDR image generating apparatus 300 via the network 700. Therefore, the effective portion can be compressed at the same compression rate as that of normal image data, and the invalid portion can be compressed at a compression rate higher than that of normal image data, and the luminance level can be obtained using the exposure code image data. Since each of the effective area image data is in a state in which the information amount (entropy) is compressed, the image data corresponding to each exposure time is not compressed as it is. The amount of data can be reduced, and the amount of data can be reduced as compared with a case where HDR image data is generated in advance and compressed. That is, the data transmission amount can be reduced as compared with the conventional compression method.

In the above embodiment, the image storage unit 20 corresponds to the image data storage unit of the form 1 or 11, the saturation region detection unit 220a ₁ corresponds to the determination unit of any _one of the forms 1, 5, and 11, The effective area image data generation process in the effective image forming unit 22a corresponds to the effective image data generation unit of any one of Embodiments 1, 2, 5, and 11, and the effective area image data compression process in the effective image compression unit 22b. Corresponds to the image compression means of the form 1 or 11, and the generation processing of the exposure code image data in the effective image forming unit 22a corresponds to the exposure time information generation means of any one of the forms 1, 3 and 11, and is effective. The compression processing of the exposure code image data in the image compressing unit 22b corresponds to the exposure time information compressing unit of mode 4, and is performed in the first to fourth mask data generating units 220a to 220d. Generation processing of binary image data corresponds to the binary image data generating means of the fifth, closing processing in the first to fourth mask data generation unit 220a~220d corresponds to the noise removal means in the form 5.

In the above embodiment, the effective image decompression unit 32b of the HDR image generation unit 32 corresponds to the effective image data restoration unit of the eighth or eleventh aspect, and the luminance level correction processing in the image composition unit 32c is performed in the eighth or eleventh embodiment. 11 and the HDR image data generation process in the image composition unit 32c corresponds to the form 8 or 11 HDR image data generation unit.
Moreover, in the said embodiment, step S202 respond | corresponds to the determination step of any one of form 6, 7, 12, and 13, and step S104 is the effective image of any one of form 6, 7, 12, and 13. Corresponding to the data generation step, Step S106 corresponds to the exposure time information generation step of any one of Embodiments 6, 7, 12, and 13, and Step S108 is any one of Embodiments 6, 7, 12, and 13. Corresponds to the image compression step.

  In the above embodiment, step S604 corresponds to the effective image data restoration step of any one of forms 9, 10, 12, and 13, and step S606 is any one of forms 9, 10, 12, and 13. Step S608 corresponds to the HDR image data generation step of any one of Embodiments 9, 10, 12 and 13.

  In the above-described embodiment, the image compression apparatus 200 and the HDR image generation unit 300 are configured mainly by hardware. However, the present invention is not limited to this, and may be configured by software itself or a mixture of software and hardware. . In this case, the image compression apparatus 200 and the HDR image generation unit 300 include a processor, a RAM (Random Access Memory), and a ROM (Read Only Memory) in which a dedicated program is stored. By executing the program, the above functions are performed. In addition, each of the above-described units may include a unit that performs its function only with a dedicated program, a unit that controls the hardware with a dedicated program, and performs the function.

1 is a block diagram illustrating an overall configuration of a camera system 1. FIG. 3 is a diagram illustrating an internal configuration of an image compression unit 22. FIG. It is a figure which shows the internal structure of the effective image formation part 22a. It is a figure which shows the internal structure of the mask data generation part 220a. 3 is a block diagram showing an internal configuration of an HDR image generation unit 32. FIG. 3 is a flowchart showing an operation process of the image compression apparatus 200. 6 is a flowchart showing mask data generation processing of the image compression unit 22; 4 is a flowchart illustrating a process for generating standard exposure effective area image data and first invalid binary image data by an image compression unit 22; 6 is a flowchart showing a process of generating effective area image data and invalid binary image data corresponding to each effective exposure time other than the standard exposure time of the image compression unit 22. 4 is a flowchart showing a process for generating exposure code image data by an image compression unit 22; It is a flowchart which shows the production | generation process of HDR image data. (a) is a figure which shows an example of the image of imaging object, (b)-(e) is a figure which shows an example of the effective area image corresponding to each exposure time, (f) is (b)- It is a figure which shows the exposure code image corresponding to the effective area | region image of (e). (A)-(e) is a figure which shows the 1st-4th invalid binary image data respectively corresponding to the effective area | region image of FIG.12 (b)-(e).

Explanation of symbols

100 is a multistage exposure camera, 200 is an image compression device, 300 is an HDR image generation device, 400 is a computer, 500 is a control means, 600 is a display device, 700 is a network, 20 is an image storage unit, 22 is an image compression unit, 24 is a data transmission unit, 30 is a data transmission unit, 32 is an HDR image generation unit, 34 is a recording device, 36 is an HDR processing unit, 22a is an effective image forming unit, 22b is an effective image compression unit, 22c is a file conversion unit, 32a is a file division unit, 32b is an effective image decompression unit, 32c is an image composition unit, 220a to 220d are first to fourth mask data generation units, 224a to 224d are first to fourth valid / invalid image generation units, 228 Is an exposure code image data generation unit, 220a ₁ is a saturation region detection unit, and 220a ₂ is a noise removal filter.

Claims (12)

Image data of an image picked up with a standard exposure time and N types shorter than the standard exposure time (
N is an image data storage means for storing image data of each image captured with an exposure time of 1 or more)
Determining means for determining whether or not the luminance indicated by each pixel data constituting each image data corresponding to each exposure time is saturated;
Based on the determination result of the determination means, pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time is replaced with a fixed value for each exposure time. Effective image data generating means for generating image data;
Image compression means for compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
Exposure time information generating means for generating exposure time information obtained by associating information related to each exposure time with respect to each compressed effective image data ,
The effective image data generating means generates the effective image data in order from the longest exposure time.
And generating the effective image data corresponding to each exposure time other than the standard exposure time.
In the process, the pixel data information replaced with a constant value at the time of generating the previous effective image data is used.
Based on the previous effective image data in the image data corresponding to each exposure time.
Selecting non-saturated pixel data from the pixel data replaced with the constant value, and
Image compression apparatus characterized that you pixel data other than pixel data by replacing the constant value generating the effective image data. The exposure time information generating means corresponds to each pixel data for each non-saturated pixel data based on position information of the non-saturated pixel data in each effective image data corresponding to each exposure time. image compression apparatus according to claim 1, wherein the generating the exposure time information formed by associating the information relating to the exposure time. The image compression apparatus according to any one of claims 1 to 2, characterized in that it comprises the exposure time information compression means for lossless compression with a predetermined lossless compression method wherein the exposure time information. Binary image data obtained by converting each pixel data constituting each image data corresponding to each exposure time into one of binary values indicating saturation and non-saturation based on the determination result of the determination means. Binary image data generating means for generating
Noise removal means for performing isolated point removal processing on data indicating saturation in binary image data corresponding to each exposure time generated by the binary image data generation means,
The effective image data generating means, based on the binary image data processed by the noise removal means, any one of claims 1 to 3, characterized in that to produce the effective image data 1
The image compression apparatus according to item. Whether or not the brightness indicated by each pixel data constituting the image data of the image captured at the standard exposure time and the image data of each image captured at a plurality of types of exposure times shorter than the standard exposure time is saturated A determination step for determining whether or not
Based on the determination result of the determination step, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. An image data generation step;
An image compression step of compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
An exposure time information generating step for generating exposure time information in which information relating to each exposure time is associated with each compressed effective image data , and
The effective image data generation step generates the effective image data in order from the longest exposure time, and in the generation process of the effective image data corresponding to each exposure time other than the standard exposure time, Based on the pixel data information replaced with a constant value at the time of image data generation, among the pixel data replaced with the constant value in the previous effective image data in the image data corresponding to each exposure time Including a program for causing a computer to execute processing including selecting non-saturated pixel data, replacing pixel data other than the selected pixel data with the constant value, and generating the effective image data. An image compression program. Whether or not the brightness indicated by each pixel data constituting the image data of the image captured at the standard exposure time and the image data of each image captured at a plurality of types of exposure times shorter than the standard exposure time is saturated A determination step for determining whether or not
Based on the determination result of the determination step, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. An image data generation step;
An image compression step of compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
Wherein the compression each valid image data, see contains each and exposure time information generating step of generating exposure time information formed by associating the information relating to the exposure time, and
The effective image data generation step generates the effective image data in order from the longest exposure time, and in the generation process of the effective image data corresponding to each exposure time other than the standard exposure time, Based on the pixel data information replaced with a constant value at the time of image data generation, among the pixel data replaced with the constant value in the previous effective image data in the image data corresponding to each exposure time select the desaturation of the pixel data, the image compression method of the containing Mukoto wherein step a pixel data other than the selected pixel data is replaced with the constant value to generate the valid image data. It claims 1 to restore the effective image data before compressing the effective image data compressed by the image compression apparatus according to any one of claims 4, based on the said reconstructed valid image data and the exposure time information An HDR image generation device that generates HDR (High Dynamic Range) image data,
Effective image data restoring means for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, multiplication means for multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
An HDR image generation device comprising: HDR image data generation means for generating HDR image data based on effective image data corresponding to each of the exposure times after multiplication by the multiplication means. It claims 1 to restore the effective image data before compressing the effective image data compressed by the image compression apparatus according to any one of claims 4, based on the said reconstructed valid image data and the exposure time information An HDR image data generation program for generating HDR (High Dynamic Range) image data,
An effective image data restoring step for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, a multiplication step of multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
HDR based on effective image data corresponding to each exposure time after multiplication in the multiplication step.
An HDR image data generation program comprising a program for causing a computer to execute processing including an HDR image data generation step for generating image data. It claims 1 to restore the effective image data before compressing the effective image data compressed by the image compression apparatus according to any one of claims 4, based on the said reconstructed valid image data and the exposure time information An HDR image data generation method for generating HDR (High Dynamic Range) image data,
An effective image data restoring step for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, a multiplication step of multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
HDR based on effective image data corresponding to each exposure time after multiplication in the multiplication step.
HDR image data generating step for generating image data
Image data generation method. Imaging means for imaging an imaging object with a standard exposure time and the N types of exposure times shorter than the standard exposure time (N is a natural number of 1 or more);
Image data of the image captured by the imaging means and captured at the standard exposure time, and the N
Image data output means for separately outputting image data of each image captured at each type of exposure time;
Image data of the image picked up by the standard exposure time output by the output means and the N
Image data storage means for storing image data of each image captured at each type of exposure time;
Determining means for determining whether or not the luminance indicated by each pixel data constituting each image data corresponding to each exposure time is saturated;
Based on the determination result of the determination means, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. Image data generating means;
Image compression means for compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
Exposure time information generating means for generating exposure time information in which information related to each exposure time is associated with each compressed effective image data;
Effective image data restoring means for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, multiplication means for multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
HDR image data generation means for generating HDR image data based on the effective image data corresponding to each exposure time after multiplication by the multiplication means ,
The effective image data generating means generates the effective image data in order from the longest exposure time.
And generating the effective image data corresponding to each exposure time other than the standard exposure time.
In the process, the pixel data information replaced with a constant value at the time of generating the previous effective image data is used.
Based on the previous effective image data in the image data corresponding to each exposure time.
Selecting non-saturated pixel data from the pixel data replaced with the constant value, and
The image processing system characterized that you generate the valid image data by the pixel data other than pixel data is replaced with the constant value. Image data of an image captured at the standard exposure time and each pixel data constituting the image data of each image captured at the N types of exposure times (N is a natural number of 1 or more) shorter than the standard exposure time. A determination step for determining whether or not the indicated luminance is saturated;
Based on the determination result of the determination step, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. An image data generation step;
An image compression step of compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
An exposure time information generating step for generating exposure time information in which information relating to each exposure time is associated with each compressed effective image data;
An effective image data restoring step for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, a multiplication step of multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
The effective image data generation step generates the effective image data in order from the longest exposure time, and in the generation process of the effective image data corresponding to each exposure time other than the standard exposure time, Based on the pixel data information replaced with a constant value at the time of image data generation, among the pixel data replaced with the constant value in the previous effective image data in the image data corresponding to each exposure time Selecting non-saturated pixel data and replacing the pixel data other than the selected pixel data with the constant value to generate the effective image data,
HDR based on effective image data corresponding to each exposure time after multiplication in the multiplication step.
An image processing program comprising a program for causing a computer to execute a process including an HDR image data generation step for generating image data. Image data of an image captured at the standard exposure time and each pixel data constituting the image data of each image captured at the N types of exposure times (N is a natural number of 1 or more) shorter than the standard exposure time. A determination step for determining whether or not the indicated luminance is saturated;
Based on the determination result of the determination step, effective image data is generated by replacing pixel data other than the pixel data determined to be non-saturated among the pixel data of each image data corresponding to each exposure time with a constant value. An image data generation step;
An image compression step of compressing the effective image data corresponding to each exposure time using a predetermined image compression method;
An exposure time information generating step for generating exposure time information in which information relating to each exposure time is associated with each compressed effective image data;
An effective image data restoring step for restoring effective image data compressed using the predetermined compression method;
Based on the restored effective image data corresponding to the exposure times and the exposure time information corresponding to the effective image data, the pixel data not replaced with the constant value in the effective image data. On the other hand, a multiplication step of multiplying a constant determined from the ratio between the standard exposure time and each exposure time;
The effective image data generation step generates the effective image data in order from the longest exposure time, and in the generation process of the effective image data corresponding to each exposure time other than the standard exposure time, Based on the pixel data information replaced with a constant value at the time of image data generation, among the pixel data replaced with the constant value in the previous effective image data in the image data corresponding to each exposure time Selecting non-saturated pixel data and replacing the pixel data other than the selected pixel data with the constant value to generate the effective image data,
HDR based on effective image data corresponding to each exposure time after multiplication in the multiplication step.
An image processing method comprising: an HDR image data generation step for generating image data.

Images