JP2017183870A - Imaging processing device, imaging processing method, computer program and electronic equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging processing device that can realize suppression of a blur of a composite image and settable range extension of exposure time at the same time in executing imaging based on a plurality of frames by means of one time shutter trigger.SOLUTION: An imaging processing device is provided, the imaging processing device comprising a control unit that generates a first image by performing a first exposure on a pixel for first exposure time, and generates a second image by performing a second exposure on a pixel for a second exposure time following the first image. The control unit minimizes an interval between a lead start of the first exposure and a shutter start of the second exposure in a predetermined row of the pixel.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to an imaging processing apparatus, an imaging processing method, a computer program, and an electronic apparatus.

  The digital still camera has a function of setting a plurality of frames in advance and imaging a plurality of consecutive frames while automatically changing the setting one by one from a predetermined imaging trigger as a starting point. There is something. Of these settings, the function of imaging while changing the exposure system settings such as exposure time, gain, flash, and high sensitivity mode is called AEB (Auto Exposure Bracketing) (see, for example, Patent Document 1).

JP 2009-284181 A

  Existing AEB images multiple frames at the same frame rate. Therefore, when the exposure time is short, a large interval is generated between the frames. Blur (subject blurring) becomes noticeable when captured images with large intervals between frames are combined. On the other hand, blurring can be suppressed by narrowing the readout interval, but on the other hand, the upper limit of the maximum exposure time is shortened, and the settable range of exposure time is narrowed.

  Therefore, in the present disclosure, when performing imaging in a plurality of frames with a single shutter trigger, it is possible to simultaneously realize suppression of blurring of a composite image and widening of a settable range of exposure time. An imaging processing apparatus, an imaging processing method, a computer program, and an electronic device are proposed.

  According to the present disclosure, a first image is generated by exposing a pixel by first exposure with a first exposure time, and a pixel is exposed by second exposure with a second exposure time following the first image. An imaging processing apparatus comprising: a control unit that generates a second image, wherein the control unit minimizes an interval between a read start of the first exposure and a shutter start of the second exposure in a predetermined row of the pixels. Provided.

  According to the present disclosure, the first image is generated by exposing the pixel by the first exposure with the first exposure time, and the pixel is exposed by the second exposure by the second exposure time following the first image. Providing an imaging processing method including: generating a second image; and minimizing an interval between the read start of the first exposure and the shutter start of the second exposure in a predetermined row of the pixels. Is done.

  Further, according to the present disclosure, the computer generates a first image by exposing the pixel by the first exposure according to the first exposure time, and then the pixel is performed by the second exposure by the second exposure time following the first image. A computer that generates a second image by exposure and minimizes the interval between the read start of the first exposure and the shutter start of the second exposure in a predetermined row of the pixels. A program is provided.

  Moreover, according to this indication, an electronic device provided with the said imaging processing apparatus is provided.

  As described above, according to the present disclosure, when performing imaging in a plurality of frames with a single shutter trigger, it is possible to simultaneously realize suppression of blurring of a composite image and widening of a settable range of exposure time. A new and improved imaging processing apparatus, imaging processing method, computer program, and electronic apparatus can be provided.

  Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is explanatory drawing explaining the conventional AEB function. It is explanatory drawing explaining the conventional AEB function. It is explanatory drawing explaining the conventional AEB function. It is explanatory drawing which shows the example which synthesize | combines the some image with which the exposure space | interval opened. 3 is an explanatory diagram illustrating a functional configuration example of an electronic device 10 according to an embodiment of the present disclosure. FIG. 2 is an explanatory diagram illustrating a configuration example of a sensor module 100 included in an imaging unit 11. FIG. It is explanatory drawing which shows the function structural example of the sensor module 100 which concerns on the same embodiment. It is a flowchart which shows the operation example of the sensor module 100 which concerns on the same embodiment. It is explanatory drawing which shows an example of the mode of the AEB function which the sensor module 100 concerning the embodiment performs. It is explanatory drawing which shows the example of an effect of the sensor module 100 which concerns on the same embodiment. 5 is an explanatory diagram illustrating an example of a pixel reset period in the sensor module 100 according to the embodiment. FIG. It is explanatory drawing which shows an example of the mode of the AEB function which the sensor module 100 which concerns on embodiment of the same execution performs. It is explanatory drawing which shows the example of the conversion process of the frame rate by the sensor module 100 concerning the embodiment. It is explanatory drawing which shows the example when the sensor module 100 which concerns on the same embodiment speeds up the time concerning the shutter and lead with respect to a pixel. It is explanatory drawing which shows the conversion process of the frame rate via the image memory | storage part 121 by the sensor module 100 in detail.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
1. Embodiment of the present disclosure 1.1. Outline 1.2. Functional configuration example 1.3. Configuration example of sensor module 1.4. Example of operation Summary

<1. Embodiment of the present disclosure>
[1.1. Overview]
Before describing the embodiments of the present disclosure in detail, an outline of the embodiments of the present disclosure will be described first.

  As described above, the digital still camera has a setting for a plurality of frames in advance, and images a plurality of continuous frames while automatically changing the settings one by one starting from a predetermined imaging trigger. Some are equipped with. Among these settings, a function for imaging while changing exposure system settings such as exposure time, gain, flash, and high sensitivity mode is referred to as AEB.

  The conventional AEB function images a plurality of frames at the same frame rate starting from a predetermined imaging trigger. FIG. 1 is an explanatory diagram illustrating a conventional AEB function, and is an explanatory diagram illustrating a conventional AEB function for imaging a plurality of frames at the same frame rate. The solid line in FIG. 1 indicates the shutter timing for the image sensor, and the broken line indicates the read timing for reading data from the image sensor.

  When multiple frames are imaged at the same frame rate, the data reading interval from the pixels is a steady interval as shown in FIG. Therefore, it is necessary to match the shutter timing of the pixel with the timing of reading data from the pixel. Therefore, as shown in FIG. 1, depending on the exposure time, a large blank may occur in the data read timing of the previous frame and the shutter timing of the next frame.

  FIG. 2 is an explanatory diagram for explaining a conventional AEB function for generating an image with a plurality of continuous frames by reading data from a pixel at the timing of a vertical synchronization signal (V synchronization signal). FIG. 2 shows a state in which images with different exposure times are generated in four consecutive frames from frames f1 to f4. Note that at least a reset period for resetting the pixels is required between the lead and the shutter. Therefore, the exposure interval includes this pixel reset period. In this way, when an image is generated with a plurality of continuous frames by reading data from the pixel at the timing of the vertical synchronization signal, the exposure interval becomes longer depending on the exposure time.

  FIG. 3 is an explanatory diagram for explaining a conventional AEB function for generating an image with a plurality of continuous frames by reading data from a pixel at the timing of a vertical synchronization signal (V synchronization signal). FIG. 3 shows a state in which images having different exposure times are generated in four consecutive frames from frames f1 to f4, as in FIG. Thus, when the exposure time is shorter than the interval of the vertical synchronization signal, the exposure interval between the data read of the previous frame and the shutter of the next frame becomes longer.

  As described above, when a plurality of images having an exposure interval are generated, subject blurring may occur when the plurality of images are combined.

  FIG. 4 is an explanatory diagram showing an example of combining a plurality of images with an exposure interval wide. FIG. 4 shows an example in which imaging with a long exposure time and imaging with a short exposure time are repeated, and an image captured with a long exposure time and an image captured with a short exposure time are combined. In the following description, imaging with a long exposure time is also referred to as “long accumulation”, and imaging with a short exposure time is also referred to as “short accumulation”.

  In the example shown in FIG. 4, an image by long accumulation and an image by short accumulation are synthesized. However, as described above, when data is read from the pixel at the timing of the vertical synchronization signal, the data of the previous frame is read. And the exposure interval between the shutter of the next frame becomes longer. Therefore, when a moving subject is continuously imaged and an image with long accumulation and an image with short accumulation are combined, blurring of the combined image increases as shown in FIG.

  In order to suppress blurring of the composite image while reading data from the pixel at the timing of the vertical synchronization signal, the data read interval from the pixel, that is, the vertical synchronization signal interval may be narrowed. However, if the interval of the vertical synchronization signals is narrowed, the upper limit of the maximum exposure time is shortened and the settable range of the exposure time is narrowed. When the settable range of the exposure time is narrowed, it becomes impossible to generate an image based on long accumulation as shown in FIG.

  Therefore, in view of the above-described points, the present disclosure has intensively studied a technique that can simultaneously realize suppression of blur and widening of a settable range of exposure time. As a result, as described below, the present inventor has devised a technology that can simultaneously realize blur suppression and widening of a settable range of exposure time by minimizing the data reading interval from the pixel. It came to do.

  The outline of the embodiment of the present disclosure has been described above.

[1.2. Functional configuration example]
Subsequently, a functional configuration example of the electronic device according to the embodiment of the present disclosure will be described. FIG. 5 is an explanatory diagram illustrating a functional configuration example of the electronic device 10 according to the embodiment of the present disclosure. Hereinafter, a functional configuration example of the electronic device 10 according to the embodiment of the present disclosure will be described with reference to FIG.

  As illustrated in FIG. 5, the electronic device 10 according to the embodiment of the present disclosure includes an imaging unit 11, an image processing unit 12, a display unit 13, a control unit 14, a storage unit 15, and an operation unit 16. And comprising.

  The imaging unit 11 includes a lens, a sensor module, and the like, and accumulates electrons for a predetermined period according to an image formed on the light receiving surface of the sensor module through the lens. The imaging unit 11 performs predetermined signal processing on a signal corresponding to the accumulated electrons. Then, the imaging unit 11 outputs the signal after the signal processing to the image processing unit 12. The configuration of the sensor module included in the imaging unit 11 will be described in detail later.

  The imaging unit 11 includes, as the predetermined signal processing, camera shake correction processing using an electronic camera shake correction method, automatic white balance processing, automatic exposure processing, distortion correction processing, defect correction processing, noise reduction processing, high dynamic range synthesis processing, and the like. Signal processing may be performed.

  The image processing unit 12 includes, for example, an application processor (AP), and executes image processing using a signal output from the imaging unit 11. The image processing executed by the image processing unit 12 includes, for example, demosaic processing using a signal output from the imaging unit 11, display processing of the demosaiced image on the display unit 13, storage processing in the storage unit 15, and the like. There is.

  The display unit 13 is a display device configured by, for example, a liquid crystal display or an organic EL display. Display contents of the display unit 13 are controlled by the control unit 14. For example, the display unit 13 displays an image captured by the imaging unit 11 and subjected to image processing by the image processing unit 12 based on the control of the control unit 14.

  The control unit 14 includes, for example, a processor such as a CPU (Central Processing Unit), a ROM, a RAM, and the like, and controls the operation of each unit of the electronic device 10.

  The storage unit 15 is configured by a storage medium such as a flash memory or other nonvolatile memory. The storage unit 15 stores an image captured by the imaging unit 11 and subjected to image processing by the image processing unit 12. The image stored in the storage unit 15 can be displayed on the display unit 13 in accordance with the operation of the user of the electronic device 10.

  The operation unit 16 is a device for operating the electronic device 10 and includes, for example, buttons and a touch panel. When the operation unit 16 includes a touch panel, the touch panel is provided on the display surface of the display unit 13. When the user of the electronic device 10 wants to record the image captured by the imaging unit 11 in the electronic device 10, the user triggers a shutter trigger by operating a predetermined button of the operation unit 16. When the imaging unit 11 and the image processing unit 12 detect the occurrence of the shutter trigger, the imaging unit 11 and the image processing unit 12 execute a process for recording an image on the electronic device 10 according to the generation of the shutter trigger.

  The electronic device 10 illustrated in FIG. 5 is not limited to a specific device, and may take various forms such as a digital camera, a smartphone, a tablet portable terminal, a portable music playback device, and a game machine.

  Heretofore, the functional configuration example of the electronic device 10 according to the embodiment of the present disclosure has been described. Next, a configuration example of the image sensor included in the imaging unit 11 of the electronic device 10 according to the embodiment of the present disclosure will be described.

[1.3. Example of sensor module configuration]
FIG. 6 is an explanatory diagram illustrating a configuration example of the sensor module 100 included in the imaging unit 11. The sensor module 100 according to the embodiment of the present disclosure is an example of the image processing apparatus of the present disclosure, and is configured by stacking three substrates as illustrated in FIG. 6. The sensor module 100 according to the embodiment of the present disclosure has a configuration in which a pixel substrate 110, a memory substrate 120, and a signal processing substrate 130 are stacked in this order.

  The pixel substrate 110 is a substrate having an image sensor composed of pixel regions in which unit pixels are formed in an array. Each unit pixel receives light from a subject, photoelectrically converts the incident light, accumulates charges, and outputs the charges as pixel signals at a predetermined timing. Pixel signals output from the pixel substrate 110 are stored in the memory substrate 120, and signal processing is performed in the signal processing substrate 130. The pixel substrate 110 includes an AD converter that converts an analog signal into a digital signal. That is, the pixel signal output from the pixel substrate 110 is a digital signal.

  The memory substrate 120 is a substrate having a memory such as a DRAM (Dynamic Random Access Memory) that temporarily stores pixel signals output from the pixel substrate 110. The memory substrate 120 has a capacity capable of temporarily storing pixel signals of a plurality of frames. The pixel signal stored in the memory substrate 120 is read based on a read command from the signal processing substrate 130.

  The signal processing board 130 performs various signal processes on the pixel signals stored in the memory board 120. The signal processing executed by the signal processing board 130 is signal processing related to image quality with respect to the pixel signals stored in the memory board 120. For example, camera shake correction processing by an electronic camera shake correction method, automatic white balance processing, automatic exposure processing, distortion Signal processing such as correction processing, defect correction processing, noise reduction processing, and high dynamic range synthesis processing can be executed. In addition, the signal processing board 130 can execute a synthesis process of images captured in a plurality of frames.

  6 illustrates a configuration in which the sensor module 100 is stacked in the order of the pixel substrate 110, the memory substrate 120, and the signal processing substrate 130. However, the present disclosure is not limited to such an example. For example, the sensor module 100 may have a configuration in which a pixel substrate 110, a signal processing substrate 130, and a memory substrate 120 are stacked in this order.

  The configuration example of the sensor module 100 has been described above with reference to FIG. Subsequently, a functional configuration example of the sensor module 100 will be described.

  FIG. 7 is an explanatory diagram illustrating a functional configuration example of the sensor module 100 according to the embodiment of the present disclosure. Hereinafter, a functional configuration example of the sensor module 100 according to the embodiment of the present disclosure will be described with reference to FIG.

  The pixel substrate 110 includes an image sensor 111 having a pixel region in which unit pixels are formed in an array, and a control unit 112 that supplies a predetermined clock signal and timing signal to the image sensor 111. A pixel signal output from the image sensor 111 in response to a signal from the control unit 112 is once sent to the signal processing board 130 and then sent to the memory board 120.

  In the present embodiment, the image sensor 111 uses a CMOS image sensor, and a pixel signal generated by exposure is read out by a rolling shutter system. As will be described later, when executing the AEB function, the control unit 112 sets an interval between the read start of exposure (first exposure) in a certain frame and the shutter start of exposure (second exposure) in the next frame. A timing signal is supplied to the image sensor 111 so as to minimize it. In this way, the control unit 112 supplies the timing signal to the image sensor 111, so that the sensor module 100 according to the embodiment of the present disclosure simultaneously realizes suppression of blur and widening of a settable range of exposure time. be able to.

  The memory substrate 120 includes an image storage unit 121 configured by a DRAM (Dynamic Random Access Memory) or the like. The image storage unit 121 temporarily stores pixel signals output from the image sensor 111. For example, the image storage unit 121 has a capacity capable of temporarily storing pixel signals of a plurality of frames. The pixel signal stored in the image storage unit 121 is read based on a read command from the signal processing board 130.

  The signal processing board 130 includes a pre-processing unit 131 and a post-processing unit 132.

  The preprocessing unit 131 performs signal processing on the pixel signal output from the image sensor 111. The preprocessing unit 131 stores the pixel signal after the signal processing in the image storage unit 121. The signal processing executed by the preprocessing unit 131 can include, for example, gain adjustment processing, clamping processing, pixel addition processing, and the like.

  The post-processing unit 132 performs signal processing on the pixel signal stored in the image storage unit 121. When the post-processing unit 132 performs signal processing on the pixel signal stored in the image storage unit 121, the post-processing unit 132 outputs the pixel signal after the signal processing to the image processing unit 12. The signal processing executed by the post-processing unit 132 can include, for example, automatic white balance processing, automatic exposure processing, distortion correction processing, defect correction processing, noise reduction processing, high dynamic range synthesis processing, and the like. Further, the post-processing unit 132 can execute a composition process of images captured in a plurality of frames.

  The functional configuration example of the sensor module 100 according to the embodiment of the present disclosure has been described above. The sensor module 100 according to the embodiment of the present disclosure has such a configuration, so that blurring of composite images and exposure are performed when performing imaging in a plurality of frames with a single shutter trigger, like the AEB function. It is possible to simultaneously realize a wide setting range of time.

[1.4. Example of operation]
Subsequently, an operation example of the sensor module 100 according to the embodiment of the present disclosure will be described. FIG. 8 is a flowchart illustrating an operation example of the sensor module 100 according to the embodiment of the present disclosure. FIG. 8 illustrates an operation example of the sensor module 100 according to the embodiment of the present disclosure when a plurality of images are continuously captured with a single shutter trigger as in the AEB function. Hereinafter, an operation example of the sensor module 100 according to the embodiment of the present disclosure will be described with reference to FIG.

  When a plurality of M images are continuously captured with a single shutter trigger, the sensor module 100 determines the N + 1th image from the read time of the Nth image (N is an integer of 1 or more and less than M) of consecutive images. It is determined whether the exposure time is longer (step S101). The determination in step S101 can be executed by the control unit 112, for example.

  Whether or not the exposure time of the (N + 1) th sheet is longer than the read time of the Nth sheet of consecutive images means that the (N + 1) th sheet is longer than the reading time from the first line to the last line of the image sensor 111 for the Nth sheet. This means whether the exposure time is long.

  If the exposure time of the (N + 1) th sheet is longer than the read time of the Nth sheet of consecutive images (step S101, Yes), the sensor module 100 performs the Nth sheet for a row with pixels provided in the image sensor 111. The interval between the start of reading and the start of the (N + 1) th shutter is minimized (step S102). The control in step S102 can be executed by the control unit 112, for example.

  FIG. 9 is an explanatory diagram illustrating an example of a state of the AEB function executed by the sensor module 100 according to the embodiment of the present disclosure. FIG. 9 shows a state in which images having different exposure times are generated in four consecutive frames from frames f1 to f4.

  As described above, at least a reset period for resetting the pixels is required between the lead and the shutter. This reset period is a period corresponding to a period required for predetermined AD conversion in a pixel, for example, as will be described later. In the present embodiment, the sensor module 100 minimizes the distance between the lead and the shutter. In the sensor module 100, for example, the interval between the lead and the shutter is narrowed to an interval corresponding to a reset period for resetting pixels provided in the image sensor 111. That is, in this embodiment, as shown in FIG. 9, the interval of the vertical synchronization signal is not constant. The interval between the vertical synchronization signal for reading the signal from the pixel in frame f1 and the vertical synchronization signal for reading the signal from the pixel in frame f2 corresponds to the exposure time of the image in frame f2 plus the reset period. To do.

  FIG. 10 is an explanatory diagram illustrating an effect example of the sensor module 100 according to the present embodiment. The sensor module 100 according to the present embodiment minimizes the interval between the lead and the shutter, thereby blurring the synthesized image in the case where the image by the long accumulation and the image by the short accumulation are alternately captured and synthesized. It can be kept small. For example, the post-processing unit 132 executes this synthesis process. This is because the amount of movement of the subject can be minimized by minimizing the distance between the lead and the shutter.

  Here, an example of a pixel reset period in the sensor module 100 according to the present embodiment will be described. FIG. 11 is an explanatory diagram illustrating an example of a pixel reset period in the sensor module 100 according to the present embodiment. A vertical synchronization signal and a horizontal synchronization signal are supplied to the pixel.

  In this embodiment, one access unit of the image sensor 111 is 32 lines. This one access unit means that eight lines are read simultaneously, and AD conversion for converting the read analog data into digital data is performed four times.

  Read access and shutter access cannot be performed simultaneously for a pixel in one access unit. Therefore, the interval from the lead to the shutter is set with the period required for the four AD conversions as a reset period. Note that the amount of one access unit varies depending on the image sensor. Accordingly, the reset period also varies depending on the image sensor.

  When the factors affecting the reset period are arranged, there are two: the number of AD conversions during one horizontal period and the number of lines that can be read simultaneously. In the sensor module 100 according to the present embodiment, the interval between the lead and the shutter is narrowed to the interval corresponding to the reset period determined by these two elements.

  On the other hand, if the N-th read time of consecutive images is less than the (N + 1) -th exposure time (No in step S101), the sensor module 100 determines the N-th sheet for a row with pixels provided in the image sensor 111. The interval between the N-th read start and the N + 1-th shutter start is minimized so that the N + 1th lead and the (N + 1) th lead do not overlap (step S103). The control in step S103 can be executed by the control unit 112, for example.

  If the exposure time is shortened, the N-th lead time and the (N + 1) -th lead time may overlap when the interval between the N-th lead start and the (N + 1) -th shutter start is too short.

  FIG. 12 is an explanatory diagram illustrating an example of a state of the AEB function executed by the sensor module 100 according to the embodiment of the present disclosure. FIG. 12 shows a state in which images with different exposure times are generated in four consecutive frames from frames f1 to f4.

  In the example shown in FIG. 12, the read period (read period 1) of the frame f1 and the read period (read period 2) of the frame f2 overlap. Similarly, the read period 2, the read period of the frame f3 (read period 3), the read period 3 and the read period of the frame f4 (read period 4) overlap each other.

  In this way, when imaging is performed under an exposure time condition in which the lead period overlaps between the preceding and following frames, the sensor module 100 outputs the image of the later frame to the outside (for example, the image processing unit 12) as it is. I can't do it.

  Therefore, in the present embodiment, the sensor module 100 may adjust the shutter start timing so that the lead period does not overlap between the previous and next frames. That is, the sensor module 100 may increase the interval between the lead between the frame and the shutter from the time corresponding to the reset period. By adjusting the shutter start timing so that the lead period does not overlap between the previous and next frames, the sensor module 100 can output an image to the outside (for example, the image processing unit 12) even if a frame with a short exposure time occurs. I can do it.

  The sensor module 100 according to this embodiment includes an image storage unit 121. Therefore, the sensor module 100 according to the present embodiment stores the data read from the image sensor 111 once in the image storage unit 121 after minimizing the interval between the lead and the shutter between frames to a time corresponding to the reset period. Then, it may be output to the outside (for example, the image processing unit 12).

  FIG. 13 is an explanatory diagram illustrating an example of frame rate conversion processing by the sensor module 100 according to the present embodiment. FIG. 13 shows a state in which the sensor module 100 according to the present embodiment stores an image read from the image sensor 111 once in the image storage unit 121 and outputs the image. As described above, the image read from the image sensor 111 is stored once in the image storage unit 121, and the images stored in the image storage unit 121 are sequentially read, so that the sensor module 100 according to the present embodiment can perform the read period. Even when the frames overlap in the preceding and following frames, the interval between the lead and the shutter between the frames can be minimized to a time corresponding to the reset period.

  The sensor module 100 according to the present embodiment can also speed up the time required for shutter and reading for the pixels. FIG. 14 is an explanatory diagram illustrating an example in which the sensor module 100 according to the present embodiment speeds up the time required for shutter and reading for a pixel. As described above, the sensor module 100 can speed up the time required for the shutter and the lead for the pixel so that the lead period does not overlap between the previous and next frames.

  Note that if the time required for shutter and reading for the pixels is increased, the interface band for outputting from the sensor module 100 to the outside may be insufficient. Considering such a case, the sensor module 100 according to the present embodiment stores an image read from the image sensor 111 in the image storage unit 121 once, and the image stored in the image storage unit 121 is converted into a frame rate. May be converted and output to the outside (for example, the image processing unit 12). For example, when data can be read from the image sensor 111 at a high speed (for example, at a speed of about 120 fps to 240 fps) but cannot be output from the sensor module 100 at that speed, the sensor module 100 can be read from the image sensor 111. The read image may be stored once in the image storage unit 121, and the image stored in the image storage unit 121 may be converted to, for example, 30 fps and output to the outside.

  FIG. 15 is an explanatory diagram showing the frame rate conversion processing by the sensor module 100 via the image storage unit 121 in more detail. When reading from the image sensor 111, the sensor module 100 is faster than the frame rate for outputting an image from the sensor module 100 (for example, the frame rate for outputting an image is 30 fps, and the frame rate for reading from the image sensor 111 is 120 fps to Read out at a speed of about 240 fps), and the image is stored in the image storage unit 121 once. When the sensor module 100 outputs an image to the outside, the sensor module 100 executes a predetermined subsequent process at a rate limited to the interface band of the external output and outputs the image.

  As described above, the sensor module 100 according to the present embodiment can convert the reading speed from the image sensor 111 and the output speed from the sensor module 100 through the image storage unit 121.

  The sensor module 100 according to the present embodiment may perform imaging with the same exposure time for all images when capturing a plurality of images with a single shutter trigger. And the sensor module 100 can output the image which suppressed noise by performing the 3D noise reduction (3DNR) process which synthesize | combines several images by the same exposure time. Of course, the sensor module 100 minimizes the period between the lead and the shutter between the previous and next frames when capturing a plurality of images with the same exposure time. When the sensor module 100 according to the present embodiment executes the 3DNR process, each pixel signal is stored once in the image storage unit 121, and then the pixel signal stored in the image storage unit 121 by the post-processing unit 132. Is read out and the composition processing is performed.

  The sensor module 100 according to the present embodiment can narrow the search range by minimizing the period between the lead and the shutter between the frames before and after, since there is less subject blur during 3DNR. Therefore, the sensor module 100 according to this embodiment can reduce the circuit for performing 3DNR by minimizing the period between the lead and the shutter between the front and rear frames.

  The sensor module 100 according to the present embodiment synthesizes a predetermined number of images, for example, six images, with a single shutter trigger when combining the above-described image with long accumulation and the image with short accumulation or when performing 3DNR processing. Images may be taken, or a predetermined number or more, for example, 7 or more images may be taken. When the sensor module 100 according to the present embodiment captures more than a predetermined number of images, the sensor module 100 selects a predetermined number from the images and synthesizes an image based on long accumulation and an image based on short accumulation, or 3DNR. Processing may be performed.

<2. Summary>
As described above, according to the embodiment of the present disclosure, there is provided the sensor module 100 that minimizes the period between the lead and the shutter between the previous and next frames when a plurality of images are captured with a single shutter trigger. The

  The sensor module 100 according to the embodiment of the present disclosure minimizes the period between the lead and the shutter between the front and rear frames, thereby simultaneously realizing the suppression of blur and the wide setting range of the exposure time. Can do.

  Each step in the processing executed by each device in the present specification does not necessarily have to be processed in time series in the order described as a sequence diagram or flowchart. For example, each step in the processing executed by each device may be processed in an order different from the order described as the flowchart, or may be processed in parallel.

  In addition, it is possible to create a computer program for causing hardware such as a CPU, ROM, and RAM incorporated in each device to exhibit functions equivalent to the configuration of each device described above. A storage medium storing the computer program can also be provided. In addition, by configuring each functional block shown in the functional block diagram with hardware or a hardware circuit, a series of processing can be realized with hardware or a hardware circuit.

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

  For example, in the above-described embodiment, the sensor module 100 having a configuration in which the pixel substrate 110, the memory substrate 120, and the signal processing substrate 130 are stacked in this order is shown, but the present disclosure is limited to such an example. It is not something. The sensor module 100 may have a structure in which two substrates of a pixel substrate 110 and a signal processing substrate 130 are stacked. In this case, the sensor module 100 has a structure in which the pixel substrate 110 is stacked on the signal processing substrate 130.

  Further, the effects described in the present specification are merely illustrative or exemplary and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. With a control unit,
The control unit is an imaging processing apparatus that minimizes an interval between a read start of the first exposure and a shutter start of the second exposure in a predetermined row of the pixels.
(2)
The imaging processing apparatus according to (1), wherein the lead period of the first exposure and the lead period of the second exposure do not overlap.
(3)
The imaging processing apparatus according to (1) or (2), wherein the first exposed pixel and the second exposed pixel are the same.
(4)
The control unit according to any one of (1) to (3), wherein an interval between the read start of the first exposure and the shutter start of the second exposure is an interval corresponding to a reset period of the pixel. The imaging processing apparatus described.
(5)
The imaging processing apparatus according to (4), wherein the reset period is a period corresponding to a period required for predetermined AD conversion of data read from the pixels.
(6)
When the second exposure time is longer than the read time of the first exposure, the control unit minimizes the interval between the read start of the first exposure and the shutter start of the second exposure in a predetermined row of the pixel. ,
When the second exposure time is less than the lead time of the first exposure, the lead of the first exposure in a predetermined row of the pixel under the condition that the lead period of the first exposure and the lead period of the second exposure do not overlap. The imaging processing apparatus according to any one of (1) to (5), wherein an interval between the start and the shutter start of the second exposure is minimized.
(7)
The imaging processing apparatus according to any one of (1) to (6), further including an image processing unit that synthesizes the first image and the second image.
(8)
The imaging processing apparatus according to any one of (1) to (7), further including a storage unit that stores the first image and the second image by the control unit.
(9)
The speed at which the first image and the second image are stored in the storage unit from the control unit and read from the storage unit is lower than the speed at which the first image and the second image are directly read from the pixels. Imaging processing apparatus.
(10)
The imaging processing apparatus according to (8) or (9), wherein the control unit is configured to change a speed at which the first image and the second image are read from the storage unit.
(11)
It is configured by laminating two semiconductor substrates consisting of a first semiconductor substrate and a second semiconductor substrate,
The first semiconductor substrate includes at least the pixel and the control unit,
The image processing apparatus according to any one of (1) to (7), wherein image processing is performed on the first image and the second image on the second semiconductor substrate.
(12)
It is configured by stacking three semiconductor substrates consisting of a first semiconductor substrate, a second semiconductor substrate, and a third semiconductor substrate,
The first semiconductor substrate includes at least the pixel and the control unit,
The third semiconductor substrate is formed with at least the storage unit,
The imaging processing apparatus according to any one of (8) to (10), wherein image processing is performed on the first image and the second image on the second semiconductor substrate.
(13)
The image processing apparatus according to (12), wherein the second semiconductor substrate is provided between the first semiconductor substrate and the third semiconductor substrate.
(14)
The image processing apparatus according to (12), wherein the third semiconductor substrate is provided between the first semiconductor substrate and the second semiconductor substrate.
(15)
The control unit further generates a sixth image from the third image by exposing the pixels from the third exposure time to the sixth exposure time through the third exposure time to the sixth exposure time, respectively, following the second image. The imaging processing apparatus according to any one of (1) to (14).
(16)
A storage unit for storing at least the sixth image from the first image by the control unit;
The imaging processing apparatus according to (15), wherein the control unit reads the sixth image from the first image from the storage unit after storing the sixth image from the first image in the storage unit.
(17)
The control unit further generates a seventh image by exposing the pixels by the seventh exposure with at least a seventh exposure time following the sixth image, and selects six images from the generated series of images. The imaging processing apparatus according to (16).
(18)
The imaging processing apparatus according to any one of (1) to (17), wherein the first exposure time is longer than the second exposure time.
(19)
The imaging processing apparatus according to any one of (1) to (18), wherein the first image and the second image are read from the pixels by a rolling shutter.
(20)
The imaging processing apparatus according to any one of (1) to (17), wherein the first exposure time and the second exposure time are equal.
(21)
A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. And
Minimizing the interval between the read start of the first exposure and the shutter start of the second exposure in a predetermined row of the pixels;
An imaging processing method.
(22)
On the computer,
A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. And
Minimizing the interval between the read start of the first exposure and the shutter start of the second exposure in a predetermined row of the pixels;
A computer program that executes
(23)
An electronic apparatus comprising the imaging processing device according to any one of (1) to (20).

10 Electronic device 100 Sensor module

Claims (23)

A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. With a control unit,
The control unit is an imaging processing apparatus that minimizes an interval between a read start of the first exposure and a shutter start of the second exposure in a predetermined row of the pixels.   The imaging processing apparatus according to claim 1, wherein a lead period of the first exposure and a lead period of the second exposure do not overlap.   The imaging processing apparatus according to claim 1, wherein the first exposed pixel and the second exposed pixel are the same.   The imaging processing apparatus according to claim 1, wherein the control unit sets an interval between the read start of the first exposure and the shutter start of the second exposure as an interval corresponding to a reset period of the pixels.   The imaging processing apparatus according to claim 4, wherein the reset period is a period corresponding to a period required for predetermined AD conversion of data read from the pixels. When the second exposure time is longer than the read time of the first exposure, the control unit minimizes the interval between the read start of the first exposure and the shutter start of the second exposure in a predetermined row of the pixel. ,
When the second exposure time is less than the lead time of the first exposure, the lead of the first exposure in a predetermined row of the pixel under the condition that the lead period of the first exposure and the lead period of the second exposure do not overlap. The imaging processing apparatus according to claim 1, wherein an interval between a start and a shutter start of the second exposure is minimized.   The imaging processing apparatus according to claim 1, further comprising an image processing unit that synthesizes the first image and the second image.   The imaging processing apparatus according to claim 1, further comprising a storage unit that stores the first image and the second image by the control unit.   The speed at which the first image and the second image are stored in the storage unit from the control unit and read from the storage unit is lower than a speed at which the first image and the second image are directly read from the pixel. Imaging processing device.   The imaging processing apparatus according to claim 8, wherein the control unit is configured to change a speed at which the first image and the second image are read from the storage unit. It is configured by laminating two semiconductor substrates consisting of a first semiconductor substrate and a second semiconductor substrate,
The first semiconductor substrate includes at least the pixel and the control unit,
The imaging processing apparatus according to claim 1, wherein image processing is performed on the first image and the second image on the second semiconductor substrate. It is configured by stacking three semiconductor substrates consisting of a first semiconductor substrate, a second semiconductor substrate, and a third semiconductor substrate,
The first semiconductor substrate includes at least the pixel and the control unit,
The third semiconductor substrate is formed with at least the storage unit,
The imaging processing apparatus according to claim 8, wherein image processing is performed on the first image and the second image on the second semiconductor substrate.   The imaging processing apparatus according to claim 12, wherein the second semiconductor substrate is provided between the first semiconductor substrate and the third semiconductor substrate.   The imaging processing apparatus according to claim 12, wherein the third semiconductor substrate is provided between the first semiconductor substrate and the second semiconductor substrate.   The control unit further generates a sixth image from the third image by exposing the pixels from the third exposure time to the sixth exposure time through the third exposure time to the sixth exposure time, respectively, following the second image. The imaging processing apparatus according to claim 1. A storage unit for storing at least the sixth image from the first image by the control unit;
The imaging processing apparatus according to claim 15, wherein the control unit reads the sixth image from the first image after storing the sixth image from the first image in the storage unit.   The control unit further generates a seventh image by exposing the pixels by the seventh exposure with at least a seventh exposure time following the sixth image, and selects six images from the generated series of images. The imaging processing apparatus according to claim 16.   The imaging processing apparatus according to claim 1, wherein the first exposure time is longer than the second exposure time.   The imaging processing apparatus according to claim 1, wherein the first image and the second image are read from the pixels by a rolling shutter.   The imaging processing apparatus according to claim 1, wherein the first exposure time and the second exposure time are equal. A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. And
Minimizing the interval between the read start of the first exposure and the shutter start of the second exposure in a predetermined row of the pixels;
An imaging processing method. On the computer,
A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. And
Minimizing the interval between the read start of the first exposure and the shutter start of the second exposure in a predetermined row of the pixels;
A computer program that executes An electronic apparatus comprising the imaging processing apparatus according to claim 1.

Images