JPH1023465A - Image pickup method and its device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To minimize a storage capacity, to reduce a processing time and to prevent the generation of a fault in a processing after image pickup by allowing a control step to control storage processing operation according to the output of an image pickup condition detecting step. SOLUTION: Through the use of a posture detecting sensor 128 and information on picked-up image, a processing control circuit 129 controls the processing starting times of corresponding point extracting circuits 122, 123 and a solid information integrating circuit 125. An image picked-up at a prescribed position is processed by the circuits 122 and 123 to extract a distance image. Continually, even though a processing is finished before moving to a next prescribed position, images obtained during it are not processed and the operation of the circuits 122 and 123 and the scanning of image pickup elements 114 and 115 are stopped. Thereby, power consumption of a picture processing circuit and the elements 114 and 115 and a peripheral circuit is reduced. Continually at the time of being positioned at the next prescribed position, processings at the circuits 122 and 123, etc., are started corresponding to the start of the vertical scanning of the elements 114 and 115.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is directed to automatically photographing an image, outputting a three-dimensional shape of a subject and a texture image of a surface from the photographed image, and sequentially storing the images for selectively displaying the photographed image. To an imaging device.

[0002]

2. Description of the Related Art Conventionally, as an example of acquiring a shape of a three-dimensional object using a plurality of images, "Stereo matching using a plurality of base line lengths" (Transactions of the Institute of Electronics, Information and Communication Engineers, D-II Vo)
l. J75-D-II No.8 pp.1317-1327 August 1992).

In this conventional example, one image pickup unit is mounted on a rail and moved in parallel to take a plurality of images, and a shape analysis is performed using mutual image correlation.

[0004] In addition, using a compound-eye imaging apparatus in which a plurality of imaging units are housed in one apparatus, two or more imaging units are simultaneously imaged.
As a technique for analyzing the shape of a three-dimensional object by using the correlation of a set of parallax images, there is Japanese Patent Publication No. Hei 7-9673. In this conventional example, an imaging device is fixed to a robot arm, and the imaging device is moved as instructed to capture an image.

On the other hand, a conventional imaging apparatus in which a photographer can freely carry the imaging apparatus main body and analyze the shape of an arbitrary subject will be described.

FIG. 9 is a diagram showing a configuration of a conventional automatic imaging apparatus which is portable and a concept of a use state. In the figure, 1
01 is a subject to be imaged (in the present embodiment, a cup)
The object 101 rests on the pad 102 and
It is assumed that the camera is shooting. A plurality of bright spot marks 103a, 103b, 103c are engraved on the pad 102, and their positional relationship is known, and is stored in advance in an imaging device 900 described later.

Reference numeral 900 denotes a portable image pickup apparatus, which includes photographing lenses 110 and 111, a shutter 112 which also serves as a lighting aperture,
113, imaging elements 114 and 115 for performing photoelectric conversion, and control circuits 116 and 1 for controlling the imaging elements 114 and 115
17, video signal processing circuits 118 and 119 for processing signals obtained from the imaging elements 114 and 115, video signal storage circuits 120 and 121 for storing video signals output from the video signal processing circuits 118 and 119, and corresponding point extraction Circuit 1
22, an imaging parameter detection circuit 123, a ROM (read only memory) 124 storing a known pad luminescent spot positional relationship, a three-dimensional information integration circuit 125, and temporarily storing the three-dimensional information integrated by the three-dimensional information integration circuit 125. It comprises buffer circuits 126 and 127.

[0008] The image pickup apparatus 900 includes a corresponding point extraction circuit 1
At the same time as extracting corresponding points from the two video signals obtained by step 22 to obtain a distance image at each time, the imaging parameter detection circuit 123 and the ROM 124 use the imaging parameters (the pad and imaging device 900 obtained from the bright point coordinates).
, And the exact focal length). The three-dimensional information integration circuit 125 calculates three-dimensional shape data and texture image data of the subject 101 based on the distance image, the imaging parameters, and the time-series change information, and stores the data in the buffer circuits 126 and 127.

[0009] In FIG.
Numeral data of the three-dimensional shape of the subject 101 output from 00 and 141 are developed image data of the surface texture of the subject 101. This output data can be transferred to a personal computer (personal computer) or the like, and can be displayed as three-dimensional CG (computer graphics) by performing texture mapping in the personal computer. And the like can be changed instantaneously, and processing such as deformation and processing can be added. Also,
By generating two CGs with slightly shifted viewpoints and outputting them to a stereoscopic display, a stereoscopic image can be viewed. In this case, the stereoscopic image can be freely rotated and deformed, so that a more realistic feeling can be obtained.

In the image pickup apparatus 900, the processing of the corresponding point extraction circuit 122 and the three-dimensional information integration circuit 125 is the most complicated and requires a long processing time, and the circuit scale and power consumption are very large. Further, in the imaging apparatus 900, a sequential processing mode in which the complicated processing is sequentially performed while shooting is performed, and a necessary image is captured into the video signal storage circuits 120 and 121 while shooting, and the processing is collectively performed after the shooting is completed. There is a batch processing mode to perform. Also, this imaging device 900
Is characterized in that the photographer can freely carry the imaging device 900 and analyze the shape of the subject 101 easily without any special preparation, without requiring a large positioning device as in the above-described conventional example.

[0011]

However, the above-mentioned prior art has the following problems.

That is, generally, the operation of the photographer cannot perform accurate positioning at a constant speed as in the above-described conventional positioning apparatus. For example, when image processing is performed by taking the image into the image image storage circuit at regular time intervals, redundant information increases in a portion moved at a lower speed than necessary, an extremely large image memory capacity is required, and a long time is required for shape analysis. I need it. Further, the analyzed data of the three-dimensional shape is also unnecessarily fine, and the generation of CG later requires more processing time and storage capacity than necessary. Conversely, when the photographer moves the imaging device at high speed, information necessary for analyzing the three-dimensional shape is insufficient, and the analysis accuracy is reduced. In the worst case, if an image of a specific side surface of the subject cannot be obtained, the shape information of that portion may be lost.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems of the prior art, and an object of the present invention is to minimize the storage capacity of storage means for storing images. It is an object of the present invention to provide an imaging method and apparatus capable of shortening the time required for processing to be performed on an image and preventing inconvenience when performing processing and display after photographing. .

[0014]

According to a first aspect of the present invention, there is provided an imaging method, comprising: an imaging step of imaging a subject image; a storage processing step of storing and processing information; And a control step of controlling the storage processing operation, wherein the control step controls the storage processing operation according to an output of the imaging condition detection step. It is characterized by the following.

[0015] In order to achieve the above object, the present invention provides a second aspect.
The imaging method described above includes an imaging step of imaging a subject image, an image analysis processing step of analyzing the captured image captured in the imaging step, and an imaging condition detection step of detecting a relative relationship between the subject and the imaging apparatus main body. And an image analysis processing control step of controlling the image analysis processing operation, wherein the image analysis processing control step controls the image analysis processing operation according to an output of the imaging condition detection step. Is what you do.

Further, in order to achieve the above object, a third aspect is provided.
The imaging method according to claim 1, wherein
The storage processing control step is characterized in that information relating to the relative relationship between the subject and the imaging device main body is controlled to be stored in the storage processing step together with the image captured in the imaging step.

[0017] In order to achieve the above object, the present invention is directed to claim 4.
The imaging method according to claim 2 is the imaging method according to claim 2,
The image analysis processing step is characterized in that an analysis operation for acquiring a three-dimensional shape and a surface image of a subject using a plurality of images is performed. In order to achieve the above object, an imaging method according to claim 5 is the imaging method according to claim 1 or 2, wherein the imaging condition detecting step includes:
It is characterized by using a sensor that detects the angle and translational movement of the imaging device main body.

According to another aspect of the present invention, there is provided a computer system comprising:
The imaging method according to claim 1 or 2, wherein the imaging condition detecting step analyzes a subject image captured by an imaging apparatus main body and an image around the subject image and captures the captured image captured in the imaging step. The motion of the angle and the parallel movement of the imaging device main body is detected by a state change.

According to another aspect of the present invention, there is provided a computer system comprising:
The imaging method described in the imaging method according to claim 1 or 2, wherein the imaging condition detecting step analyzes a subject image captured by the imaging apparatus main body and images around the subject image,
It is characterized in that a change in the relative positional relationship between the subject and the imaging apparatus main body is detected based on an error signal generated when analyzing the image.

[0020] In order to achieve the above object, the present invention provides an eighth aspect.
The imaging method according to claim 1 or 2, wherein the imaging condition detecting step analyzes a subject image captured by the imaging device body and detects a change in an occlusion state of the subject. Is what you do.

[0021] In order to achieve the above object, the present invention provides a ninth aspect.
The imaging method according to claim 1 or 2, wherein the imaging condition detecting step analyzes an object image captured by the imaging apparatus body and detects an area of an overlapping region between time-series object images. It is characterized by the following.

[0022] In order to achieve the above object, a first aspect is provided.
0. The imaging method according to claim 1, wherein the imaging condition detecting step analyzes a subject image captured by the imaging apparatus body and detects a change in a distance image of the subject. It is assumed that.

[0023] In order to achieve the above object, a first aspect is provided.
The imaging method according to claim 1, wherein in the imaging method according to claim 2, the imaging condition detecting step stops the imaging step and the image analysis processing step during a period in which storage processing and analysis processing are not performed. Is what you do.

Further, in order to achieve the above object, a first aspect is provided.
2. The imaging apparatus according to item 2, wherein the imaging unit captures a subject image, a storage processing unit that stores and processes information, an imaging condition detection unit that detects a relative relationship between the subject and the imaging device main body, and the storage processing unit. Storage processing control means for controlling, wherein the storage processing control means controls the storage processing means in accordance with an output of the imaging condition detection means.

Further, in order to achieve the above-mentioned object, a first aspect is provided.
3. An imaging apparatus according to claim 3, wherein the imaging unit captures a subject image, an image analysis processing unit analyzes the captured image captured by the imaging unit, and an imaging condition detection unit detects a relative relationship between the subject and the imaging device body. Means, and image analysis processing control means for controlling the image analysis processing means, wherein the image analysis processing control means controls the image analysis processing means according to the output of the imaging condition detection means. It is assumed that.

[0026] In order to achieve the above object, a first aspect of the present invention is provided.
The imaging device according to claim 4, wherein the storage processing control means includes information about a relative relationship between the subject and an imaging device main body together with an image captured by the imaging means. Is controlled to be stored in the memory.

Further, in order to achieve the above object, a first aspect is provided.
The imaging device according to claim 5, wherein the image analysis processing means performs an analysis operation for acquiring a three-dimensional shape and a surface image of a subject using a plurality of images. is there. In order to achieve the above object, an imaging apparatus according to claim 16 is the imaging apparatus according to claim 12 or 13, wherein the imaging condition detecting means detects an angle and a translational movement of the imaging apparatus body. Is characterized by the following.

[0028] In order to achieve the above object, a first aspect is provided.
The imaging device according to claim 7, wherein the imaging condition detection unit analyzes a subject image captured by the imaging device body and an image around the subject, and captures an image captured by the imaging unit. An angle and a translational movement of the imaging device main body are detected based on a change in an image state.

[0029] In order to achieve the above object, a first aspect is provided.
The imaging device according to claim 8, wherein the imaging condition detecting means analyzes a subject image captured by the imaging device main body and an image around the subject, and generates an image when the image is analyzed. A change in the relative positional relationship between the subject and the imaging apparatus body is detected based on the error signal.

Further, in order to achieve the above object, a first aspect is provided.
The imaging device according to claim 9, wherein the imaging condition detecting means analyzes a subject image captured by the imaging device main body and detects a change in the occlusion state of the subject in the imaging device according to claim 12 or 13. It is assumed that.

Further, in order to achieve the above object, a second aspect is provided.
14. The imaging apparatus according to claim 12, wherein the imaging condition detection unit analyzes a subject image captured by the imaging apparatus body, and detects an overlap area between time-series subject images. It is characterized by the following.

Further, in order to achieve the above object, a second aspect is provided.
14. The imaging device according to claim 1, wherein the imaging condition detecting means analyzes a subject image captured by the imaging device body and detects a change in a distance image of the subject. It is assumed that.

Further, in order to achieve the above object, a second aspect is provided.
The imaging device according to claim 2, wherein the imaging condition detection unit stops the imaging unit and the image analysis processing unit during a period in which storage processing and analysis processing are not performed. Is what you do.

[0034]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below with reference to FIGS.

(First Embodiment) First, the first embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS. FIG.
FIG. 10 is a diagram showing a configuration and a concept of a use state of an automatic imaging apparatus 100 which is an imaging apparatus according to the first embodiment of the present invention, in which the same parts as those in FIG. Are denoted by the same reference numerals. 9 differs from FIG. 9 in that an attitude detection sensor 128, a processing control circuit 129, and a subject recognition circuit 130 are added to the configuration in FIG. In FIG. 1, reference numerals 142 and 143 indicate signal lines.

In the automatic image pickup apparatus 100 of the present invention, a plurality of means can be used as the image pickup condition detecting means. For example, the posture detecting sensor 128, the image pickup parameter detecting circuit 123, the object recognizing circuit 130, and the corresponding point extracting circuit in FIG. 122 is equivalent.

The operation when each component is used will be described below.

First, the operation in the batch processing mode will be described as an example. In the automatic imaging device 100, the rotation angle and the movement amount are always detected by the posture detection sensor 128,
The processing control circuit 129 performs processing control so that the video signal is taken into the video signal storage circuits 120 and 121 each time the automatic imaging apparatus 100 changes by a predetermined position and angle. When the posture detection sensor 128 detects that the circuit 101 has made one round around the subject 101, the processing control circuit 1
Reference numeral 29 fetches an image from the video signal storage circuits 120 and 121, and starts collective processing of the corresponding point extraction circuit 122, the imaging parameter detection circuit 123, and the three-dimensional information integration circuit 125.

FIG. 2 is a diagram showing a detailed configuration of the attitude detection sensor 128. As shown in the figure, three small vibrating gyroscopes 201, 202, and 203 are arranged so as to be orthogonal to each other, and independently detect the rotational angular velocity (pitch, yaw, and roll) of the automatic imaging apparatus 100. . The integrators 203, 204, and 205 respectively integrate the detected values and convert the detected values into rotation angles of the automatic imaging apparatus 100. When the photographer is always photographing the subject 101 so as to fit on the screen, the rotation angle of the automatic imaging apparatus 100 itself is almost equal to information on how much the automatic imaging apparatus 100 has rotated around the subject 101. Based on this information, the processing control circuit 129 controls to store an image when either the pitch direction or the yaw direction changes by a predetermined angle. The change in the roll direction is not directly related to the processing control, but when the automatic imaging apparatus 100 rolls greatly and the information of the pitch and the yaw are mixed and output, these outputs are accurately separated. Used to
The configuration using this angular velocity sensor can be made very small, has high sensitivity of the sensor, and has an effect that the accuracy is very high because the integration is performed only once.

The posture detecting sensor 128 may alternatively detect acceleration by an acceleration sensor.

FIG. 3 shows acceleration sensors 301, 302, 303, 304, 30 constituting the attitude detection sensor 128.
FIG. 5 is a diagram showing the arrangement of 5,306. In general, an acceleration sensor detects only vibration in a linear direction, and therefore, as shown in FIG. 310-31
Reference numeral 5 denotes an integration circuit that performs integration twice, and integrates the output of each acceleration sensor twice to calculate a position movement amount. The integration components from the channels from the pair of acceleration sensors are added to each other, so that the translation components (X, Y, Z) in the mounting direction of the pair of acceleration sensors are subtracted from each other, and the rotation component (α) is subtracted from each other. , Β, γ). Therefore, the addition circuits 320, 321, 322
And subtraction circuits 330, 331, 332 are provided.

In the processing control circuit 129, the automatic imaging device 1
Judgment is made as to how much 00 has moved with respect to the subject 101, and the timing at which images are taken into the video signal storage circuits 120 and 121 is controlled. Although this detection method has a complicated sensor configuration, all degrees of freedom (horizontal X, vertical Y, front and rear Z, pitch α, yaw β, roll γ) of the automatic imaging apparatus 100 can be detected at the same time. This has an effect that a viewpoint change with respect to the subject 101 can be accurately detected.

Further, some methods for detecting the relative positional relationship between two objects in a non-contact body are described in "Survey of Helmet".
tracking technologies ”SPIE Vol. 1456 Large-Scree
n-Projection, Avionic, and Helmet-Mounted Displays (1
991) p.86 (hereinafter referred to as materials) explains the principle and characteristics of each system.

Attitude detection sensor 1 of automatic imaging apparatus 100
28, these principles are also applicable. This document describes the concept of analyzing a relative position from a bright spot image captured by a camera. When this technique is applied to the automatic image capturing apparatus 100, the image capturing parameters are determined using the signal lines 142 and 143 in FIG. The detection circuit 123 is always operated without passing through the video signal storage circuits 118 and 119. The image of the known bright spot pattern is analyzed by the imaging parameter detection circuit 123 to detect the movement amount and the attitude of the automatic imaging apparatus 100.

FIG. 4 is a diagram showing an example of the image storage timing of the automatic image pickup apparatus 100. In the figure, reference numeral 400 denotes a trajectory when the photographer moves the automatic imaging apparatus 100 around the subject 101 by hand. Reference numeral 401 denotes a position at which shooting is started, which is a timing at which the first image is stored. Subsequently, 402, 403, 404, 405,...
09 is the movement amount (X, Y) or the imaging system is the subject 10
1 at the timing when the rotation angle (α, β) has changed by a predetermined amount,
Images are stored at the timings of 2,403,404,405, ... 409. At the timing of the position 409 after one rotation, the coordinate X and the rotation angle β become equal to the value of the position 401, but the other values (Y, Z, α, γ) do not always match. However, in the automatic imaging apparatus 100, the start point and the end point do not need to exactly coincide with each other. When Y, Z, and α are smaller than predetermined values, the image capture ends when X and β coincide. .

The automatic image pickup apparatus 100 does not necessarily move the camera in a plane horizontal to the ground to shoot an image.
There is also a shooting method that passes over the top of the camera. Reference numeral 410 in FIG. 4 denotes a trajectory when the photographer moves the automatic imaging apparatus 100 by hand so as to pass over the subject 101 and photographs. 4
11, 412, 413,... 419 are timing positions to be stored. In this photographing mode, image capture control is performed by detecting the values of Y and α instead of X and β, and α
Is changed by 180 degrees, the image capturing is terminated.

FIG. 5 is a diagram showing an example of an input image when an image is captured at the timing of the positions 401 to 405 in FIG. As can be seen from FIGS. 11A, 11B and 11C, the time-series image is obtained as if the subject 101 was sequentially viewed from slightly different viewpoints.

FIG. 6 is a diagram showing an example of an image fetch timing different from the above. In this example, a situation is described in which a photographing method is employed in which the subject 101 does not fit on the screen using an image pickup unit with a large image pickup magnification, and the automatic image pickup apparatus 100 is moved almost in the X direction. I have. In such a photographing method, when a region overlapped with an image photographed in the past in each screen has a predetermined area, that is, a hatched portion in FIGS. 6B and 6C has a predetermined area. At a certain timing, the captured image is taken into the video signal storage circuits 120 and 121. In this method, since the imaging magnification is set to a large value, a detailed image and shape of the subject 101 can be analyzed, and an image without interruption can be stably captured by the operation of the processing control circuit 129.

In the above, the case of controlling the storage and processing of the signal from the position and the angle of the automatic image pickup apparatus 100 has been described as an example. However, as described below, the image itself of the subject 101 is analyzed to control the storage and processing. May be performed.

For example, the object recognition circuit 13 shown in FIG.
Use 0. The subject recognition circuit 130 detects a change in the subject image from a time-series change in the video signal. For example, a difference from a past image is detected, and when the difference reaches a predetermined value, a video signal is captured. Since this method does not directly detect the movement of the automatic imaging apparatus 100, the accuracy of the processing timing is low, but the processing is simple and there is no need to newly provide a sensor.
0 can be reduced in size.

Further, the corresponding point extracting circuit 122 in FIG.
May be operated at all times, and the processing control may be performed by analyzing the distance image data output from the corresponding point extraction circuit 122. When the automatic imaging device 100 moves by a predetermined amount,
The detected distance image also changes accordingly. When the time change of the distance image becomes a predetermined amount, the image may be captured. According to this method, when the subject 101 has a portion having a large unevenness, a large signal is output even if the position change of the imaging system is small. Is performed. In general, it is desired to analyze the shape of the uneven portion in detail, and according to this method, more efficient image capturing can be performed.

Similarly, the corresponding point extracting circuit 122 in FIG.
There is also a method of using an error signal output from the device. Here, the error signal is information indicating a pixel position at which a corresponding point was not correctly found when finding a corresponding point of a video signal obtained from the left and right imaging units for each pixel.
Such a phenomenon occurs when a so-called occlusion occurs in which a portion that is visible in one imaging unit is not visible in the other imaging unit, and directly reflected light from the illumination is incident on only one imaging unit. If the illumination conditions are significantly different between the left and right imaging units, such as
There is a case where the corresponding point cannot be searched because the surface of No. 1 is flat and has no texture. However, for these imaging conditions, if the viewpoint of the imaging apparatus is changed, corresponding points can be easily extracted and no error occurs.

In the automatic image pickup apparatus 100, processing control is performed at the timing when the error output of the corresponding point extraction circuit 122 changes in a time series, and an image is captured. According to this method, the features of the subject 101, which could not be accurately detected at one time due to occlusion or the like, can be supplemented with images at other times, so that a three-dimensional shape can be efficiently extracted even from a subject having severe irregularities. Can be.

Next, the flow of processing controlled by the processing control circuit 129 will be described. In the above description, the timing of capturing images into the video signal storage circuits 120 and 121 in the batch processing mode has been described.
0 also has a sequential processing mode in which shape extraction processing is performed while photographing the subject 101. In this case as well, processing more images than necessary is required by the corresponding point extraction circuits 122 and 12.
3 and the amount of operation of the three-dimensional information integration circuit 125 increases, and the output data from the signal lines 140 and 141 increases, so that the buffer circuits 126 and 127 also require a large storage capacity.

Therefore, the attitude detection sensor 128 described above is used.
The processing control circuit 129 uses the corresponding point extraction circuits 122 and 123 and the three-dimensional information integration circuit 1
25 is controlled. Specifically, when the automatic imaging apparatus 100 moves along the trajectory 400 in FIG. 4, first, the image captured at the position 401 is processed by the corresponding point extraction circuits 122 and 123 to extract a distance image. Subsequently, even if the processing is completed before moving to the position 402, the corresponding point extraction circuits 122 and 123 are stopped without processing the image acquired during the processing. Meanwhile, the image sensor 11
The video signals obtained from the image sensors 114 and 115 are discarded or the shutters 112 and 113 are closed, and the image sensors 114 and 115 are closed.
The scanning of 115 is stopped. As a result, the image processing circuit and the image sensor 1 that consume large power in the automatic image capturing apparatus 100
It is possible to reduce the power consumption of circuits around 14, 115. Subsequently, when the automatic imaging apparatus 100 is located at the position 402 in FIG. 4, the processing of the corresponding point extraction circuits 122 and 123 and the three-dimensional information integration circuit 125 is started in accordance with the start of vertical scanning of the imaging elements 114 and 115.

Here, if the automatic imaging device 100 moves quickly and the processing cannot be completed in time, the video signal storage circuits 120 and 1
21 stores the images that could not be processed sequentially.
The processing control circuit 129 detects that the processing of the corresponding point extraction circuits 122 and 123 has been completed, and transfers the next image from the video signal storage circuits 120 and 121 to the processing circuit.

In the above-described embodiment, the case where two image pickup units are used has been described as an example. However, with respect to the image capturing timing control of the present invention, the three-dimensional shape is analyzed using one image pickup unit. The same can be applied to such a device.

As described in detail above, according to the imaging apparatus of the present embodiment, image capture control and processing start are performed in accordance with the position / angle relationship between the subject 101 and the imaging apparatus 100 and the state of change of the subject image. By performing the control, the video signal storage circuits 120 and 121 and the buffer circuits 126 and 12 are controlled.
7 can be reduced to a minimum and a complicated image processing step can be processed in a minimum time.

(Second Embodiment) Next, a second embodiment of the present invention will be described with reference to FIGS.

The present embodiment is adapted to a system in which a picture is taken so as to go around a subject, image information is stored as it is, and a captured image is selected and displayed as it is, instead of a CG. Here, a case in which two imaging units are provided to facilitate realism and a stereoscopic image is displayed on a stereoscopic display will be exemplified. However, the image capturing timing of the present invention is the same for a single imaging system. .

FIG. 7 shows an image pickup apparatus 700 according to this embodiment.
FIG. 2 is a diagram showing the configuration of FIG. 1 and an operation concept at the time of acquiring an image, and the same reference numerals in FIG. 2 denote the same parts as in FIG. 1 in the first embodiment. 7 is different from FIG. 1 in that the circuits related to the stereoscopic image analysis such as the corresponding point extraction circuit 122, the imaging parameter detection circuit 123, the ROM 124, the stereoscopic information integration circuit 125, and the buffer circuits 126 and 127 are deleted from the configuration of FIG. Also, an imaging condition storage circuit 702 is added to the configuration of FIG.

In the present embodiment, the video signal storage circuit 1
20 and 121 are stored together with the imaging condition storage circuit 702 in a storage unit 701 that is detachable from the imaging device 700. After the imaging is completed, the storage unit 701 can be detached and carried separately.

In the imaging device 700 according to the present embodiment,
Information on the imaging position and angle detected by the attitude detection sensor 128 is stored in the imaging condition storage circuit 702 at the same time as the captured image.

The other configuration and operation of the imaging device 700 according to the present embodiment are the same as those of the imaging device 100 according to the above-described first embodiment, and a description thereof will be omitted.

FIG. 8 shows an imaging device 700 according to the present embodiment.
FIG. 3 is a diagram illustrating a configuration of an image display unit that displays an image captured in step (a). In the figure, reference numeral 801 denotes an image reproducing unit; 802, a three-dimensional display; 803, a three-dimensional mouse; and 804, a coordinate comparison circuit. The subject 10 is stored in the storage unit 701.
The information about the image around 1 and its shooting direction and position is stored. When the operator uses the three-dimensional mouse 803 to specify the direction in which the subject 101 is to be observed, the coordinate comparison circuit 804 determines whether an image in the specified observation direction is stored in the imaging condition storage circuit 702. And
If an image in the designated observation direction is stored, the image data is extracted from the video signal storage circuits 120 and 121 and displayed on the stereoscopic display 803. If the image in the designated observation direction is not stored, the closest image is found and displayed on the stereoscopic display 803.

Such an imaging / display system does not calculate a stereoscopic image as numerical information, but displays a stored image by selecting an image in a viewpoint direction desired by an operator, and is discrete. Can instantly display the subject image viewed from almost any direction, so that it is possible to feel as if there is a real object there.

As described in detail above, according to the imaging apparatus 700 according to the present embodiment, an image is appropriately captured at an appropriate position without the operator making a round at a constant speed. The advantage is that an image relatively close to the instruction can always be provided.

[0068]

As described above in detail, according to the imaging method and apparatus of the first and twelfth aspects of the present invention, an image used for image display and three-dimensional shape analysis always stores a minimum necessary capacity. Operation, the storage capacity of the storage processing means can be reduced, the operation time of the three-dimensional shape analysis processing means can be shortened, and the entire apparatus can be made compact and inexpensive without losing necessary images. This has the effect that it can be realized.

According to the imaging method and apparatus of the second and thirteenth aspects of the present invention, an image used for three-dimensional shape analysis is operated so as to always analyze only a necessary minimum image. The effect of shortening the operation time of the shape analysis and realizing a compact and low-cost device as a whole without losing necessary images can be achieved.

According to the imaging method and apparatus of the third and fourteenth aspects of the present invention, the operation is performed so that the captured image and the position information of the imaging device at the time of capturing the image are simultaneously stored. Therefore, it is possible to easily compare the desired observation direction information designated by the observer at the time of image reproduction, and to display an appropriate image instantaneously.

According to the imaging method and apparatus of the fourth and fifteenth aspects of the present invention, since information on the three-dimensional shape and the surface image of the subject can be acquired, the subject image can be obtained by computer graphics using a technique such as texture mapping. The effect of generating is that the observer can freely change the observation direction, the distance, the three-dimensional shape, and the surface state.

Further, according to the imaging method and apparatus of the present invention, the image is stored and processed each time the imaging apparatus moves by a predetermined position / angle relationship with respect to the subject. Since the operation is performed, there is an effect that substantially regular sampling positions can be provided in space with a simple device configuration.

Further, according to the imaging method and apparatus of the present invention, each time the state of the captured image changes, the image is stored and processed. The effect is that the interval at which the image is sampled changes appropriately in accordance with.

According to the imaging method and apparatus of the present invention, the image is stored and processed every time an error signal generated when analyzing the image changes. Since the shape information of the subject area that could not be analyzed at a certain time can be supplemented by the information analyzed by the image at a different time, an effect that always accurate three-dimensional shape data can be output.

According to the imaging method and apparatus of the present invention, a subject area not captured in an image captured in a specific direction is supplemented by a subject area image captured in another direction. Because it operates, there are very few areas that can not be analyzed even for subjects with complicated shapes,
There is an effect that highly accurate information can be output as a whole.

According to the imaging method and apparatus of the ninth and twentieth aspects of the present invention, the operation is performed so that the images are always stored and processed so that the weighted abdomen between the images has a predetermined area. In particular, when high-magnification imaging is performed, it is possible to perform a joint analysis between images with a predetermined accuracy from an image, and it is possible to obtain a desired image.

Further, according to claims 10 and 21 of the present invention,
According to the imaging method and apparatus of the above, when the distance image changes by a predetermined amount, the image is operated and stored, so that more sampling is performed in a subject area having a complicated three-dimensional shape, and high accuracy is achieved. There is an effect that three-dimensional shape data can be output.

Further, according to the present invention, there is provided a semiconductor device comprising:
According to the imaging method and apparatus described above, during the period when image storage / processing is not performed, the operation is performed to stop the imaging unit and the image analysis unit that consume relatively large power, so that the power consumption can be significantly reduced. This has the effect.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a concept of a configuration and a use state of an imaging device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of a posture detection sensor in the same device.

FIG. 3 is a diagram showing an arrangement configuration of an acceleration sensor constituting a posture detection sensor in the same device.

FIG. 4 is a diagram illustrating an example of an image capturing timing in the apparatus.

FIG. 5 is a diagram showing an example of a captured image in the same device.

FIG. 6 is a diagram showing an example of a captured image in the same device.

FIG. 7 is a diagram illustrating a concept of a configuration and a use state of an imaging device according to a second embodiment of the present invention.

FIG. 8 is a diagram showing a configuration of a stereoscopic image display unit in the same device.

FIG. 9 is a diagram illustrating a configuration and a usage concept of a conventional imaging apparatus.

[Explanation of symbols]

 REFERENCE SIGNS LIST 100 automatic imaging apparatus 101 subject 102 pad 103 a bright spot on 103 a pad 103 b bright spot on pad 103 c bright spot on pad 110 imaging lens 111 imaging lens 112 shutter 113 shutter 114 imaging device 115 imaging device 116 control circuit 117 control circuit 118 video Signal processing circuit 119 Video signal processing circuit 120 Video signal storage circuit 121 Video signal storage circuit 122 Corresponding point extraction circuit 123 Imaging parameter detection circuit 124 ROM 125 Three-dimensional information integration circuit 126 Buffer circuit 127 Buffer circuit 128 Attitude detection sensor 129 Processing control circuit 130 Subject recognition circuit 140 Signal line 141 Signal line 142 Signal line 143 Signal line 201 Small vibration gyro 202 Small vibration gyro 203 Small vibration gyro 204 Integrator 2 05 Integrator 206 Integrator 301 Acceleration sensor 302 Acceleration sensor 303 Acceleration sensor 304 Acceleration sensor 305 Acceleration sensor 306 Acceleration sensor 310 Integrator 311 Integrator 312 Integrator 313 Integrator 314 Integrator 315 Integrator 320 Addition circuit 321 Addition circuit 322 Addition Circuit 330 Subtraction circuit 331 Subtraction circuit 332 Subtraction circuit 702 Imaging condition storage circuit

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Nao Kurahashi 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc. (72) Inventor Motohiro Ishikawa 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inside the corporation

Claims (22)

[Claims] An imaging step of imaging a subject image;
A storage processing step of storing and processing information; an imaging condition detection step of detecting a relative relationship between a subject and an imaging apparatus main body; and a control step of controlling the storage processing operation. An imaging method, wherein the storage processing operation is controlled according to an output of a condition detecting step. 2. An imaging step of imaging a subject image;
An image analysis processing step of analyzing the captured image captured in the imaging step; an imaging condition detection step of detecting a relative relationship between a subject and an imaging apparatus body; and an image analysis processing control step of controlling the image analysis processing operation Wherein the image analysis processing control step controls the image analysis processing operation according to the output of the imaging condition detection step. 3. The storage processing control step is characterized in that information relating to a relative relationship between the subject and an imaging device main body is controlled so as to be stored in the storage processing step together with an image captured in the imaging step. The imaging method according to claim 1. 4. The imaging method according to claim 2, wherein the image analysis processing step performs an analysis operation for acquiring a three-dimensional shape and a surface image of the subject using a plurality of images. 5. The imaging method according to claim 1, wherein the imaging condition detecting step is performed using a sensor that detects an angle and a translational movement of the imaging apparatus main body. 6. The imaging condition detecting step includes analyzing a subject image captured by an imaging device main body and an image around the subject image, and changing an angle and a translation of the imaging device main body based on a state change of the captured image captured in the imaging step. 3. The imaging method according to claim 1, wherein the movement of the image is detected. 7. The imaging condition detecting step includes analyzing a subject image taken by the imaging device body and an image around the subject image, and determining whether the subject and the imaging device body are based on an error signal generated when the image is analyzed. 3. The imaging method according to claim 1, wherein a change in the relative positional relationship is detected. 8. The imaging method according to claim 1, wherein the imaging condition detecting step analyzes a subject image captured by the imaging device main body and detects a change in an occlusion state of the subject. 9. The imaging apparatus according to claim 1, wherein the imaging condition detecting step analyzes an object image captured by the imaging apparatus main body and detects an area of an overlapping region between time-series object images. Method. 10. The imaging method according to claim 1, wherein the imaging condition detecting step analyzes a subject image captured by the imaging device body and detects a change in a distance image of the subject. 11. The imaging method according to claim 2, wherein the imaging condition detection step stops the imaging step and the image analysis processing step during a period in which the storage processing and the analysis processing are not performed. 12. An image capturing means for capturing an image of a subject, a storage processing means for storing and processing information, an image capturing condition detecting means for detecting a relative relationship between the subject and the image capturing apparatus main body, and controlling the storage processing means. Storage processing control means,
The imaging apparatus, wherein the storage processing control unit controls the storage processing unit according to an output of the imaging condition detection unit. 13. An image capturing means for capturing an image of a subject, an image analysis processing means for analyzing the captured image captured by the image capturing means, and an image capturing condition detecting means for detecting a relative relationship between the subject and the image capturing apparatus main body. And an image analysis processing control means for controlling the image analysis processing means, wherein the image analysis processing control means controls the image analysis processing means in accordance with an output of the imaging condition detection means. Imaging device. 14. The storage processing control means controls to store information on a relative relationship between the subject and an imaging device main body in the storage processing means together with an image taken by the imaging means. The imaging device according to claim 12, wherein: 15. The imaging apparatus according to claim 13, wherein said image analysis processing means performs an analysis operation for acquiring a three-dimensional shape and a surface image of a subject using a plurality of images. 16. The imaging condition detecting means according to claim 12, wherein said imaging condition detecting means comprises a sensor for detecting an angle and a translation movement of the imaging device main body.
An imaging device according to any one of the preceding claims. 17. The image pickup condition detecting means analyzes a subject image picked up by the image pickup device main body and an image around the subject image, and changes the angle and parallelism of the image pickup device main body based on a state change of the image picked up by the image pickup means. 14. The imaging device according to claim 12, wherein a movement of the movement is detected. 18. The imaging condition detecting means analyzes a subject image taken by the imaging device body and an image around the subject image, and detects a difference between the subject and the imaging device body based on an error signal generated when the image is analyzed. 14. The imaging apparatus according to claim 12, wherein a change in a relative positional relationship is detected. 19. The imaging apparatus according to claim 12, wherein the imaging condition detection unit analyzes a subject image captured by the imaging apparatus body and detects a change in an occlusion state of the subject. 20. The imaging apparatus according to claim 12, wherein said imaging condition detecting means analyzes a subject image taken by said imaging apparatus main body and detects an area of an overlapping region between time-series subject images. apparatus. 21. The imaging apparatus according to claim 12, wherein the imaging condition detection unit analyzes a subject image captured by the imaging apparatus body and detects a change in a distance image of the subject. 22. The imaging apparatus according to claim 13, wherein said imaging condition detection means stops said imaging means and image analysis processing means during a period in which storage processing and analysis processing are not performed.