JP4862926B2 - REPRODUCTION CONTROL DEVICE, REPRODUCTION CONTROL METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to a playback control apparatus, playback control method, and program suitable for continuously viewing panoramic images.

  2. Description of the Related Art Conventionally, an electronic still camera that has been widely used converts light that has passed through a lens by capturing an image of an object into an image signal by an individual image sensor such as a CCD (Charge Coupled Device), and records the image signal on a recording medium. The recorded image signal can be reproduced. Many electronic still cameras include a monitor that displays a captured still image, and a specific one of the recorded still images is selected and displayed. Such an electronic still camera has a narrow range in which the shooting range in one shooting is limited by the angle of view of the lens, and a wide range of images exceeding the angle of view of the lens cannot be taken. Various camera systems have been proposed for shooting images.

  For example, this camera system can be broadly divided into a multi-lens system that captures a wide field of view at once with a plurality of lenses, and a monocular system that continuously captures images by shifting the imaging direction sequentially using a monocular lens. being classified. The multi-lens camera system has an advantage that a panoramic overall image can be generated by substantially the same operation as a normal camera, but has a disadvantage that the entire system is expensive.

  On the other hand, the monocular camera system can capture a panoramic image at a relatively low cost, and several methods have been proposed as a method for capturing an image while shifting the imaging direction. For example, a method of shifting or rotating the imaging system with a hand at a constant speed (see, for example, Patent Document 1), or a method of shifting and rotating the imaging system with a hand and a stationary operation (for example, see Patent Document 2). And a method of shifting, rotating, and stopping the imaging system using a stepping motor or the like (see, for example, Patent Document 3).

  Further, as a panoramic image appreciation method shot by the above-described shooting method, a landscape printing method (for example, see Patent Document 4) and scroll reproduction by a TV monitor or the like (for example, see Patent Document 5) have been proposed. Yes. For example, when displaying a panoramic image on a television monitor, if an attempt is made to display the entire panoramic image, it is difficult to check details because a partial area in the screen is displayed horizontally small. In addition, when it is desired to automatically switch and display a plurality of panorama images, each panorama image is displayed in a horizontally small size, so that it is difficult to confirm details. Therefore, when panorama images are scrolled and reproduced when browsing a plurality of panorama images continuously on a television monitor, the entire panorama images can be viewed in detail.

JP 11-88754 A Japanese Patent Laid-Open No. 11-88811 JP 2005-333396 A Japanese Patent Laid-Open No. 7-212663 JP 2002-77800 A

  However, in the methods described in Patent Documents 1 to 3, since images with different shooting times are combined, a time difference occurs in the panorama composite image screen. Therefore, even if one shooting time is recorded in association with the panorama composite image, the time required for panorama shooting is not accurately recorded, and the transition and direction of the shooting time in the panorama composite image are not known. Arise.

  In addition, the conventional method is not suitable for efficiently viewing panoramic images continuously because only the unique method of scroll playback is determined.

  The present invention solves the above-described conventional problems, and an object thereof is to provide a playback control device, a playback control method, and a program suitable for efficiently viewing panoramic images continuously.

That is, the playback control device according to the present invention relates to a panoramic image synthesized from a plurality of image data shot a plurality of times from the start of shooting to the end of shooting converted based on the image incident through the lens system, Based on the imaging direction from the start of shooting of the panoramic image to the end of shooting or the coordinate information of the subject detected by feature detection in the panoramic image, and the panoramic image based on the imaging direction or the coordinate information A scroll direction determining unit that determines a scroll direction for reproduction, and the recording unit records face coordinates based on the face image of the subject detected from the image data as the coordinate information in association with the panoramic image , The scroll direction determination unit, when the imaging direction and the face coordinates are associated with the panoramic image Determines the scroll direction based on the face coordinates.

In addition, the reproduction control method and program according to the present invention relate to a panoramic image synthesized from a plurality of image data shot a plurality of times from the start of shooting to the end of shooting converted based on an image incident through a lens system. Based on the imaging direction from the start of shooting of the panoramic image to the end of shooting or the coordinate information of the subject detected by the feature detection in the panoramic image in the recording unit, and the imaging direction or the coordinate information A scroll direction determining step for determining a scroll direction when reproducing the panoramic image, and in the recording step, face coordinates based on the face image of the subject detected from the image data are used as the coordinate information. recorded in association with, in the scroll direction determining process, the imaging direction and the face coordinate the panorama When associated with the image, determining the scrolling direction based on the face coordinates.

  According to the present invention, when scrolling and reproducing a plurality of panoramic images, an image that a viewer wants to browse can be browsed efficiently in a short time.

It is a block diagram which shows the structural example of the imaging device which concerns on this Embodiment. It is a schematic diagram for demonstrating an example of a trimming process. It is a schematic diagram for explaining an imaging direction when reproducing (a) a subject at the start of photographing and (b) a subject immediately before the end of photographing. It is a block diagram which shows the structural example of the imaging device which concerns on this Embodiment. It is a flowchart for demonstrating the process sequence which synthesize | combines a panoramic image. It is a block diagram which shows the structural example of the reproduction | regeneration control apparatus which concerns on this Embodiment. It is a flowchart for demonstrating an example of the scroll method of a panoramic image. It is a schematic diagram for demonstrating an example of the scrolling method of a panoramic image. It is a schematic diagram for demonstrating an example of the scrolling method of a panoramic image. It is a schematic diagram for demonstrating an example of the scrolling method of a panoramic image. It is a schematic diagram for demonstrating an example of the scrolling method of a panoramic image. It is a schematic diagram for demonstrating an example of the speed which scrolls a panoramic image. It is a schematic diagram for demonstrating the scroll method of a panoramic image.

  Hereinafter, specific embodiments to which the present invention is applied will be described with reference to the drawings. As shown in FIG. 1, the imaging apparatus 1 includes a lens system 10, an imaging element 11, an image RAM 12, an imaging direction detection unit 13, a panorama synthesis unit 14, a clock unit 15, and a recording medium 16. Although not shown, the imaging device 1 includes a drive source for turning the imaging unit, such as a stepping motor.

  For example, a CCD or a CMOS (Complementary Metal Oxide Semiconductor) sensor is used as the image sensor 11. In the image sensor 11, the subject image irradiated through the lens system 10 is converted into an electrical signal. The image sensor 11 has a predetermined signal processing circuit (not shown), further converts the subject image converted into an electric signal into digital image data, and outputs the digital image data to the image RAM 12.

  Image data output from the image sensor 11 is supplied to the image RAM 12. The image RAM 12 stores a plurality of image data, and supplies the stored image data to the imaging direction detection unit 13 and the panorama synthesis unit 14.

  Image data is supplied from the image RAM 12 to the imaging direction detector 13. The imaging direction detection unit 13 detects, as data for performing panorama synthesis processing, an imaging direction and a movement amount between a plurality of continuous image data, that is, a movement vector between individual image data. Then, the imaging direction detection unit 13 supplies the detected movement vector to the panorama synthesis unit 14. The imaging direction detection unit 13 detects a movement vector by, for example, comparing image data between adjacent fields in time series.

  When detecting the magnitude of the movement amount between the image data, the imaging direction detection unit 13 averages the movement amount in order to detect the imaging direction from the start of shooting to the end of shooting as one shooting direction. Further, when averaging the movement amount between the image data, the imaging direction detection unit 13 may average the movement amount after excluding the movement amount of a predetermined amount or more. The imaging direction detection unit 13 accumulates the movement amount between the image data for each image data. When accumulating the movement amount between the image data, the imaging direction detection unit 13 may exclude the movement amount having a predetermined threshold value or more.

  The imaging direction detection unit 13 detects the imaging direction from the start of panorama shooting to the end of shooting based on the detected movement vector. Then, the imaging direction detection unit 13 encodes the detected imaging direction into several directions. That is, the imaging direction detection unit 13 divides the imaging direction into several directions. For example, the imaging direction detection unit 13 encodes the detected imaging direction into four values, up, down, left, and right. In this manner, the imaging direction detection unit 13 can reduce the amount of data supplied to the recording medium 16 by encoding the detected imaging direction.

  Then, the imaging direction detector 13 supplies the recording direction from the start to the end of the encoded panorama shooting to the recording medium 16 in association with the panoramic image synthesized by the panorama synthesizing unit 14 as one imaging direction. .

  As described above, the imaging direction detection unit 13 can be used for panoramic image synthesis and imaging direction detection. Therefore, in the imaging apparatus 1, it is possible to create an apparatus for recording panoramic image synthesis, an imaging direction associated with the panoramic image, and shooting time information at a low cost.

  Image data is supplied from the image RAM 12 to the panorama composition unit 14. The panorama synthesis unit 14 is supplied with a movement vector between image data as data for performing panorama synthesis processing from the imaging direction detection unit 13. The panorama synthesis unit 14 synthesizes a panoramic image based on the movement vector supplied from the imaging direction detection unit 13 and the plurality of image data supplied from the image RAM 12.

  For example, the panorama composition unit 14 generates a composite image by joining partial images within a read range having a predetermined width from a part of an image obtained on the imaging surface of the image sensor so that the overlapping ranges overlap each other ( For example, see Patent Document 3). The panorama synthesis unit 14 supplies the synthesized panorama image to the recording medium 16.

  Further, the panorama composition unit 14 performs a trimming process on a panoramic image composed of a plurality of pieces of image data. The trimming process is an extraction process for cutting out a partial image area of image data, for example, an area image near the center. The panorama synthesizing unit 14 performs a trimming process on the image data 20, the image data 21, and the image data 22 shown in FIG. 2, for example, thereby extracting a necessary part of the image data and synthesizing a good panoramic image.

  For example, as shown in a region 23 in FIG. 2A, the panorama composition unit 14 performs a trimming process so as to extract a rectangular panoramic image. Further, when the panorama composition unit 14 has an inclination in continuous image data, that is, when the user shoots the image capturing apparatus 1 while tilting the image capturing apparatus 1 without keeping the image capturing apparatus 1 horizontal, FIG. As shown in a region 24), trimming may be performed in an oblique direction.

  The clock unit 15 constitutes time measuring means for detecting shooting time information from the start of shooting to the end of shooting. The clock unit 15 detects, for example, a shooting start time and a shooting end time of panoramic shooting. Then, the clock unit 15 supplies the detected shooting start time and shooting end time to the recording medium 16 in association with the panoramic image synthesized by the panorama synthesizing unit 14. Note that the clock unit 15 may detect, for example, the time for each captured image as the shooting time information, or may detect the length between a certain shooting time and the shooting time.

  The recording medium 16 is a recording medium capable of recording a digital signal. For example, a hard disk, a magneto-optical disk, a DVD, an MD (trademark), a semiconductor memory, or a magnetic tape can be used. In the recording medium 16, the panorama image synthesized by the panorama synthesizing unit 14, the imaging direction from the start to the end of shooting of panorama shooting detected by the imaging direction detection unit 13, the shooting start time detected by the clock unit 15, and A shooting end time is supplied. The recording medium 16 records an imaging direction from the start of shooting to the end of shooting, and a shooting start time and a shooting end time of panorama shooting in association with the panoramic image as one shooting direction. That is, the recording medium 16 records one imaging direction for the synthesized panoramic image as the imaging direction.

  In this way, the recording medium 16 can accurately know the shooting time of panoramic shooting by recording one shooting direction from the start of shooting to the end of shooting together with the shooting start time and shooting end time in association with the panoramic image. it can. That is, in the imaging apparatus 1, it is possible to know a more accurate time by recording the imaging direction of panoramic imaging on the recording medium 16 together with the imaging start time and the imaging end time. Therefore, when the viewer scrolls and reproduces a panoramic image 26 having a time difference in the screen on a TV monitor or the like as shown in FIG. Allows browsing.

  In other words, according to the imaging apparatus 1, by appropriately changing the scroll playback direction and start point based on the panorama shooting start time, shooting end time, and shooting direction, an image that the viewer wants to view can be efficiently viewed in a short time. It becomes possible to browse well. For example, when the user wants to shoot a panoramic image 25 as shown in FIG. 3A at the start of shooting, an unexpected object 27 that he / she did not want to shoot immediately before shooting is shot as shown in FIG. 3B. Suppose it has been done. In this case, according to the imaging apparatus 1, it is possible to prevent the panoramic image 26 from being scroll-reproduced from the unexpected object 27 by appropriately changing the scroll reproduction direction and start point. That is, the imaging apparatus 1 can easily scroll and reproduce the panoramic image 26 along the imaging direction indicated by the arrow in FIG.

  In the recording medium 16, information about the shooting start time, shooting end time, and shooting direction of panoramic shooting is recorded in a user area of EXIF (Exchangeable Image File Format) standard. In the EXIF standard, a user area is provided as incidental information in addition to shooting conditions, and uniquely defined information can be recorded by a user or a device. Note that the imaging apparatus 1 may record information related to the shooting start time, shooting end time, and shooting direction of panoramic shooting in a form in which a new definition is given to an area other than the user area, that is, an undefined area.

  Further, the recording time may be recorded on the recording medium 16 together with the photographing start time detected by the clock unit 15, for example, and the photographing end time may not be recorded. Furthermore, the time for each captured image or the average time between captured images may be recorded on the recording medium 16 together with the number of captured image data used for panorama composition in the panorama composition unit 14. Furthermore, the recording medium 16 may record all shooting times of image data used for synthesizing the panoramic image in the panorama synthesizing unit 14. In addition, the recording medium 16 may record image data captured with the imaging device 1 stationary. The recording medium 16 may detect the coordinate information of the subject by detecting the feature of the subject included in the image data, and record the coordinate information in association with the panoramic image. For example, the recording medium 16 records face coordinates based on the face image of the subject detected by a face detection unit (not shown) in association with the panoramic image, together with information on the shooting start time, shooting end time, and shooting direction of panoramic shooting.

  Next, another embodiment of the imaging apparatus 1 will be described with reference to FIG. The same components as those shown in FIG. 1 are described with the same reference numerals.

  The optical axis variable element 28 is, for example, a shift lens type or a mirror block type, and is controlled by a driver 35. The lens system 10 adjusts optical systems such as iris, focus, and zoom. Adjustments performed in the lens system 10 are controlled by the lens system driving circuit 36, respectively. For example, when shooting a panoramic image, the optical axis changes in a direction to cancel the movement of the imaging device 1 at least while the electronic shutter of the imaging device 11 is open. Further, while the electronic shutter is closed, the optical axis is controlled to return to approximately the center of the movement of the imaging device 1. These controls are automatically performed based on the control of the system controller 42 described in detail later. The adjustment of the optical system can also be performed automatically. In the imaging apparatus 1, control information is output from the lens system driving circuit 36 based on such control, and is supplied to the system controller 42 and the recording medium 30. As the control information, for example, the respective values of iris, focus, and zoom are output from the lens system drive circuit 36.

  The image sensor 11 is irradiated with a subject image via the optical axis variable element 28 and the lens system 10. For example, a CCD or a CMOS sensor is used as the image sensor 11. The image sensor 11 converts the irradiated subject image into an electrical signal. The image sensor 11 has a predetermined signal processing circuit (not shown), further converts the converted electrical signal into digital image data, and outputs the digital image data to the REC terminal of the switch 32 and the compression circuit 29.

  The output from the image sensor 11 is supplied to the compression circuit 29. In addition, a motion detection circuit 38 is coupled to the compression circuit 29. Further, the composite image information is supplied from the panorama composition unit 14 to the compression circuit 29. The compression circuit 29 performs compression processing on the digital image data output from the image sensor 11 and the combined image information output from the panorama combining unit 14 to generate compressed image data, and the generated compressed image data is recorded on the recording medium. 30.

  The motion detection circuit 38 obtains a motion vector by comparing image data between adjacent fields in time series, and performs motion detection of the image data based on the obtained motion vector. The output from the motion detection circuit 38 is supplied to the compression circuit 29 and the mixing circuit 37. In the imaging apparatus 1, since the motion detection circuit 38 can be used for panoramic image synthesis and imaging direction detection, a circuit for detecting the imaging direction, that is, the imaging direction detector 13 is small and efficient. Can be created.

  The mixing circuit 37 mixes the motion information supplied from the integration circuit 40, which will be described in detail later, and the output from the motion detection circuit 38, and supplies the mixing result to the driver 35 and the recording medium 30.

  The recording medium 30 is a recording medium capable of recording a digital signal. For example, a hard disk, a magneto-optical disk, a DVD, an MD (trademark), a semiconductor memory, or a magnetic tape can be used. Compressed image data from the compression circuit 29, control information from the lens system drive circuit 36, and a mixing result from the mixing circuit 37 are supplied to the recording medium 30. The recording medium 30 is supplied with imaging direction information from the imaging direction detection unit 13 and time information from the clock unit 15.

  The recording medium 30 records the control information and the mixing result supplied for each compressed image data as subcode data. The compressed image data and subcode data read by the recording medium 30 are supplied to the decompression circuit 31. The recording medium 30 may include the image RAM 12 and the recording medium 16 shown in FIG.

  The decompression circuit 31 is supplied with the compressed image data and subcode data read by the recording medium 30. In the decompression circuit 31, the compression encoding applied to the compressed image data during recording on the recording medium 30 is released. The decompression circuit 31 supplies the image data that has been subjected to compression encoding to the panorama synthesis unit 14 as reproduced image data.

  The panorama synthesizing unit 14 is supplied with reproduced image data from the decompression circuit 31, lens control information from the recording medium 30 in the lens system driving circuit 36, and subcode data resulting from the mixing in the mixing circuit 37. . The panorama synthesizing unit 14 synthesizes the panorama images and supplies the synthesized panorama images to the PB terminal of the switch 32 and the compression circuit 29.

  A plurality of motion detection mixed result information is supplied from the recording medium 30 to the imaging direction detection unit 13. The imaging direction detection unit 13 detects the imaging direction from the start of panorama shooting to the end of shooting based on the supplied mixed result information of the plurality of motion detections. The imaging direction detection unit 13 supplies the detected information on the imaging direction from the start of shooting to the end of shooting in association with the combined image information combined by the panorama combining unit 14 to the recording medium 16.

  The clock unit 15 detects time information at the start and end of shooting as shooting time information, and supplies the detected time information to the recording medium 16 in association with the panoramic image synthesized by the panorama synthesizing unit 14.

  The switch 32 causes the viewfinder 33 to display an image being photographed by the image sensor 11 by selecting the terminal REC at the time of photographing. In addition, the switch 32 allows an image being captured by the image sensor 11 to be derived to the video output terminal 34 and displayed on an external video monitor or the like. At the time of synthesizing the panorama image and at the time of reproducing the panorama image, the PB terminal is selected by the switch 32, and the reproduced image data is derived to the video output terminal 34 and the reproduced image data is supplied to the viewfinder 33. In the switch 32, the playback image supplied from the recording medium 30 is displayed on the viewfinder 33 by selecting the PB terminal when synthesizing the panorama image and reproducing the panorama image. Note that when the panorama image is reproduced, the processing in the panorama composition unit 14 is not performed.

  The angular acceleration sensor 39 is a gyro sensor, for example, and is a sensor for detecting a change in the imaging direction. As the angular acceleration sensor 39, for example, a sensor for a so-called camera shake prevention function mounted on the imaging apparatus 1 can be used. By using the output of the angular acceleration sensor 39 in this way, the user can know the relative position of the captured image even when the motion detection circuit 38 cannot correctly obtain the movement vector.

  The integration circuit 40 integrates the output from the angular acceleration sensor 39 to obtain motion information. In the integration circuit 40, the average motion is generated by averaging the angular acceleration detection results obtained by the angular acceleration sensor 39. That is, the integration circuit 40 generates motion information while suppressing fine motion. The integration circuit 40 outputs the generated motion information to the mixing circuit 37.

  The key input unit 41 is provided with a switch for switching various settings of the imaging apparatus 1 and a switch for switching on / off of panoramic shooting. Setting information by these various setting keys is supplied from the key input unit 41 to the system controller 42 and stored in, for example, a RAM. For example, the photographer turns on a panorama shooting ON / OFF setting switch provided in the key input unit 41 to start panorama shooting. The photographer shoots the lens system 10 so as to trace the subject, thereby recording the shooting start time and the shooting end time together with the synthesized panoramic image and information regarding the shooting direction from the shooting start to the shooting end. be able to.

  The system controller 42 includes a CPU, a RAM, a ROM, and the like, and controls the operation of the imaging device 1. For example, the system controller 42 controls the operation based on setting information by various setting keys supplied from the key input unit 41. The system controller 42 has a timer for setting a predetermined time (not shown). Further, the system controller 42 may display an instruction on the liquid crystal screen of the viewfinder 33 so that the moving direction between images is constant based on the moving vector between images. For example, a sounding device (not shown) may be provided and this instruction may be notified by sound.

  Next, a processing procedure for synthesizing a panoramic image when the configuration shown in FIG. 1 is used will be described with reference to the flowchart shown in FIG. In the following description, it is assumed that the system controller 42 shown in FIG. 4 controls the operation of each unit in FIG.

  The imaging apparatus 1 performs various imaging parameter processes in step S1 after performing hardware diagnosis and initialization. In the imaging apparatus 1, for example, information regarding brightness identified by an exposure meter (not shown) is acquired, and imaging parameters such as an aperture value and a shutter speed are calculated.

  In step S2, the system controller 42 determines whether or not the shutter button has been finger pressed. In other words, the system controller 42 identifies the timing of starting shooting based on whether or not an operation signal based on the shutter button pressing input operation is generated. Specifically, if the system controller 42 can identify the generation of the operation signal (true), the process proceeds to step S3. If the system controller 42 cannot identify the generation of the operation signal (false), the process of step S1 is executed again.

  In step S <b> 3, the recording medium 16 acquires the shooting start time from the clock unit 15.

  In step S4, the imaging device 1 captures an image a plurality of times. For example, the system controller 42 adjusts the aperture system drive unit of the lens system 10 using the parameters acquired in step S <b> 1 and executes the imaging of the image with the imaging device 11. In the imaging element 11, the captured subject image is converted into an electrical signal, for example, an imaging signal C 1, and this imaging signal C 1 is supplied to the image RAM 12.

  In step S5, the imaging direction detection unit 13 detects a movement vector. Specifically, the imaging direction detection unit 13 calculates the relative displacement between the imaging signal C1 stored in the image RAM 12 and the synthesized image composed of the imaging signal extracted in the previous imaging, and moves between the imaging images. The direction and amount of movement, that is, the movement vector is detected.

  In step S6, the panorama composition unit 14 executes panorama composition processing. Specifically, the panorama synthesizing unit 14 superimposes the imaging signals C1 stored in the image RAM 12 in step S4 based on the movement vector detected in step S5 to synthesize a panoramic image.

  In step S7, the system controller 42 determines whether or not the shutter button is being depressed. Specifically, the system controller 42 identifies the photographing end timing based on the operation signal from the key input unit 41. If the system controller 42 determines that the pressing operation is continued (true), the system controller 42 returns to the process of step S4 to continue shooting and repeats shooting of the subject. On the other hand, when the system controller 42 determines that the pressing operation has been completed (false), the system controller 42 proceeds to the process of step S8 in order to execute the photographing end operation.

  In step S <b> 8, the recording medium 16 acquires the shooting end time from the clock unit 15.

  In step S9, the panorama synthesizing unit 14 performs a trimming process on the panoramic image obtained in step S6.

  In step S10, the imaging direction detection unit 13 detects the imaging direction from the start of shooting to the end of shooting based on the plurality of movement vectors obtained in step S5. That is, the imaging direction detection unit 13 detects one imaging direction from the start of shooting to the end of shooting based on a plurality of movement vectors.

  In step S11, the recording medium 16 captures the panoramic image obtained in step S9, information on the imaging direction from the imaging start to the imaging end obtained in step S10, the imaging start time obtained in step S3, The shooting end time obtained in S8 is recorded in association with each other.

  As described above, according to the imaging apparatus 1 according to the present embodiment, it is possible to record the panoramic composite image having a time difference in the screen in association with the imaging direction and the imaging time from the imaging start to the imaging end. . Therefore, according to the imaging apparatus 1 according to the present embodiment, when scrolling and reproducing a plurality of panoramic images, an image that the viewer wants to browse can be browsed efficiently in a short time.

  In addition, according to the imaging apparatus 1 according to the present embodiment, an image in which an unexpected object or person is recorded when an object or person that was not desired to be captured at the start of capturing is captured immediately before the end of the capturing. It is possible to prevent scroll playback from being performed. That is, the imaging device 1 enables the viewer to browse the image he / she wants to browse efficiently in a short time when the viewer plays, for example, a panoramic image recorded on the recording medium 16 with a TV monitor or the like.

  In addition, according to the imaging apparatus 1 according to the present embodiment, when it is desired to efficiently delete image data in which an unexpected object or person is recorded, scroll reproduction is started from a direction opposite to the imaging direction. Thus, it is possible to efficiently delete image data of an unexpected part.

  In the flowchart shown in FIG. 5, a processing example of combining panoramic images while shooting, that is, so-called real-time combining processing has been described, but the present invention is not limited to this example. For example, the panorama synthesizing unit 14 may synthesize a panoramic image after image data from the photographing start to the photographing end is stored in the recording medium 16.

  Next, an example of a playback control apparatus that performs scroll control when playing back a panoramic image according to the present embodiment will be described. As shown in FIG. 6, the playback control device 50 includes a recording medium 51, a compression / decompression circuit 52, a RAM 53, a clipping circuit 54, a control microcomputer 55, a display driver 56, and a display unit 57.

  The recording medium 51 is a recording medium capable of recording a digital signal. For example, a hard disk, a magneto-optical disk, a DVD, an MD (trademark), a semiconductor memory, a magnetic tape, or the like can be used. In the recording medium 51, for example, shooting information relating to panorama shooting is recorded in association with the panorama image together with a panorama image compressed in a format such as JPEG.

  As shooting information related to panorama shooting, for example, an imaging direction from shooting start to shooting end of panorama shooting, subject coordinate information detected by feature detection, shooting time information of panorama shooting, and the like are recorded. As the imaging direction from the start of panorama shooting to the end of shooting, for example, quaternized information indicating the vertical and horizontal directions is recorded in the recording medium 51. As the coordinate information of the subject detected by the feature detection, for example, the face coordinates based on the face image of the subject detected by the face detection, specifically, the positions of both eyes of the subject, the position of the nose, the position of the mouth, and both eyes Information on the positions of a plurality of organs such as the nose is recorded on the recording medium 51. The subject is not limited to a person as long as the subject has information about facial features such as eyes, nose, and mouth, and may be an animal such as a dog or a cat. As the shooting time information, for example, shooting start time and shooting end time of panoramic shooting are recorded in the recording medium 51. These pieces of shooting information are recorded in the EXIF standard user area on the recording medium 51. The recording medium 51 supplies the panoramic image to the compression / decompression circuit 52, and outputs shooting information associated with the panoramic image to the control microcomputer 55.

  The compression / decompression circuit 52 is supplied from the recording medium 51 with a panoramic composite image compressed in a format such as JPEG. The compression / decompression circuit 52 performs a compression / decompression process on the panorama composite image supplied from the recording medium 51, and supplies the panorama image data decompressed by the compression / decompression process to the RAM 53.

  The RAM 53 is supplied with panoramic image data after compression / decompression processing from the compression / decompression circuit 52. The RAM 53 supplies a part of the panoramic image data supplied from the compression / decompression circuit 52 to the clipping circuit 54 based on an instruction from the control microcomputer 55.

  The cut-out circuit 54 performs processing for enlarging or reducing the panoramic image data supplied from the RAM 53 to a desired size to obtain image data to be displayed on the display unit 57. The clipping circuit 54 supplies the generated image data to the display driver 56.

  The control microcomputer 55 is connected to the recording medium 51, the compression / decompression circuit 52, the RAM 53, the cutout circuit 54, the display driver 56, and the display unit 57, and controls these operations. The control microcomputer 55 is supplied with photographing information associated with the panoramic image data recorded on the recording medium 51. Then, the control microcomputer 55 supplies the cutout area of the panoramic image data and information regarding the magnification to be enlarged or reduced to the cutout circuit 54.

  For example, the control microcomputer 55 supplies an instruction value for scroll display on the display unit 57 to the cutout circuit 54 while shifting the cutout coordinates left and right or up and down as information about the cutout area. In this way, as will be described in detail later, the control microcomputer 55 determines the scroll direction for reproducing the panoramic image based on the imaging direction from the start of shooting to the end of shooting associated with the panoramic image and the coordinate information of the subject. decide. In addition, the control microcomputer 55 determines a scroll start point when scrolling and reproducing the panoramic image based on the imaging direction associated with the panoramic image and the coordinate information of the subject. Further, the control microcomputer 55 determines the scroll speed when reproducing the panoramic image based on the coordinate information of the subject.

  That is, the playback control device 50 determines a scroll direction, a scroll start point, and a scroll speed when playing back a panoramic image based on shooting information associated with the panoramic image. Therefore, the playback control device 50 enables the viewer to efficiently view the panoramic image when scrolling and playing back a plurality of panoramic images continuously.

  When the coordinate information of the subject is not recorded on the recording medium 51 in association with the panoramic image, the control microcomputer 55 performs processing for detecting the coordinate information. For example, the control microcomputer 55 detects the position of both eyes as a reference position in the face image from the panoramic image, performs normalization based on the detected position of both eyes, and extracts the luminance as the feature amount of the normalized face image. . Then, the control microcomputer 55 performs various determination processes based on the luminance value in each part of the extracted face image and the determination information recorded in advance. For example, the control microcomputer 55 performs an adult / child determination process, a smile / non-smile determination process, a determination process regarding whether the face is a registered / non-registered face, and the like. In the above-described example, the position of both eyes is used as the reference position, but other organs other than both eyes included in the face may be used. For example, the position of the nose included in the face and the positions of a plurality of organs such as both eyes and nose can be used.

  A part of the panoramic image data to be displayed on the display unit 57 is supplied from the clipping circuit 54 to the display driver 56. The display driver 56 supplies the image data supplied from the clipping circuit 54 to the display unit 57.

  The display unit 57 is configured by an LCD, for example, and displays the image data supplied from the display driver 56.

  Next, an example of the reproduction control method by the reproduction control device 50 shown in FIG. 6 will be described with reference to the flowchart shown in FIG.

  In the reproduction control device 50, after performing hardware diagnosis and initialization, in step S20, the control microcomputer 55 selects a panoramic image to be reproduced.

  In step S <b> 21, the control microcomputer 55 determines whether or not a playback button in an operation unit (not shown) has been pressed. In other words, the control microcomputer 55 identifies the playback start timing based on whether or not an operation signal based on the pressing input operation of the playback button has been generated. If the generation of the operation signal can be identified (step S21: true), the control microcomputer 55 proceeds to the process of step S22. If the generation of the operation signal cannot be identified (step S21: false), the control microcomputer 55 executes the process of step S20 again.

  In step S22, the control microcomputer 55 executes a reading operation of the selected panoramic image. Specifically, the control microcomputer 55 decompresses the panorama image selected in step S <b> 20 from the compressed panorama image recorded on the recording medium 51 by the compression / decompression circuit 52. Then, the control microcomputer 55 stores the panoramic image decompressed by the compression / decompression circuit 52 in the RAM 53.

  In step S23, the control microcomputer 55 selects a scroll direction when reproducing the panoramic image. Specifically, the control microcomputer 55 determines the scroll direction of the panoramic image based on the information related to the imaging direction associated with the panoramic image that has been read into the RAM 53 in step S22 and the coordinate information of the subject.

  As an example of a method for determining the scroll direction, a case where a panoramic image is reproduced in the longitudinal direction as shown in FIGS. 8 to 10 will be described.

  First, if the imaging direction from the start of panorama shooting to the end of shooting is associated with the panoramic image, the same direction as the shooting direction is set as the scroll direction for reproduction. For example, in the panoramic image 60 shown in FIG. 8, the control microcomputer 55 associates information indicating that the shooting start is on the left side and the shooting end is on the right side, as indicated by an arrow in the shooting direction 61. Therefore, the same direction as the shooting direction 61 is determined as the playback direction 62. That is, the control microcomputer 55 determines the same direction as the shooting direction 61 as the scroll direction.

  Further, when face coordinates are associated with the panoramic image as the coordinate information of the subject, or when face coordinates are detected in advance, the face coordinates are shifted from the end closer to the left and right ends to the end farther away. The direction is the scroll direction. That is, the control microcomputer 55 scrolls and reproduces the panoramic image from one end to the other end in the longitudinal direction of the panoramic image close to the face coordinate position of the subject. For example, in the panoramic image 60 shown in FIG. 9, the control microcomputer 55 determines the direction from the right end to the left end close to the position of the face coordinates 64 as the reproduction direction 65.

  Furthermore, when the face coordinates and the imaging direction from the start of shooting to the end of shooting are associated with the panoramic image, the scroll reproduction direction is determined based on the face coordinates. That is, the control microcomputer 55 prioritizes the information about the face coordinates over the information about the imaging direction, and determines the scroll reproduction direction. For example, in the panoramic image 60 shown in FIG. 10, the control microcomputer 55 associates the shooting direction 67 and the face coordinates 68 with each other. The playback direction is 69. That is, the control microcomputer 55 determines the scroll direction by giving priority to the face coordinates 68 instead of the shooting direction 67.

  Furthermore, when a plurality of face coordinates are associated with the panoramic image and weight information related to the face coordinates is recorded, the control microcomputer 55 is based on the face coordinate having the highest weight among the plurality of face coordinates. Determine the scroll direction. For example, the control microcomputer 55 sets the direction from one end to the other end in the longitudinal direction of the panoramic image close to the center position of the face coordinate having the highest weight, that is, the priority face coordinate as the playback direction. The priority criteria for face coordinates are, for example, a child for adults and children, a smile for smiles and non-smiles, and a registration for unregistered faces and registered faces. You can prioritize the face you are in. Note that, for example, the control microcomputer 55 may set the direction from one end to the other end of a near panoramic image far from the center of a plurality of face coordinates as the scroll direction.

  The control microcomputer 55 may determine the scroll direction based on the imaging direction and the imaging time information when the imaging time information of the panoramic imaging is associated with the panoramic image 60. For example, the control microcomputer 55 determines the direction from the image with the earlier photographing time to the image with the later photographing time as the scroll direction.

  The control microcomputer 55 may determine, for example, a direction opposite to the shooting direction 61 in FIG. 8, the playback direction 65 in FIG. 9, and the playback direction 69 in FIG. The direction of the image from the image with the latest shooting time to the image with the earlier image may be determined as the scroll direction.

  In step S24, the control microcomputer 55 calculates an initial display area for extracting an area to be displayed at the initial stage of reproduction from the entire panoramic image. For example, as shown in FIG. 11, the control microcomputer 55 determines the scroll start point based on the face coordinates 72 associated with the panorama image 60 that has been read in step S22 and the face coordinates obtained by face detection during panorama shooting. That is, the display area 71 is determined. The image in the display area 71 is displayed on the display unit 57 as the display image 73.

  Specifically, since the panoramic image 60 shown in FIG. 8 is associated with a shooting direction 61 that is shot from the left to the right, the control microcomputer 55 sets the start point of the scroll reproduction as the leftmost image. . That is, the control microcomputer 55 determines the reproduction start image 63 as an initial display area for scroll reproduction. Note that the control microcomputer 55 may use the area at the end of the shooting direction 61 as a playback start image according to the setting.

  In addition, when the face coordinates are associated with the panoramic image or when the face coordinates are detected in advance, the control microcomputer 55 uses one end in the longitudinal direction of the panoramic image close to the position of the face coordinates as the scroll start point. Also good. For example, in the panoramic image shown in FIG. 9, the control microcomputer 55 determines the right end as the reproduction start image 66 because the position of the face coordinate 64 is close to the right end in the longitudinal direction. Note that the control microcomputer 55 may determine one end of the face coordinate 64 far from the left and right ends as the reproduction start image 66 according to the setting.

  Furthermore, the control microcomputer 55 determines the scroll start point based on the face coordinates when the information on the face coordinates and the imaging direction is associated with the panoramic image. That is, the control microcomputer 55 determines the reproduction start image by giving priority to the face coordinate information over the imaging direction. For example, in the panoramic image 60 shown in FIG. 10, since the photographing direction 67 and the face coordinates 68 exist, one end in the longitudinal direction of the panoramic image 60 close to the position of the face coordinates 68 is set as the reproduction start image 70. Note that the control microcomputer 55 may set one end far from the position of the face coordinates 68 as a reproduction start image according to the setting.

  Furthermore, when a plurality of face coordinates are associated with the panoramic image and weight information related to the face coordinates is recorded, the control microcomputer 55 is based on the face coordinate having the highest weight among the plurality of face coordinates. Determine the scroll start point. For example, the control microcomputer 55 sets one end in the longitudinal direction of the panoramic image close to the face coordinate with the highest weight, that is, the face coordinate to be prioritized, as the scroll start point. For example, as described above, the priority criterion for the face coordinates is a child for an adult and a child, a smile for a smile and a non-smile, and a face that is not registered in advance and a registered face. Gives priority to registered faces. Note that the control microcomputer 55 may use, for example, one end in the longitudinal direction of a panoramic image far from the center of a plurality of face coordinates as a scroll start point.

  In step S25, the control microcomputer 55 identifies the playback stop timing of scroll playback by determining whether or not the playback stop button has been pressed. That is, if the control microcomputer 55 can identify the generation of the operation signal (step S25: true), the control microcomputer 55 stops the panorama reproduction and ends the process. If the generation of the operation signal cannot be identified (step S25: false), the control microcomputer 55 proceeds to the process of step S7.

  In step S <b> 26, the control microcomputer 55 determines whether or not to skip to the next panoramic image display by determining whether or not a button for skipping to the next image on the operation unit has been pressed. That is, when the control microcomputer 55 can identify the generation of the operation signal (step S26: true), the control microcomputer 55 proceeds to the operation of step S27. The control microcomputer 55 stops the reproduction of the currently displayed panoramic image and shifts to the next image reading operation. If the control microcomputer 55 cannot identify the generation of the operation signal (step S26: false), the control microcomputer 55 proceeds to the operation of step S28. That is, the control microcomputer 55 shifts to display of the currently selected panoramic image.

  In step S27, the control microcomputer 55 performs a reading operation of the panorama image next to the currently selected panorama image. The next panorama image means a panorama image having a file number one larger than the file number of the currently selected panorama image when there is a file number associated with the panorama image. If there is no panoramic image with a file number one larger than the file number, it means a panoramic image with two larger file numbers. If there is no panorama image with one file number and no panorama image with two file numbers, the same selection is performed for three or more numbers. When performing such a selection operation, when the maximum number of file numbers that can be selected is reached, the panorama image with the smallest file number is selected as a candidate for the next image. In step S27, the control microcomputer 55 uses the compression / decompression circuit 52 to decompress the next selected panorama image from the compressed panorama image stored in the recording medium 51, as in step S22. Further, the control microcomputer 55 stores the panoramic image decompressed by the compression / decompression circuit 52 in the RAM 53.

  In step S28, a panoramic image cut out image is displayed. The control microcomputer 55 causes the cutout circuit 54 to read a part of the panoramic image in the cutout area from the RAM 53. Then, the cutout circuit 54 enlarges or reduces the panoramic image to an optimum size on the display unit 57. The optimum size in the display unit 57 means a standard format size such as 1920 × 1080 when performing high-definition playback. The panoramic image that has been standardized by the clipping circuit 54 is D / A converted by the display driver 56 and a part of the panoramic image is displayed on the display unit 57. Note that in step S28, the panorama cutout area calculated in step S24 is based on the first display, but the area calculated in step S29 is based on the second and subsequent displays.

  In step S29, the cropped image is shifted. In other words, the control microcomputer 55 controls the shift amount and timing of the cutout image to thereby cut out the cutout image in any one of the upper, lower, left, and right directions selected in step S23 with respect to the currently selected cutout area. Shift.

  The control microcomputer 55 reduces the shift amount of the display area more than usual when face coordinates are included in the selected coordinates. For example, as shown in FIG. 12, when the face coordinate 74 is included in the selected coordinates, the control microcomputer 55 slowly delays the shift timing while the face coordinate 74 is included in the display area 75. Scroll display may be performed. As described above, the control microcomputer 55 enables efficient browsing by slowing down the scroll speed when the image that the viewer wants to browse is displayed.

  Further, the control microcomputer 55 can apply various methods as a method for controlling the scroll speed. For example, the control microcomputer 55 gradually decreases the scroll speed to the center of the screen while the face coordinates are displayed on the display unit 57, and increases the scroll speed step by step when passing the center of the screen. Also good. Further, while the face coordinates are displayed on the display unit 57, the control microcomputer 55 gradually decreases the scroll speed to the center of the screen and enlarges and displays the face coordinate portion in stages. When the position is passed, the scroll speed may be increased stepwise and the face coordinate portion may be reduced and displayed stepwise. In addition, while the face coordinates are displayed on the display unit 57, the control microcomputer 55 increases the scroll speed to the center of the screen stepwise and enlarges and displays the face coordinate portion stepwise. , The scroll speed may be reduced stepwise and the face coordinate portion may be reduced and displayed step by step.

  Subsequently, in step S30, the end of the panoramic image is detected. Specifically, the control microcomputer 55 detects whether or not the display area has reached the top, bottom, left and right edges of the panoramic image after shifting the cutout coordinates in step S29. For example, when the display area 76 reaches the end in the panoramic image 60 shown in FIG. 13 (step S30: true), the control microcomputer 55 proceeds to step S27 and proceeds to the next panoramic image reading operation. To do. Further, when the display area has not reached the end (step S30: false), the control microcomputer 55 proceeds to the process of step S25, that is, the operation of detecting the pressing of the reproduction stop button.

  Thus, according to the playback control device 50, when a plurality of panoramic images are continuously scrolled and played back, it is possible to efficiently browse images that the user wants to browse in a short time.

  Specifically, when the user wants to shoot a panoramic image 25 as shown in FIG. 3A at the start of shooting, an unexpected object that he / she did not want to shoot immediately before the end of shooting as shown in FIG. Suppose that 27 has been shot. In this case, according to the playback control device 50, it is possible to prevent the panoramic image 26 from being scroll-played from the unexpected object 27 by appropriately changing the scroll playback direction and the scroll start point.

  Further, according to the playback control device 50, when it is desired to efficiently delete an image in which an unexpected object or person has been recorded, the scroll playback is started from the direction opposite to the imaging direction, thereby efficiently failing the image. Can be deleted. As described above, when deleting the failed image, the entire panoramic image may be deleted, or the image of the part designated by the user may be deleted.

  Further, according to the playback control device 50, it is possible to browse with less stress for the user by slowing down the scroll speed when playing back an image that the user wants to watch more.

  Further, the series of processes described in the present embodiment can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM as a recording medium. Alternatively, the program can be stored (recorded) temporarily or permanently in a removable recording medium such as a flexible disk, CD-ROM, MO disk, DVD, magnetic disk, or semiconductor memory. Such a removable recording medium can be provided as so-called package software.

  In addition to installing the program on the computer from the above-described removable recording medium, the program is wirelessly transferred from the download site to the computer, or wired to the computer via a network such as a LAN or the Internet. Thus, the transferred program can be received and installed in a recording medium such as a built-in hard disk. In addition, the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

DESCRIPTION OF SYMBOLS 10 Lens system, 11 Imaging device, 12 Image RAM, 13 Imaging direction detection part, 14 Panorama synthesis part, 15 Clock part, 16 Recording medium, 28 Optical axis variable element, 29 Compression circuit, 30 Recording medium, 31 Expansion circuit, 32 switch, 33 viewfinder, 34 video output terminal, 35 driver, 36 lens system drive circuit, 37 mixing circuit, 38 motion detection circuit, 39 angular acceleration sensor, 40 integration circuit, 41 key input unit, 42 system controller, 50 playback Control device, 51 recording medium, 52 compression / decompression circuit, 53 RAM, 54 clipping circuit, 55 control microcomputer, 56 display driver, 57 display unit

Claims (5)

In association with a panoramic image synthesized from a plurality of image data shot a plurality of times from the start of shooting to the end of shooting converted based on the image incident through the lens system, from the start of shooting the panoramic image to the end of shooting. A recording unit for recording the coordinate information of the subject detected by the feature detection in the imaging direction or the panoramic image;
A scroll direction determining unit that determines a scroll direction when reproducing the panoramic image based on the imaging direction or the coordinate information;
The recording unit records face coordinates based on the face image of the subject detected from the image data as the coordinate information in association with the panoramic image ,
The playback control device, wherein the scroll direction determination unit determines the scroll direction based on the face coordinates when the imaging direction and the face coordinates are associated with the panoramic image . The scroll direction determination unit, when a plurality of the face coordinates are associated with the panoramic image and weight information related to the face coordinates is recorded, a coordinate having the highest weight among the plurality of face coordinates. based on the reproduction control apparatus according to claim 1, wherein determining the scrolling direction. The scroll direction determination unit determines, as the scroll direction, a direction from one end to the other end in the longitudinal direction of the panoramic image that is close to the center position of the coordinate with the highest weight among the plurality of face coordinates in the panoramic image. The reproduction control apparatus according to claim 2 . In association with a panoramic image synthesized from a plurality of image data shot a plurality of times from the start of shooting to the end of shooting converted based on the image incident through the lens system, from the start of shooting the panoramic image to the end of shooting. A recording step of recording the coordinate information of the subject detected by the feature detection in the imaging direction or the panoramic image in a recording unit;
A scroll direction determining step for determining a scroll direction when reproducing the panoramic image based on the imaging direction or the coordinate information;
In the recording step, face coordinates based on the face image of the subject detected from the image data are recorded as the coordinate information in association with the panoramic image ,
A reproduction control method for determining the scroll direction based on the face coordinates when the imaging direction and the face coordinates are associated with the panoramic image in the scroll direction determination step . In association with a panoramic image synthesized from a plurality of image data shot a plurality of times from the start of shooting to the end of shooting converted based on the image incident through the lens system, from the start of shooting the panoramic image to the end of shooting. A recording step of recording the coordinate information of the subject detected by the feature detection in the imaging direction or the panoramic image in a recording unit;
A scroll direction determining step for determining a scroll direction when reproducing a panoramic image based on the imaging direction or the coordinate information,
In the recording step, face coordinates based on the face image of the subject detected from the image data are recorded as the coordinate information in association with the panoramic image ,
A program for causing a computer to execute a process of determining the scroll direction based on the face coordinates when the imaging direction and the face coordinates are associated with the panoramic image in the scroll direction determination step .

Images