JPH10276388A - Device, method for processing and reproducing image and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily edit a moving image in a short time by detecting one set of scenes from the inputted moving image, generating a clip picture as the representative picture of scenes and generating a virtual album to which that clip picture is stuck. SOLUTION: An image compressing part 25 or a voice compressing part 26 at a compressing part 13 respectively performs MPEG coding to images or voices from a selector part 23 or 24, outputs them to a bus interface 27 and records them on a hard disk. When recording onto the hard disk is started, a microprocessor reads these data and performs scene filing processing for constituting a scene file. Further, the microprocessor constitutes the virtual album, displays it on a display and completes image and voice fetching processing.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing method, an image reproducing apparatus and an image reproducing method, and a recording medium. Image processing apparatus and image processing method capable of dynamic editing
The present invention relates to an image reproducing device, an image reproducing method, and a recording medium.

[0002]

2. Description of the Related Art For example, a method of editing a moving image captured by a video camera or the like is roughly classified into linear editing and non-linear editing.

FIG. 26 shows an example of the configuration of a conventional linear editing apparatus.

In the video camera 201, a moving image is photographed, and the photographed moving image is played back.
2 is supplied. The image input to the editing machine 202 is a TV
(TeleVision) Displayed on the monitor 203, the user operates the editing machine 202 while viewing the display to edit the image from the video camera 201. The edited image is supplied to a recording device such as a VTR (Video Tape Recorder) 204 and recorded.

In the above-described linear editing apparatus, images reproduced from the video camera 201 are sequentially recorded by the VTR 204 via the editing machine 202.
In the middle of the image (edited image) recorded in 204,
When trying to insert a new image, it is necessary to redo the editing from the inserted portion, which is troublesome and takes time.

Therefore, recently, non-linear editing apparatuses constructed based on a computer or the like are rapidly spreading.

FIG. 27 shows an example of the configuration of a non-linear editing apparatus.

[0008] In the nonlinear editing apparatus, first, an image to be edited is taken into the editing machine 222. That is, a moving image captured by the video camera 211 is reproduced and supplied to, for example, an editing machine 222 based on a computer. In the editing device 222, images from the video camera 222 are recorded in an auxiliary storage device such as a hard disk (not shown) that can be accessed randomly. Therefore, taking the image into the editing machine 222 basically takes a time equal to the time required to reproduce the image with the video camera 211.

When an image is a digital signal, an analog image signal (for example, when the image is a signal of the NTSC system or the PAL system) is transmitted via a dedicated interface. It is converted into a digital signal via a / D converter (not shown) and recorded on a hard disk.

Here, when an image is recorded on a hard disk, the image is stored in a motion JPE, for example.
G (Joint Photographic coding Experts Group) or M
It can be compressed in accordance with standards such as PEG (Moving Picture Experts Group). In this case, the amount of image data to be recorded on the hard disk can be drastically reduced, and the size and cost of the nonlinear editing device can be reduced.

When an image is recorded without compression, it is reduced to about 20 Mbytes / second, and when the image is compressed and recorded, it is reduced to about several hundred kbytes / second to several Mbytes / second. A hard disk with a capacity multiplied by the image playback time is required. Therefore, in the case of recording an image without compression, the size and cost of the apparatus are increased, but high image quality can be maintained. On the other hand, when an image is compressed and recorded, as described above, the apparatus can be reduced in size and cost, but the image quality is slightly degraded.

When the capture of the image is completed, the image is edited. That is, the image to be edited is displayed on a predetermined area of the computer display 223 or on the TV monitor 227, and the user operates the keyboard 224 or the mouse 225 of the editing machine 222 while viewing the display to edit the captured image. . As a result, the editing machine 222 performs, for example, cutting of unnecessary parts, insertion of images and characters, replacement of image scenes, and the like, based on the time stamp attached to the image, and special effects. Is appended. Then, the edited image is recorded on the hard disk.

The edited image recorded on the hard disk is supplied from the editor 222 to the VTR 226 and recorded, or used as a material for another editing.

When the edited image is output as a digital signal, the image is output through a dedicated interface. When the edited image is output as an analog signal, the image is output as a D signal. The signals are output via an A / A converter (not shown). Furthermore, if the edited image is compressed, the compressed image is output after being decompressed.

As described above, in the nonlinear editing device,
Since the editing is performed after all the images to be edited are imported to the editing device 222, it is possible to easily insert a new image or the like into the already edited portion.

[0016]

Incidentally, in the case of non-linear editing, as in the case of linear editing, obtaining an edited image is the purpose of the work, and it is not meaningful just to import the image.

Whether the editing operation is linear editing or non-linear editing, the user looks at the images, checks them, decides an image to be used for editing, and operates the editing machine 202 or 222. Therefore, there is a problem that takes time.

Further, the operations of the editing machines 202 and 222 are as follows.
Since it requires some skill, it is difficult for a novice user to edit.

For example, when an image is cut and edited for each set of scenes, conventionally,
For example, there is an apparatus disclosed in Japanese Patent Application Laid-Open No. H8-9328, in which an edit point is manually input and the scene is cut at the edit point. Was troublesome.

The present invention has been made in view of such a situation, and it is an object of the present invention to edit a moving image easily and in a short time.

[0021]

An image processing apparatus according to a first aspect of the present invention includes a scene detecting means for detecting a set of scenes from an input moving image, and a clip screen which is a still image representing the scene. It is characterized by comprising a clip screen generating means for generating, and an album generating means for generating a virtual album on which the clip screen is pasted.

According to the image processing method of the twenty-sixth aspect, a set of scenes is detected from an input moving image, a clip screen as a still image representing the scene is generated, and the clip screen is pasted. A characteristic album is generated.

The image reproducing apparatus according to the twenty-seventh aspect detects a set of scenes from an input moving image, generates a clip screen as a still image representing the scene, and pastes the clip screen. And a reproduction means for reproducing an album and a scene file which is a file of image data of a scene obtained by generating a typical album.

According to an image reproducing method of the present invention, a set of scenes is detected from an input moving image, a clip screen as a still image representing the scene is generated, and the clip screen is pasted. A scene file and an album, which are files of image data on scenes obtained by generating a typical album, are reproduced.

According to the recording medium of the present invention, a set of scenes is detected from an input moving image, a clip screen as a still image representing the scene is generated, and the clip screen is pasted. A scene file and an album, which are files of image data on scenes obtained by generating a new album, are recorded.

In the image processing apparatus according to the first aspect, the scene detecting means detects a group of scenes from the input moving image, and the clip screen generating means determines whether the clip is a still image representing the scene. The screen has been adapted to generate. The album generating means generates a virtual album to which the clip screen is pasted.

In the image processing method according to the twenty-sixth aspect, a group of scenes is detected from an input moving image, a clip screen as a still image representing the scene is generated, and the clip screen is pasted. A virtual album is generated.

In the image reproducing apparatus according to the twenty-seventh aspect, the reproducing means detects a group of scenes from the input moving image, generates a clip screen as a still image representing the scene, and generates a clip screen. A scene file and an album, which are files of image data of a scene obtained by generating a virtual album with a tag attached thereto, are reproduced.

According to the image reproducing method of the present invention, a set of scenes is detected from an input moving image, a clip screen as a still image representing the scene is generated, and the clip screen is pasted. A scene file, which is a file of image data of a scene obtained by generating a virtual album, and an album are reproduced.

In the recording medium according to the thirty-first aspect, a set of scenes is detected from an input moving image, a clip screen as a still image representing the scene is generated, and the clip screen is pasted. A scene file, which is a file of image data of a scene obtained by generating a typical album, and an album are recorded.

[0031]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below, but before that, the correspondence between each means of the invention described in the claims and the following embodiments will be clarified. For this reason, the features of the present invention are described as follows by adding the corresponding embodiment (however, an example) in parentheses after each means.

That is, the image processing apparatus according to claim 1 is
An image processing apparatus for processing a moving image, which is a scene detecting means (for example, processing step S1 of a program shown in FIG. 4) for detecting a group of scenes from an input moving image
9 and S20, and processing steps S31 to S38 of the program shown in FIG. 6) and a clip screen generating means for generating a clip screen which is a still image representing the scene (for example, processing step S43 of the program shown in FIG. 7)
) And an album generating means (for example, processing step S45 of the program shown in FIG. 7) for generating a virtual album with a clip screen pasted thereon.

An image processing apparatus according to a fourth aspect of the present invention provides an invalid frame detecting means (for example, processing steps S22 and S24 of the program shown in FIG. 5) for detecting an invalid frame which is not required by the user from a moving image. , S26
And the like, and invalid frame deleting means (for example, processing steps S23, S25, and S27 of the program shown in FIG. 5) for deleting invalid frames from the scene.

According to the ninth aspect of the present invention, the image processing apparatus converts the image data of the scene into an MPEG (Moving Picture Exposure).
(erts Group) encoding means (for example, the compression unit 13 shown in FIG. 1), and the clip screen generating means sets a first I picture in a scene as a clip screen. .

The image processing apparatus according to the tenth aspect is a recording means (for example, an auxiliary storage interface 10 or a VTR 16 shown in FIG. 1) for recording a scene file and an album, which are files of image data of a scene.
Is further provided.

[0036] The image processing apparatus according to the eleventh aspect of the present invention further comprises a reproducing means (for example, processing steps S131 to S137 of a program shown in FIG. 24) for reproducing the scene file and the album recorded by the recording means. Features.

The image processing apparatus according to the fourteenth aspect further comprises designation means (for example, a mouse 9 shown in FIG. 1) for designating the clip screen pasted on the album, and the reproduction means designates the clip screen by the designation means. A scene file corresponding to the selected clip screen.

The image processing apparatus according to the fifteenth aspect further includes an encoding unit (for example, the compression unit 13 shown in FIG. 1) for encoding the image data, and the recording unit is encoded by the encoding unit. Characterized in that the recorded image data is recorded.

The image processing apparatus according to a sixteenth aspect further comprises a decoding unit (for example, the decompression unit 15 shown in FIG. 1) for decoding the image data encoded by the encoding unit. I do.

An image processing apparatus according to a seventeenth aspect provides an album editing means for editing an album (for example, processing steps S51 and S52 of a program shown in FIG.
1 and the processing steps S121 to S1 of the program shown in FIG.
29).

An image processing apparatus according to a twentieth aspect is characterized by further comprising a scene editing means (for example, processing steps S71 to S84 of a program shown in FIG. 13) for editing a scene.

According to a twenty-fifth aspect of the present invention, there is provided an image processing apparatus for photoelectrically converting incident light and outputting a moving image signal (for example, a CCD (Charge Coupled Device) shown in FIG.
Device) 102).

An image reproducing apparatus according to a twenty-seventh aspect is an image reproducing apparatus for reproducing a moving image, comprising detecting a group of scenes from an input moving image and displaying a clip as a still image representing the scene. A playback unit that plays back a scene file and an album that is a file of image data on a scene obtained by generating a screen and generating a virtual album to which a clip screen is pasted (for example,
Processing steps S131 to S131 of the program shown in FIG.
137 etc.).

An image reproducing apparatus according to a twenty-ninth aspect further comprises a designating means (for example, a mouse 9 shown in FIG. 1) for designating a clip screen pasted on an album, wherein the reproducing means designates the clip screen by the designating means. A scene file corresponding to the selected clip screen.

Of course, this description does not mean that each means is limited to those described above.

FIG. 1 shows a configuration example of an embodiment of a non-linear editing apparatus to which the present invention is applied.

This non-linear editing apparatus is configured based on, for example, a computer. That is, the microprocessor 1 executes an application program also recorded on the hard disk 12 under the control of the operating system recorded on the hard disk 12, thereby performing image and audio editing processing and reproduction of the edited image and audio. A predetermined process such as a process is performed. The main memory 2 stores programs executed by the microprocessor 1,
Data necessary for the operation of the microprocessor 1 is stored. The frame buffer 3 is, for example, a DRAM (Dynami
c Random Access Memory) and stores an image generated by the microprocessor 1 and the like. The bus bridge 4 includes an internal bus and, for example, a PCI (Peripheral Compo
nent Interconnect) Controls the exchange of data with an extended bus such as a local bus.

The above-described microprocessor 1, main memory 2, frame buffer 3, and bus bridge 4
The blocks are connected to each other via an internal bus, and the remaining blocks are connected to each other via an expansion bus. The bus bridge 4 is connected to both the internal bus and the expansion bus.

The tuner 5 receives a television broadcast signal broadcast using, for example, a terrestrial wave, a satellite line, or a CATV network. Here, images and sounds received by the tuner 5 can also be edited.
The modem 6 controls communication via a telephone line. Here, in the modem 6, for example, images and sounds received from the Internet or the like can be edited, and the edited images and sounds can be transmitted to the outside.

The I / O (Input / Output) interface 7 outputs an operation signal corresponding to the operation of the keyboard 8 and the mouse 9. The keyboard 8 is used to input predetermined data or commands, and the mouse 9 is used to move a cursor displayed on a display (computer display) 17 or to indicate a position.
Each is operated.

The auxiliary storage interface 10 is a CD-ROM.
It controls reading / writing of data from / to an R (Compact Disc Recodable) 11 and a hard disk (HD (Hard Disk)) 12. On the CD-R 11, for example, images and sounds after editing are recorded. The hard disk 12 stores an operating system, application programs for causing the microprocessor 1 to execute non-linear editing and other processes, and the like. Further, the hard disk 12 has images and sounds to be edited,
The edited image and sound are also recorded.

The compression section 13 compresses and encodes the image and sound input thereto, for example, in accordance with the MPEG standard. In the compression unit 13, data supplied via an expansion bus, data supplied via an expansion unit 15, and an external device such as a video camera 14
It is also possible to compress data supplied from the Internet.

The video camera 14 records, for example, images and sounds to be edited. The compression unit 13 has an interface with the video camera 14, so that images and sounds recorded by the video camera 14 can be input to the compression unit 13.

The expansion unit 15 decodes (expands) the data encoded (compressed) by the compression unit 13 and outputs the data. Note that the decompression unit 15 is provided with the frame buffer 3 as necessary.
And overlays the decoded image on the image stored in. Here, the extension unit 15
In the embodiment of FIG. 1, the supply of image data to the image data is performed directly between them.
This can be performed via the internal bus, the bus bridge 4 and the extension bus. However, when the image data is supplied from the frame buffer 3 to the decompression unit 15 via the internal bus, the bus bridge 4, and the extension bus, if the capacity of the internal bus or the extension bus is low, the data is congested. There is a risk.

The VTR 16 records images and sounds output from the decompression unit 15 as necessary. Display 17
Displays the image output by the expansion unit 15 as necessary. The display of the image output by the decompression unit 15 includes a display 17 which is a display for a computer,
It can also be performed by a TV monitor or the like.

FIG. 2 shows an example of the configuration of the compression section 13 of FIG.

An analog image signal or an audio signal is input to the A / D converter 21 or 22, respectively. The analog image signal or the audio signal is converted into an analog signal or an analog signal, respectively. A digital signal is supplied to the selection unit 23 or 24.

The output of the A / D converter 21, the digital image signal, and the output of the image / sound insertion unit 28 are supplied to the selection unit 23, where a predetermined one of them is supplied. The image is selected and supplied to the image compression unit 25. The output of the A / D converter 22, the digital audio signal, and the output of the image / audio insertion unit 28 are supplied to the selection unit 24. Is selected and supplied to the audio compression unit 26.

In the selection section 23, the output of the A / D converter 21, the digital image signal, or the image /
When the output of the voice inserting unit 28 is selected, the selecting unit 2
In 4, the output of the A / D converter 22, the digital audio signal, or the output of the image / audio insertion unit 28 is selected. Selector 2
When the output of the A / D converter 21 is selected in 3, the selector 24 synchronizes with the image signal based on the synchronization signal of the digital image signal output from the A / D converter 21. Thus, the digital audio signal supplied from the A / D converter 22 is output. Therefore, the sampling clock for A / D conversion of the audio signal in the A / D converter 22 is A / D
It is necessary to synchronize with the synchronizing signal of the analog image signal input to the D converter 21.

In the image compression unit 25 or the audio compression unit 26, the image or the audio from the selection unit 23 or 24 is encoded (MPEG encoded) according to, for example, the MPEG (eg, MPEG2) standard. As a result, an elementary stream defined by the standard is output. This elementary stream is, for example, DM
A (Direct Memory Access) controller, a bus interface 27, and a DM
A is transferred.

The bus interface 27 supplies images and sounds transferred via the extension bus to the image / sound insertion unit 28, and the image / sound insertion unit 28 sends the images or sounds from the bus interface 27. To the selection unit 2
3 or 24 respectively.

The image compression section 25 encodes and outputs an image input thereto, and also outputs an invalid flag and a scene change flag as described later. The invalid flag and scene change flag of
Is output along with the compressed data obtained by compressing the image.
The transfer has been made.

Next, the operation of the nonlinear editing apparatus shown in FIG. 1 will be described.

For example, when editing an image and a sound recorded by the video camera 14, the user loads the image and the sound into the non-linear editing device. That is,
The user causes the video camera 14 to reproduce an image and a sound, for example, by operating a play button (not shown) of the video camera 14.

The reproduction of images and sounds by the video camera 14 can also be performed by operating the keyboard 8 and the mouse 9, for example. That is, when the keyboard 8 and the mouse 9 are operated so as to instruct the video camera 14 to reproduce, an operation signal to that effect is received by the microprocessor 1 via the I / O interface 7 and the bus bridge 4. You. When receiving the operation signal, the microprocessor 1 transmits a control signal for instructing reproduction to the video camera 14 via the bus bridge 4 and the compression unit 13. In the video camera 1, reproduction of images and sounds is started according to a control signal from the microprocessor 1.

When the reproduction of the image and the sound by the video camera 14 is started, the image and the sound are fetched in the non-linear editing apparatus according to, for example, a flowchart shown in FIG.

That is, first, in step S1, the image and sound reproduced by the video camera 14 are M
PEG encoded (compressed).

Specifically, the image and the sound reproduced by the video camera 14 are supplied to the compression unit 13. In the compression unit 13 (FIG. 2), if the image or sound reproduced by the video camera 14 is a digital signal, the image or sound is selected by the selection unit 23 or 24 and supplied to the image compression unit 25 or the sound compression unit 26. Is done. If the image or sound reproduced by the video camera 14 is an analog signal, the A / D converter 21 or 2
2 is converted into a digital signal through the selector 2
3 or 24. In this case, the selector 23 or 24 selects a digital image or audio output from the A / D converter 21 or 22, and supplies the digital image or audio to the image compressor 25 or the audio compressor 26, respectively.

In the image compression unit 25 or the audio compression unit 26, the image or the audio from the selection unit 23 or 24 is MPEG-coded in step S 1, and is output to the bus interface 27. Further, in the image compression section 25, an invalid flag and a scene change flag are output as appropriate, and output to the bus interface 27. The bus interface 27 includes an image compression unit 25
The output of the audio compression unit 26 is supplied to the auxiliary storage interface 10 and recorded on the hard disk 12.

When the recording of data on the hard disk 12 is started as described above, the microprocessor 1 reads the data in step S2, and performs a scene file conversion process for forming a scene file as described later. Further, the microprocessor 1 performs step S
In step 3, a virtual album as described later is formed and displayed on the display 7, and the image and audio capturing process ends.

Next, the image compression section 25 in FIG. 2 outputs the above-mentioned invalid flag and scene change flag by performing the following image classification information detection processing.

That is, first, in step S11, the total sum S (Y) of the luminance values Y of the image of one frame to be encoded now is calculated. Specifically, 1
In the case where the image frame is composed of N pixels, then the n-th luminance value of the pixel represented as Y _n, in step S11, according to the following formula, the sum of luminance values of an image of one frame S (Y) is calculated.

S (Y) = ΣY _n (1) where Σ in the equation (1) means a summation where n is changed from 1 to N.

[0074] Then, the process proceeds to step S12, the total sum S of the brightness values calculated in step S11 (Y) is, whether the predetermined threshold epsilon ₁ less is determined. Step S1
In 2, the case where S (Y) is determined to epsilon ₁ smaller, the process proceeds to step S13, first invalid flag indicating that effect is output, the process proceeds to step S14.

In step S12, S (Y)
If but that is judged not to be less than epsilon _1, step S1
3 is skipped, that is, without outputting the first invalid flag, the process proceeds to step S14, and the DC
All T (Discrete Cosine Transform) coefficient of a DC component (direct current component) whether greater than a predetermined threshold epsilon ₂ is determined. Here, the DCT coefficient is calculated by the image compression unit 25.
When the image is MPEG-encoded in (1), since the image is obtained in units of macro blocks (blocks of 16 × 16 pixels), the determination processing in step S14 is performed based on the DC component.

[0076] In step S14, if all DC components of DCT coefficients for one frame is determined to be greater than epsilon _2, the process proceeds to step S15, a second invalid flag indicating that effect is output, the process proceeds to step S16 .

[0077] Further, in step S14, if the DC component of the DCT coefficients for one frame is not greater than epsilon _2, skips step S15,
That is, without outputting the second invalid flag, step S16
To calculate the correlation (inter-frame correlation) C between the frame to be encoded and the frame immediately before the current frame. That is, for example, in a case where an image of one frame is composed of N pixels and the luminance value of the n-th pixel of the f-th frame is represented as Y _{f, n} , in step S16, Inter-frame correlation C
Is calculated.

C = {| Y _{f, n} -Y _{f-1, n} | (2) However, as in the case of the equation (1), Σ in the equation (2) also Means summation.

After calculating the inter-frame correlation C, the process proceeds to step S17, where the inter-frame correlation C is set to a predetermined threshold ε.
_It is determined whether it is greater than ₃ . In step S17, if the inter-frame correlation C is determined to be greater than epsilon _3, the process proceeds to step S18, the third invalid flag that represents that fact is output, the process proceeds to step S21.

[0080] Further, in step S17, if the inter-frame correlation C is not greater than epsilon _3, the process proceeds to step S19, the inter-frame correlation C is the predetermined threshold epsilon ₄
(However, it is determined whether it is smaller than ε ₃ > ε ₄ ).
In step S19, if the inter-frame correlation C is determined epsilon ₄ smaller, the process proceeds to step S20, the scene change flag indicating that fact is output, the step S
Proceed to 21. Further, in step S19, if the inter-frame correlation C is determined to not less than epsilon _4, skips step S20, the process proceeds to step S21, whether the input image from the video camera 21 has been completed is determined If it is determined that the processing has not been completed, the process returns to step S11, and the same processing is performed on the next frame.

On the other hand, if it is determined in step S21 that the input of all the images from the video camera 21 has been completed, the invalid flag (first to third information) as information for classifying the images from the video camera 21 is determined. Invalid flag)
Then, the image classification information detection processing for detecting the scene change flag ends.

In the bus interface 27, the invalid flag and the scene change flag are associated with the encoded image data of the frame that was the target of the image classification information detection processing when each flag was output. Is output.

Next, in the microprocessor 1,
Based on the invalid flag obtained by the above-described image classification information detection processing, an image deletion processing of detecting and deleting an invalid frame that is not required by the user is performed according to the flowchart shown in FIG.

That is, first, in step S22, the first invalid flag is set as a predetermined number of frames.
For example, it is determined whether or not the number of frames constituting one GOP (Group Of Picture) or the like is continuous. If it is determined in step S22 that the first invalid flag has continued for one GOP or more, the process proceeds to step S23, in which the encoded image data is stored in the hard disk 1
2 and the process proceeds to step S24. That is, the first invalid flag is the sum S of the luminance values of the image of one frame.
(Y) is output when the threshold epsilon ₁ is smaller than but, S
(Y) is that when epsilon ₁ less than, nothing is considered that recording that is not. For example, a portion where such a frame continues for one GOP or more, which is an interval at which I-pictures appear, is considered to be an invalid frame not required by the user, and is therefore deleted in step S23.

When the first invalid flag continues for a long time, it is considered that the recording by the video camera 14 has been completed. Accordingly, whether or not the input of all the images from the video camera 21 in step S21 in FIG. 4 has been completed can be determined by determining whether or not the output of the first invalid flag has continued for a long time.

On the other hand, if it is determined in step S22 that the first invalid flag is not continuous for one GOP or more, step S23 is skipped and step S24 is performed.
It is determined whether or not the second invalid flag has continued for several tens of seconds, for example. If it is determined in step S24 that the second invalid flag has continued for several tens of seconds or more, the process proceeds to step S25, in which the encoded image data is deleted from the hard disk 12, and
Proceed to step S26. That is, the second invalid flag is 1
While the DC component of the DCT coefficients for frames is output if all larger than the threshold epsilon _2, because in this way the DC component is epsilon ₂ smaller frames, such as the entire frame is filled with a certain single color It is considered to be something. A portion where such a frame continues for several tens of seconds or more is considered to be an invalid frame that is not required by the user, and is therefore deleted in step S25.

Note that a single-color frame may be used at the time of editing, for example, when a fade is applied. Therefore, as in the case of a frame in which nothing is recorded, the time of one GOP is used. As described above, it is deleted not when it is short but when it is continuous for a long time such as several tens of seconds.

On the other hand, if it is determined in step S24 that the second invalid flag is not continuous for more than several tens of seconds, step S25 is skipped, the process proceeds to step S26, and the third invalid flag is set to, for example, It is determined whether or not continuous for several tens of seconds or more. In step S26,
If it is determined that the third invalid flag has continued for several tens of seconds or more, the process proceeds to step S27, in which the encoded image data is deleted from the hard disk 12, and the process returns to step S22. That is, the third invalid flag is output when the inter-frame correlation C is larger than the threshold ε ₃ ,
Thus, a frame in which the inter-frame correlation C is larger than ε ₃ (a frame in which the inter-frame correlation is very large) is considered to be the same image as the previous frame. A portion where such a frame continues for several tens of seconds or more is considered to be an invalid frame not required by the user, and is therefore deleted in step S27.

After the processing in step S27, step S2
2, the same process is repeated.

Note that steps S23, S25, and S
When the image is deleted at 27, the sound corresponding to the image is also deleted.

In the above case, the image recorded on the hard disk 12 is deleted. However, the deletion of the image (similarly for the sound) is performed by the image compression unit 25.
Alternatively, it may be performed before the image is encoded (for the audio, the audio compression unit 26 may be used).

Here, the thresholds ε _{1 to} ε ₃ in the image classification information detection processing of FIG. 4 are set to appropriate values based on the viewpoint of eliminating invalid frames as described above.

Next, details of the scene file conversion process in step S2 of FIG. 3 will be described with reference to the flowchart of FIG.

In the scene file conversion process, a moving image is
A set of scenes is detected, and image data and audio data for each scene are made into one file (hereinafter, a scene file as appropriate).

That is, first, in step S31, a variable i representing a scene number as a unique number for identifying a scene file is initialized to, for example, 1 as an initial value, and the flow proceeds to step S32. In step S32, the scene file with the scene number #i is opened, and the process proceeds to step S33, in which the image and the sound recorded on the hard disk 12 are MPEG-encoded data (hereinafter, appropriately referred to as compressed data). Is recorded. That is, when the scene file creation process is started, the compressed data recorded on the hard disk 12 is sequentially read from its head, and the compressed data is recorded in the scene file opened in step S32. Then, the process proceeds to step S34, where it is determined whether a scene change flag has been received. If it is determined that the scene change flag has not been received, the process returns to step S33, and recording of the compressed data in the scene file of the scene number #i is continued. You.

On the other hand, if it is determined in step S34 that the scene change flag has been received, the process proceeds to step S34.
35, the scene number #i is associated with the scene file opened in step S32,
Proceed to 36. In step S36, the scene file with the scene number #i is closed.

[0097] Here, the scene change flag, but the inter-frame correlation C is output when the threshold epsilon ₄ smaller,
Because thus inter-frame correlation C is epsilon ₄ smaller frames (inter-frame is very small frame correlation), considered to be finished the recording of the scene, which is the frame start recording a new scene Can be Therefore,
In step S33, the compressed data until the scene change flag is detected is recorded in one scene file, assuming that the data constitutes a set of scenes.

After the scene file of the scene number #i is closed, the process proceeds to step S37, and it is determined whether or not reading of the compressed data recorded on the hard disk 12 has all been completed. If it is determined in step S37 that the reading of all the compressed data has not been completed, the process proceeds to step S38, where the scene number #i is incremented by 1, and the process returns to step S32. That is,
In this case, a scene file for the next scene is configured. On the other hand, if it is determined in step S37 that the reading of the compressed data has all been completed, the scene file conversion process ends.

As described above, after the creation of the scene file in which the compressed data is converted into a file for each scene is completed, the compressed data recorded on the hard disk 12 is deleted.

In the above-described case, the image deletion processing and the scene file conversion processing are performed after the compressed data obtained by the compression unit 13 is recorded on the hard disk 12. Is transferred to the main memory 2 via the bus bridge 4 and DMA-stored. The compressed data stored in the main memory 2 is subjected to image deletion processing and scene file conversion processing. The resulting scene file can be recorded on the hard disk 12.

Next, the details of the album display process in step S3 in FIG. 3 will be described with reference to FIGS.

In the album display processing, first, in step S41, the variable i representing the scene number is set to the value shown in FIG.
Is initialized to 1, which is the initial value, and the process proceeds to step S42, where the scene number #
i scene file (hereinafter, scene file #i
Is written) is opened. Then, step S4
Proceeding to No. 3, a clip screen which is a still image representing the scene corresponding to the scene file #i is generated.

That is, in step S43, the microprocessor 1 reads out the compressed data of the image recorded in the scene file #i and performs a predetermined decoding algorithm (a decoding algorithm corresponding to the processing of the decompression unit 15).
(Decoded by software).
Then, for example, the microprocessor 1 reduces the I-picture constituting the GOP decoded first, for example, by thinning out the number of pixels, and sets it as a clip screen. This clip screen is recorded on the hard disk 12 in step S44 in association with the scene file #i.

Then, the process proceeds to a step S45, in which the microprocessor 1 displays the clip screen generated in the step S45 so as to be pasted on the virtual album previously displayed on the display 17.

That is, when the data fetch process described with reference to FIG. 3 is started, the microprocessor 1
A virtual album (album image) as shown in FIG. 8A is written in the frame buffer 3 and transferred to the display 17 via the decompression unit 15 for display. Then, a clip screen is displayed in this album, for example, as shown in FIG.

Thereafter, the flow advances to step S46 to determine whether or not clip screens have been generated for all scene files recorded on the hard disk 12. If not, the flow advances to step S47 to indicate a scene number. The variable i is incremented by 1, and the process returns to step S42. That is, in this case, a clip screen for the next scene file is generated and pasted to the album.

On the other hand, if it is determined in step S46 that clip screens have been generated for all scene files recorded on the hard disk 12, the album display processing ends.

As described above, according to the album display processing, a clip screen for each scene file is generated and pasted to the album.

Here, FIG. 8C shows a state where clip screens are generated for five scene files and pasted to an album. In the figure, the numbers attached to the upper part of the clip screen indicate scene numbers.

In the album, a clip screen is pasted with its arrangement position shifted in the order in which the clip screen is generated, with a predetermined position as the origin. Further, when there is no more space for pasting a clip screen on the album, the microprocessor 1 opens a new page of the album and pastes the clip screen on the page.

As described above, a group of scenes is detected from the input moving image, and a clip screen for each scene is generated and pasted to the album. As described above, invalid frames (and accompanying sounds) are deleted from each scene. Therefore,
The user is not required to search for and delete such invalid frames and perform editing operations such as cutting for each scene, but only to perform operations for capturing images and sounds to be edited. Even if it is not performed, invalid frames are deleted, and data edited and cut for each scene can be easily obtained. Further, since the invalid frame is deleted, the recording capacity of the hard disk 12 can be effectively used.

Further, by looking at the clip screen pasted on the album, it is possible to grasp the contents without reproducing each scene. The display 17
The album displayed above can be used in the same manner as the album on which the pictures are actually pasted.

In the above-described case, the clip screen is pasted to the album after the image and the sound are compressed and converted into a scene file, but these processes are performed simultaneously (progress in parallel). Is possible.

That is, the image and sound are compressed to form a scene file, and the compressed data of the image is decoded by the decompression unit 15 and the reduced image is displayed at the position of the album where the clip screen of the scene file is pasted. Let it. Then, after the compression and the creation of a scene file for a certain scene are completed, a clip screen for the scene is generated, and the clip screen is pasted at the position of the album where the reduced image was displayed, and thereafter, The same processing is performed for the next scene. In this case, the image being compressed is displayed on the album, and when the scene file is created, the clip screen is displayed on the album and the image of the next scene (compression is not performed). Since the (started image) is displayed in the album, the user can grasp the progress of the processing.

Next, after the album has been constructed as described above, the scene files corresponding to the clip screen pasted on the album are reproduced in the order of the scene numbers, for example, as described later. Therefore, there is no need to perform subsequent operations for a user who is satisfied with such reproduction.

However, in some cases, it is required to change the order in which the scene files are reproduced, or to delete some scenes from the scene files created by the data import process of FIG.

Therefore, in the nonlinear editing apparatus shown in FIG.
It is made possible to easily replace or delete scenes.

In other words, when changing the playback order of scenes (when replacing scenes), the user inputs a command by operating the keyboard 8, for example. (It may be clicked with the mouse 9), and the operation mode of the apparatus is set to the sorting mode. When the operation mode is set to the rearrangement mode, the microprocessor 1 performs processing according to the flowchart of FIG. That is, step S51
In, it is determined whether the clip screens pasted to the album have been rearranged (whether or not the position of the clip screen has been moved so that the rearrangement has been performed). S51
Return to

If it is determined in step S51 that the clip screens pasted on the album have been rearranged, the flow advances to step S52 to arrange the rearranged clip screens and, accordingly, the scene file corresponding to each clip screen. Are changed, and the process ends.

That is, the clip screen pasted on the album can be moved by dragging with the mouse 9, for example. Now, for example, as shown in FIG. 10 (A), in a case where a row of clips screen album, the Clip screen C ₁ scene number # 3, the position of the clip window C ₂ scene number # 4, It is assumed that the mouse 9 has been replaced by operating the mouse. In this case, as shown in FIG. (B), the scene number of the scene file corresponding to the clip screen C ₁ to 4,
Scene number of the scene file corresponding to the clip screen C ₂ to 3, are replaced respectively.

Accordingly, the user can easily change the reproduction order of the scene.

Note that the clip screen can be moved not only to the page on which the clip screen is pasted but also to another page.

Next, for example, when the operation mode of the apparatus is set to the sort mode as described above, or when the clip mode is set to the clip screen edit mode described later, the clip screen is moved out of the album. Sometimes, the scene corresponding to the clip screen is cut (deleted).

That is, as described above, the clip screen can be moved by dragging. In such a case, when the clip screen is moved out of the album and the drag is released, for example, As shown in the flowchart of FIG. 11, in step S61,
The data about the clip screen is stored on the hard disk 1
2 is deleted. Specifically, the image data of the clip screen and the scene file corresponding thereto are deleted.

Then, in step S62, the clip screens are rearranged, and the flow advances to step S63 to arrange the clip screens rearranged in step S62, so that the scene numbers of the scene files corresponding to the respective clip screens are rearranged. To end.

That is, for example, as shown in FIG. 12A, the clip screen of scene number # 3 is dragged,
If the clip is moved out of the album and the drag is released, the clip screen of scene number # 3 is deleted.
Then, as shown in FIG. 7B, the clip screens of scene numbers # 4 and # 5 are moved so as to fill the blank at the deleted position. Furthermore, the scene numbers # 4 and #
The scene numbers of the scene files corresponding to the clip screen No. 5 are changed to # 3 and # 4, respectively.

Therefore, the user can easily delete a scene.

Next, the user may want to edit the contents of each scene. In this case, the user
By operating the keyboard 8 and the mouse 9, the operation mode of the apparatus is set to the scene editing mode. When the operation mode is set to the scene editing mode, the microprocessor 1 performs, for example, processing according to the flowchart of FIG.

That is, first, in step S71, it is determined whether or not the clip screen corresponding to the scene to be edited is selected. If it is determined that the clip screen has not been selected, the process returns to step S71. Step S71
When it is determined that the clip screen corresponding to the scene to be edited is selected, the process proceeds to step S72,
An editing window is opened and displayed on the display 17.

Specifically, for example, as shown in FIG. 14A, when an album on which a clip screen is pasted is displayed, the user clicks the mouse 9 to select one of the albums. Is selected. In this case, for example, an editing window as shown in FIG.

A play button 48 for playing back a scene corresponding to the selected clip screen is displayed in the edit window.
In step S73, it is determined whether the play button 48 has been clicked with the mouse 9, and if it is determined that the play button 48 has not been clicked, the process returns to step S73. If it is determined in step S73 that the play button 48 has been clicked, the process proceeds to step S74, and the scene file corresponding to the clip screen determined to be selected in step S71 is played at a predetermined playback speed. .

That is, in step S74, the scene file is read from the hard disk 12, and the
5 is supplied. In the decompression unit 15, the compressed data of the image in the scene file is decoded and displayed on the display window 41 of the editing window. Specifically, the image data of the editing window is written in the frame buffer 3 by the microprocessor 1 and
Overlays the image obtained by decoding the compressed data on the display window 41 of the editing window and causes the display 17 to display the image.

Here, when the display of the moving image on the display window 41 is started, as shown in FIG. 15A, the image displayed on the display window 41 is displayed in the time code column 42 of the editing window. Corresponding time code (time stamp)
Is displayed, and the reproduction indicator 44 of the editing window is filled in in accordance with the data amount of the reproduced image. Therefore, the user can grasp the reproduction time of the image (scene) by referring to the time code column 42 and the reproduction indicator 44.

Thereafter, the flow advances to step S75 to determine whether a start point for cutting the image has been designated. That is, when the user wants to start cutting an image from a certain frame while looking at the image displayed on the display window 41, the start point can be designated at that timing by, for example, clicking the reproduction indicator 44. In step S75, it is determined whether this designation has been made. If it is determined in step S75 that the start point has not been specified, the process proceeds to step S76, in which it is determined whether or not all the reproduction of the scene file selected in step S71 has been completed. In this case, the process returns to step S74, and the reproduction of the scene file is continued.

If it is determined in step S76 that the reproduction of the scene file has been completed, the process proceeds to step S83, and a portion sandwiched between the start point and an end point (described below) in the reproduced scene. , Or a cutscene, as appropriate). If it is determined in step S83 that no cut scene exists, that is, if no start point has been designated, step S84 is skipped and the process ends.

On the other hand, if it is determined in step S75 that the start point has been designated, the process proceeds to step S77,
For example, the start point is set to the first frame of the GOP including the frame whose start point is specified, and the process proceeds to step S78.

When the start point is set in step S77, for example, as shown in FIG. 15B, a display indicating the start point is displayed at the position of the reproduction indicator 44 corresponding to the start point. This allows the user to easily recognize the position where the start point is set.

Returning to FIG. 13, in step S78, reproduction of the scene file is continued, and the flow advances to step S79.
It is determined whether an end point for cutting the image has been designated. That is, after the start point is set, a frame for ending the cut of the image can be specified in the same manner as specifying the start point.
At 9, it is determined whether or not this designation has been made. If it is determined in step S79 that the end point has been specified, the process proceeds to step S80, where the end point is set to, for example, the last frame of the GOP including the frame whose end point is specified, and the process returns to step S74.

When the end point is set in step S80, for example, as shown in FIG. 15C, a display indicating the end point is displayed at the position of the reproduction indicator 44 corresponding to the end point. Thus, the user can easily recognize the position where the end point is set. Further, when an end point is set, the portion of the playback indicator 44 corresponding to the end point and the start point set immediately before the end point, that is, the portion corresponding to the cut scene is shown in FIG. C) is shaded in a different color as shown by diagonal lines, so that the user can easily recognize the cut scene.

Returning to FIG. 13 again, if it is determined in step S79 that the end point has not been designated, the flow advances to step S81 to determine whether or not all the reproduction of the scene file selected in step S71 has been completed. If it is determined that the processing has not been completed, step S78
And the reproduction of the scene file is continued.

If it is determined in step S81 that all the scene files have been reproduced, the flow advances to step S82 to set an end point for the last reproduced frame, and the flow advances to step S84.

In step S84, a cut scene is extracted (cut) from the scene file, cut edit processing for a new scene file is performed, and the processing ends.

Next, details of the cut editing process in step S84 in FIG. 13 will be described with reference to the flowchart in FIG.

In the cut editing process, first, in step S91, it is determined whether or not a cut scene has been selected. That is, the user operates the cut scene portion displayed on the reproduction indicator 44 (for example, FIG. 15C).
By clicking, for example, with a mouse 9, the cut scene can be selected as a scene constituting a new scene file. In step S91, this selection is made. It is determined whether the operation has been performed.

If it is determined in step S91 that a cut scene has not been selected, the process proceeds to step S9.
Return to 1. If it is determined in step S91 that a cut scene has been selected, the process proceeds to step S92.
And a new scene file is created for the selected cut scene.

That is, a portion corresponding to the cut scene is read out from the scene file in which the compressed data of the image and the sound constituting the selected cut scene is recorded, and is set as a new scene file. Will be recorded.

Here, for example, as shown in FIG. 17A, when three cut scenes # 1, # 2, and # 3 are set, one of them, for example, cut scene # 1 Is selected, and when a new scene file is created for the cut scene # 1, the reproduction of the new scene file is performed in the time code column 43 of the editing window as shown in FIG. The time (in this case, the playback time of the cut scene # 1) is displayed, and the edit indicator 45 of the edit window is filled in according to the data amount.
Therefore, the user can grasp the reproduction time of a new scene file by referring to the time code column 43 and the edit indicator 45.

Returning to FIG. 16, after creating a new scene file, the flow advances to step S93 to determine whether or not a new cut scene has been selected. If it is determined in step S93 that no cut scene has been selected, step S94 is skipped and step S94 is skipped.
Go to 95. If it is determined in step S93 that a cut scene has been newly selected, the process proceeds to step S94, and the compressed data corresponding to the newly selected cut scene is added to the scene file created in step S92. The process is added, and the process proceeds to step S94.

That is, for example, in the case shown in FIG. 17B, when cut scene # 3 is selected, for example, a new scene file created for the previously selected cut scene # 1 is added to the new scene file. , Compressed data of cut scene # 3 is added. In this case, FIG.
As shown in the figure, in the time code column 43 of the editing window, the playback time of the scene file to which the compressed data of cut scene # 3 is added (accordingly, the total time of the playback times of cut scenes # 1 and # 3) is displayed. While being displayed, the edit indicator 45 of the edit window is also
Again, the compressed data of the cut scene # 3 is painted in accordance with the data amount of the added scene file. Therefore, the user can grasp the reproduction time of the scene file to which cut scene # 3 has been added by referring to the time code column 43 and the edit indicator 45.

Returning to FIG. 16 again, in step S95, it is determined whether or not the keyboard 8 or the mouse 9 has been operated so as to end the cut editing process. If it is determined that the keyboard 8 and the mouse 9 have not been operated, the process proceeds to step S93. Return. If it is determined in step S95 that an operation has been performed to end the cut editing process, the process ends.

Here, in FIG. 16 and FIG.
Although the cut scenes are arranged in the new scene file in the selected order, the arrangement order of the cut scenes can be changed. That is, for example, FIG.
Shows that cut scenes # 1 and # 2 are sequentially selected and arranged in a new scene file in such an order. In such a case, for example, if cut scene # 3 is selected and then the insertion position is specified by clicking the boundary between cut scenes # 1 and # 2 of edit indicator 45 with mouse 9, etc. The selected cut scene # 3 is arranged between cut scenes # 1 and # 2, as shown in FIG.

In the editing window, the reverse playback button 46 is operated when playing back the scene file selected in step S71 in FIG.
Reference numeral 7 is used to stop reproduction (reproduction in the forward direction) or reverse reproduction of the scene file. The reverse playback button 49 is used to reversely play a newly created scene file, and the stop button 50 is used to stop playback or reverse playback of a newly created scene file. Each operation is performed when a newly created scene file is reproduced. Therefore, the user can confirm the contents of the new scene file by operating the play button 51, for example.

When a new scene file is created as described above, a clip screen for the scene file is also generated and pasted to an album. That is, for example, as shown in FIG. 19A, when clip screens of five scene files are pasted on an album, one of the scene files is edited,
When a new scene file is created, a clip screen of the new scene file is generated, for example,
As shown in FIG. 7B, the data is added to the end of the album. Therefore, here, the scene number of the new scene file is 6 which is a value obtained by adding 1 to the number of clip screens already pasted to the album.

As described above, the user can easily edit a scene file simply by setting a cut scene by selecting a start point and an end point and selecting the cut scene.

After the scene file is edited and a new scene file is obtained, if the original scene file is unnecessary, as described above, the clip screen is easily taken out of the album so that it can be easily obtained. Can be deleted. Also, the scene playback order can be easily changed by changing the arrangement of the clip screens as described above.

In the above case, a new scene file is generated from one scene file. However, a new scene file can be generated from a plurality of scene files.

Next, the scene file includes, for example, images and sounds received from outside by the tuner 5 and the modem 6,
It is also possible to insert image and sound data created by the microprocessor 1 (hereinafter, appropriately referred to as insertion editing).

When insertion and editing are instructed, image data and audio data for a predetermined number of frames to be inserted are input to the compression unit 13 and, for example, processing according to the flowchart of FIG. 20 is performed.

That is, in the compression section 13 (FIG. 2), the input image data and audio data are received by the bus interface 27 and transferred to the image / audio insertion section. In the image / audio insertion unit 28, the image data or the audio data from the bus interface 27 is selected by the selection unit 23.
Alternatively, the data is supplied to the image compression unit 25 or the audio compression unit 26 via the or. In the image compression unit 25 or the audio compression unit 26, the input image data or audio data is encoded (compressed) in step S101, and the resulting compressed data is output to the bus interface 27. Bus interface 2
Reference numeral 7 DMA-transfers the compressed data from the image compression unit 25 and the audio compression unit 26 to the main memory 2 via the bus bridge 4.

When the compressed data to be inserted into the scene file is stored in the main memory 2 as described above, the microprocessor 1 reads the scene file to be inserted and edited from the hard disk 12 in step S102, and The compressed data stored in the main memory 2 is inserted into the file and recorded on the hard disk 12 again, and the process is terminated. The position at which the compressed data stored in the main memory 2 is inserted into the scene file is set in advance by the user before the above-described processing is performed, and is inserted at that position.

Next, for example, with the tuner 5 or the modem 6,
Images and sounds received from outside, microprocessor 1
In addition to inserting the image and audio data created in step (1) into the scene file, it is also possible to overwrite (replace with a part of the scene file) a part of the scene file (hereinafter, appropriately referred to as overwrite editing).

When overwrite editing is instructed, image data and audio data for a predetermined number of frames to be overwritten (hereinafter, also referred to as overwrite data as appropriate), and the position where they are overwritten (for example, a scene file for overwriting them) And the number of frames of image data to be overwritten (the overwriting position and the number of frames are assumed to have been input by the user in advance). The scene file to be edited is hard disk 1
2 is input to the decompression unit 15 and, for example, processing according to the flowchart of FIG. 21 is performed.

That is, in the decompression unit 15, step S111
In, the scene file input thereto is decompressed, and the resulting image data and audio data (hereinafter, also appropriately referred to as decompressed data) are supplied to the compression unit 13.

In the compression section 13 (FIG. 2), the bus interface 27 receives the input overwrite data, the position at which the overwrite data is overwritten, and the number of frames to be overwritten. Furthermore, bus interface 2
In 7, the decompression data from the decompression unit 15 is also received.

In the bus interface 27,
In step S112, it is determined based on the received position where the overwrite data is to be overwritten and the number of frames to be overwritten which of the overwrite data and the decompressed data is to be output. Step S
If it is determined in step 112 that it is time to output the decompressed data, the process proceeds to step S113, where the decompressed data is encoded.

That is, the bus interface 27 transfers the decompressed data to the image / sound insertion unit 28, and the image / sound insertion unit 28 selects the image data or the sound data as the decompression data from the bus interface 27.
The data is supplied to the image compression unit 25 or the audio compression unit 26 via 3 or 24, respectively. The image compression unit 25 or the audio compression unit 26 encodes (compresses) the input image data or audio data, respectively, and supplies the resulting compressed data to the bus interface 27.

On the other hand, if it is determined in step S112 that it is time to output overwrite data, the flow advances to step S114 to encode the overwrite data. That is, the bus interface 27 transfers the overwrite data to the image / audio insertion unit 28, and the image / audio data as the overwrite data from the bus interface 27 is selected in the image / audio insertion unit 28.
Alternatively, the data is supplied to the image compression unit 25 or the audio compression unit 26 via the or. The image compression unit 25 or the audio compression unit 26 encodes the input image data or audio data, respectively, and supplies the resulting compressed data to the bus interface 27.

The bus interface 27 discards the decompressed data received from the decompression unit 15 while outputting the overwrite data to the image / sound insertion unit 28, whereby the image compression unit The compressed data obtained by replacing the decompressed data of a predetermined portion with the overwrite data is supplied from the audio compression unit 25 and the audio compression unit 26 to the bus interface 27.

After the processing in steps S113 and S114, the flow advances to step S115 to supply the expanded data obtained by expanding the compressed data recorded in the scene file to be overwritten from the expansion unit 15 to the compression unit 13. It is determined whether or not all the processing has been completed. If it is determined that the processing has not been completed, the process returns to step S111, and the compressed data that has not been expanded is expanded. If it is determined that the supply of the decompressed data from the decompression unit 15 to the compression unit 13 has been completed, the process proceeds to step S116, where the compression data supplied from the image compression unit 25 and the audio compression unit 26 to the bus interface 27 is recorded. Is created and recorded on the hard disk 12 in place of the scene file targeted for overwrite editing.

In the above-described insertion editing and overwriting editing, it is possible to insert and overwrite still images in addition to moving images. Therefore, the user can create, for example, a still image as a title screen of a scene, and insert or overwrite the title screen in the scene.

Next, in the nonlinear editing apparatus shown in FIG. 1, in addition to the scene, for example, as shown in FIG. 8, an album and a clip screen pasted thereon can be edited.

That is, for example, when the keyboard 8 and the mouse 9 are operated in a predetermined manner and the operation mode of the apparatus is set to an album / clip screen editing mode for editing an album or a clip screen, the microprocessor 1 executes, for example, as shown in FIG.
The processing according to the flowchart of FIG.

That is, in this case, first, step S
At 121, it is determined whether an operation to move the clip screen has been performed. If it is determined in step S121 that an operation to move the clip screen has been performed, the process proceeds to step S122, the clip screen is moved in response to the operation, and the process proceeds to step S123. As described above, the clip screen can be moved by dragging with the mouse 9, and the step S
In 121 and S122, when such an operation is performed, the clip screen is moved. When the drag of the clip screen is released outside the album, the clip screen is deleted together with the corresponding scene file as described above.

Here, the clip screen can be moved not only within the same page but also to a different page as described above. Further, the clip screen allows a new album to be opened and moved to that album.
After the clip screen is moved, the scene number is changed as necessary, as described above.

If it is determined in step S121 that the operation of moving the clip screen has not been performed, step S122 is skipped and step S1 is performed.
Proceeding to 23, it is determined whether an operation to enlarge / reduce the clip screen has been performed. If it is determined in step S123 that an operation for enlarging / reducing the clip screen has been performed, the process proceeds to step S124, and the clip screen is enlarged / reduced in response to the operation.
Go to 125.

That is, the clip screen is enlarged / deleted by, for example, dragging and moving the boundary with the mouse 9.
The clip screen can be reduced. When such an operation is performed, the clip screen is enlarged / reduced in step S124.

On the other hand, if it is determined in step S123 that an operation for enlarging / reducing the clip screen has not been performed, step S124 is skipped, and the process proceeds to step S125 to determine whether an operation for deforming the clip screen has been performed. Is determined. If it is determined in step S125 that an operation to deform the clip screen has been performed, the process proceeds to step S126, and the clip screen is deformed according to the operation.

That is, the clip screen can be deformed by dragging and moving the boundary portion of the clip screen with the mouse 9 after operating the keyboard 8 and the mouse 9 in a predetermined manner. If any operation is performed, the clip screen is transformed in step S126.

If it is determined in step S125 that an operation for deforming the clip screen has not been performed, the flow advances to step S127 to determine whether an operation for inputting a character has been performed. If it is determined in step S127 that an operation of inputting a character has been performed, the process proceeds to step S128, the character input by the operation is added to the album, and the process proceeds to step S128.

That is, in the album, characters can be added to the clip screen or to a blank portion where the clip screen is not pasted. If a position is specified and a character is input by operating the keyboard 8 thereafter, the character input by the keyboard 8 is added to the position specified by the mouse 9 in step S128.

On the other hand, if it is determined in step S127 that an operation for inputting a character has not been performed, step S128 is skipped, and the flow advances to step S129 to perform an operation for ending the editing of the album and clip screen. Is determined, and if it is determined that the determination has not been performed, the process returns to step S121. If it is determined in step S129 that an operation to end the editing of the album and clip screen has been performed, the process ends.

Here, FIG. 23 shows an example of an editing result obtained by editing the album shown in FIG. 8C as described above, for example.

As described above, the user can edit the album to which the clip screen is pasted in the same manner as the album to which the picture is actually pasted.

[0184] Information on the result of editing the album is recorded on the hard disk 12, for example.

Next, a process of reproducing a scene file will be described.

For example, when the keyboard 8 and the mouse 9 are operated and the operation mode of the apparatus is set to the reproduction mode for reproducing the scene file, the microprocessor 1 executes, for example, the processing according to the flowchart of FIG. Will be

That is, first, in step S131, it is determined whether an album to be reproduced has been designated. Specifically, for example, the created album is
The state of the icon is displayed on the display 17, and whether any of the icons is clicked with the mouse 9 is determined in step S.
It is determined at 131. If it is determined in step S131 that the album to be played has not been specified, the process returns to step S131.

If it is determined in step S131 that the album to be reproduced has been designated, the flow advances to step S132, and the album is displayed on the display 17, for example, as shown in FIG.

Thereafter, in step S133, for example, it is determined whether or not the keyboard 8 and the mouse 9 have been operated so as to reproduce the entire album. If it is determined that the operation has been performed, the process proceeds to step 134, where the album is reproduced. Scene files corresponding to the pasted clip screen are sequentially reproduced, for example, in ascending order of scene numbers. That is, the compressed data recorded in the scene file to be reproduced is read from the hard disk 12 and supplied to the decompression unit 15. In the decompression unit 15, the compressed data is decoded, and the image as the decoding result is displayed on the display 1.
7 is displayed. The sound is output from a speaker (not shown). When all the scene files corresponding to the clip screen pasted on the album have been completely reproduced, the process proceeds to step S137.

On the other hand, if it is determined in step S133 that the operation to reproduce the entire album has not been performed, the process proceeds to step S135, and one of the clip screens pasted on the album is displayed, for example, as follows. It is determined whether or not selection has been made by clicking with the mouse 9. If it is determined in step S135 that the clip screen has not been selected, step S1
Return to 33. If it is determined in step S135 that the clip screen has been selected, the process proceeds to step S13.
Proceeding to 6, the scene file corresponding to the selected clip screen is reproduced as described above.

After the end of the reproduction, step S1
Proceeding to 37, for example, it is determined whether the keyboard 8 or the mouse 9 has been operated to end the playback mode, and if it is determined that the keyboard 8 or the mouse 9 has not been operated, the process proceeds to step S1.
Return to 31. If it is determined in step S137 that an operation has been performed to end the playback mode, the process ends.

[0192] The edited data is recorded on the hard disk 12 and can be conveniently carried.
D-R11 or DVD (Digital Versatile not shown)
Disc) -R and the like. When the edited data is recorded on the CD-R 11 or the like, it becomes possible to easily organize and save the moving image, as in the case of the album to which the photo is pasted.

The edited data can be reproduced in the above-mentioned reproduction mode and recorded on a video tape by the VTR 16, for example.

Next, FIG. 25 shows an MPE to which the present invention is applied.
1 shows a configuration example of an embodiment of a G video camera.

When the operation unit 112 is operated and recording is instructed by the operation unit 112, the light incident on the lens 101 is changed to the CCD.
The light is condensed on the lens 102, and the CCD 102
Is photoelectrically converted, and the resulting image signal as an electric signal is output to the A / D converter 103.
The A / D converter 103 converts the image signal from the CCD 102 into a digital image signal by A / D conversion, and supplies the digital image signal to the editing unit 106.

On the other hand, the sound input to the microphone (microphone) 104 is converted into an audio signal as an electric signal there and supplied to the A / D converter 105. The A / D converter 105 converts the audio signal from the microphone 104 into a digital audio signal by A / D conversion, and supplies the digital audio signal to the editing unit 106.

In the editing unit 106, the image signal or the audio signal from the A / D converter 103 or 105 is
The image signal and the audio signal are subjected to the above-described image classification information detection processing (FIG. 4), image deletion processing (FIG. 5), and scene file processing (FIG. 6). You. Then, the resulting scene file is recorded on the hard disk (HD) 107.

When the scene files are stored in the hard disk 107, the albuming section 108 generates a clip screen for each scene file and pastes it into a virtual album (hereinafter, referred to as albuming processing as appropriate). Is performed, and the data obtained as a result of the albuming process (hereinafter, appropriately referred to as album data) is recorded on the hard disk 107.

After that, when the operation unit 112 is operated and the reproduction is instructed, the reproduction unit 109 reproduces the album data from the hard disk 107 and outputs it to the display unit 110 constituted by, for example, a liquid crystal display. . As a result, the display unit 110 displays, for example, FIG.
An album to which the clip screen is pasted as shown in (C) is displayed. Then, the processing in the reproduction mode as shown in FIG. 24 is performed.

That is, when the operation unit 112 is operated to reproduce the entire album, the scene files corresponding to the clip screens pasted on the album are sequentially reproduced, for example, in ascending order of the scene numbers. . That is, the compressed data recorded in the scene file is read from the hard disk 107 and
G decoding is performed. Then, the resulting image or sound is displayed on the display unit 110 or the speaker 11
1 is output. The above processing is repeated for the scene file corresponding to the clip screen pasted on the album.

When the operation unit 112 is operated to select one of the clip screens pasted on the album, the scene file corresponding to the selected clip screen is set as described above. Will be played.

Therefore, after recording, the user can immediately obtain a desired scene without performing editing for classifying each scene or deleting invalid frames.

The linear editing apparatus and the video camera to which the present invention has been applied have been described above. However, the present invention can be applied to, for example, an apparatus for recording an image such as a VTR and the like.

In the present embodiment, the data is encoded / decoded by the MPEG method. However, the data encoding / decoding method is not only the MPEG method but also the MPEG method.
For example, it is possible to adopt a motion JPEG system, a DV system, or the like. Furthermore, data encoding / decoding does not always need to be performed.

In the present embodiment, the scene change is detected based on the inter-frame correlation. However, the method for detecting the scene change is not limited to this. When a scene change is detected based on the inter-frame correlation, the scene is separated by a frame having a small inter-frame correlation. Scenes can be separated by panning at high speed.

[0206]

According to the image processing apparatus of the first aspect and the image processing method of the twenty-sixth aspect, a group of scenes is detected from an input moving image, and a still image representative of the scene is detected. A certain clip screen is generated. Then, a virtual album to which the clip screen is pasted is generated. Therefore, editing of a moving image can be performed easily and in a short time.

According to the image reproducing apparatus of the twenty-seventh aspect and the image reproducing method of the thirty-second aspect, a group of scenes is detected from an input moving image, and a clip which is a still image representative of the scene is detected. A scene file and an album, which are files of image data of a scene obtained by generating a screen and generating a virtual album to which a clip screen is pasted, are reproduced. Therefore, it is possible to easily obtain a desired scene.

[0208] In the recording medium according to the thirty-first aspect, a set of scenes is detected from an input moving image, a clip screen as a still image representing the scene is generated, and the clip screen is pasted. A scene file, which is a file of image data of a scene obtained by generating a typical album, and an album are recorded.
Therefore, it is possible to easily obtain a desired scene.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration example of a non-linear editing apparatus according to an embodiment of the present invention;

FIG. 2 is a block diagram illustrating a configuration example of a compression unit 13 in FIG.

FIG. 3 is a flowchart for explaining the operation of the nonlinear editing device of FIG. 1;

FIG. 4 is a flowchart illustrating an image classification information detection process.

FIG. 5 is a flowchart illustrating an image deletion process.

FIG. 6 is a flowchart illustrating details of a process in step S2 of FIG. 3;

FIG. 7 is a flowchart illustrating details of a process in step S3 of FIG. 3;

FIG. 8 is a diagram for explaining album display processing.

FIG. 9 is a flowchart illustrating a process in a rearrangement mode.

FIG. 10 is a diagram for explaining processing in a rearrangement mode;

FIG. 11 is a flowchart illustrating a process performed when a clip screen is output from an album.

FIG. 12 is a diagram for describing processing when a clip screen is output from an album.

FIG. 13 is a flowchart illustrating a process in a scene editing mode.

FIG. 14 is a diagram for describing processing in a scene editing mode.

FIG. 15 is a diagram for describing processing in a scene editing mode.

FIG. 16 is a flowchart illustrating details of a process in step S84 in FIG. 13;

FIG. 17 is a diagram for explaining cut editing processing.

FIG. 18 is a diagram for explaining cut editing processing.

FIG. 19 is a diagram for explaining cut editing processing.

FIG. 20 is a flowchart for explaining an insertion editing process.

FIG. 21 is a flowchart illustrating an overwrite editing process.

FIG. 22 is a flowchart illustrating processing in a clip screen editing mode.

FIG. 23 is a diagram showing a processing result in a clip screen editing mode.

FIG. 24 is a flowchart illustrating a process in a playback mode.

FIG. 25 is a block diagram illustrating a configuration example of an embodiment of a video camera to which the present invention has been applied.

FIG. 26 is a diagram showing a configuration of an example of a conventional linear editing device.

FIG. 27 is a diagram illustrating a configuration of an example of a conventional nonlinear editing device.

[Explanation of symbols]

1 microprocessor, 2 main memory, 3
Frame buffer, 4 bus bridge, 5 tuner, 6 modem, 7 I / O interface,
Reference Signs List 8 keyboard, 9 mouse, 10 auxiliary storage interface, 11 CD-R, 12 hard disk, 13 compression unit, 14 video camera, 15
Extension unit, 16 VTR, 17 computer display, 21, 22 A / D converter, 23, 2
4 selection unit, 25 image compression unit, 26 audio compression unit, 27 bus interface, 28 image / audio insertion unit, 101 lens, 102 CCD, 1
03 A / D converter, 104 microphone, 105 A
/ D converter, 106 editing unit, 107 hard disk, 108 albuming unit, 109 playback unit,
110 display unit, 111 speaker, 112 operation unit

Claims (31)

[Claims] 1. An image processing apparatus for processing a moving image, comprising: a scene detecting unit that detects a group of scenes from an input moving image; and generating a clip screen as a still image representing the scene. An image processing apparatus comprising: a clip screen generating unit; and an album generating unit that generates a virtual album to which the clip screen is pasted. 2. The apparatus according to claim 1, wherein the scene detecting means detects a scene change of the moving image and detects a moving image from a previous scene change to a current scene change as one scene. An image processing apparatus according to claim 1. 3. The image processing apparatus according to claim 2, wherein the scene detecting unit detects the scene change based on a correlation between frames of the moving image. 4. The image processing apparatus according to claim 1, further comprising: an invalid frame detecting unit that detects an invalid frame that is not required by the user from the moving image; and an invalid frame deleting unit that deletes the invalid frame from the scene. The image processing apparatus according to claim 1. 5. The image processing apparatus according to claim 4, wherein the invalid frame detecting unit detects the invalid frame based on a luminance value of a frame forming the moving image. 6. The invalid frame detecting means converts a frame constituting the moving image into a DCT (Discreate Cosine Trajectory).
The image processing apparatus according to claim 4, wherein the invalid frame is detected based on a DC component of a DCT coefficient obtained by performing nsfer). 7. The image processing apparatus according to claim 4, wherein the invalid frame detection unit detects the invalid frame based on a correlation between frames of the moving image. 8. The image processing apparatus according to claim 1, wherein the clip screen generating unit sets a predetermined frame constituting the scene as the clip screen. 9. The image data of the scene is represented by MP
The encoding apparatus according to claim 8, further comprising an encoding unit that performs EG (Moving Picture Experts Group) encoding, wherein the clip screen generation unit sets the first I picture in the scene as the clip screen. Image processing device. 10. The image processing apparatus according to claim 1, further comprising a recording unit that records the scene file, which is a file of image data of the scene, and the album.
An image processing apparatus according to claim 1. 11. The image processing apparatus according to claim 10, further comprising a reproducing unit that reproduces the scene file and the album recorded by the recording unit. 12. The image processing apparatus according to claim 11, wherein the reproducing unit reproduces the scene file corresponding to the clip screen pasted on the album in a predetermined order. 13. The image processing apparatus according to claim 11, wherein the recording unit also records an image obtained by the reproduction unit reproducing the scene file. 14. The apparatus according to claim 1, further comprising a designating unit for designating the clip screen pasted on the album, wherein the reproducing unit reproduces the scene file corresponding to the clip screen designated by the designating unit. The image processing apparatus according to claim 11, wherein: 15. The image according to claim 10, further comprising encoding means for encoding the image data, wherein the recording means records the image data encoded by the encoding means. Processing equipment. 16. The image processing apparatus according to claim 15, further comprising decoding means for decoding image data encoded by said encoding means. 17. The image processing apparatus according to claim 1, further comprising an album editing unit that edits the album. 18. The image processing apparatus according to claim 17, wherein the album editing unit deletes, enlarges, reduces, transforms, or moves the clip screen pasted on the album.
An image processing apparatus according to claim 1. 19. The image processing apparatus according to claim 17, wherein said album editing means adds characters to said album. 20. The image processing apparatus according to claim 1, further comprising a scene editing unit for editing the scene. 21. The image processing apparatus according to claim 20, wherein the scene editing unit deletes some frames constituting the scene. 22. The image processing apparatus according to claim 20, wherein the scene editing unit sets a part of frames constituting the scene as a new scene. 23. The image processing apparatus according to claim 20, wherein the scene editing unit inserts a predetermined frame into the scene. 24. The image processing apparatus according to claim 20, wherein the scene editing unit replaces some frames of the scene with other frames. 25. The image processing apparatus according to claim 1, further comprising a photoelectric conversion unit that photoelectrically converts incident light and outputs a signal of the moving image. 26. An image processing method for processing a moving image, comprising: detecting a group of scenes from an input moving image; generating a clip screen as a still image representing the scene; An image processing method characterized by generating a virtual album on which an image is pasted. 27. An image reproducing apparatus for reproducing a moving image, comprising: detecting a set of scenes from an input moving image; generating a clip screen as a still image representing the scene; An image reproducing apparatus, comprising: a scene file which is a file of image data on the scene obtained by generating a virtual album to which the image is pasted; and reproducing means for reproducing the album. 28. The image reproducing apparatus according to claim 27, wherein said reproducing means reproduces the scene files corresponding to the clip screen pasted on the album in a predetermined order. 29. Designating means for designating the clip screen pasted on the album, wherein the reproducing means reproduces the scene file corresponding to the clip screen specified by the designating means. The image reproducing device according to claim 27, wherein the image reproducing device is an image reproducing device. 30. An image reproducing method for reproducing a moving image, comprising: detecting a set of scenes from an input moving image; generating a clip screen as a still image representing the scene; And reproducing the album and a scene file, which is a file of image data of the scene obtained by generating a virtual album to which the image is pasted. 31. A recording medium on which data about a moving image is recorded, wherein a set of scenes is detected from the input moving image, and a clip screen which is a still image representing the scene is generated. A recording medium, wherein a scene file, which is a file of image data of the scene obtained by generating a virtual album to which the clip screen is pasted, and the album are recorded.