JP2000148107A - Image processing device and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device or the like capable of reproduction of an image in musical rhythm. SOLUTION: This device is equipped with a music input terminal 1 for inputting musical data, a recording medium 8 for recording the musical data or image data, a second CPU 22 for reading the musical data and extracting a low-frequency component and a pattern and generating timing information based on the rhythm information included in these data, a TFT panel 13 for displaying the generated timing information, and a first CPU 21 capable of producing an image display sequence by watching the timing information, for recording the sequence or the timing information in the recording medium 8, and for controlling changeover from one image to another image so as to be synchronized with the timing information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and a recording medium, and more particularly, to an image processing apparatus and a recording medium for processing image data in association with music data.

[0002]

2. Description of the Related Art Conventionally, there has been proposed an electronic album in which a plurality of image data are appropriately combined and displayed. In such an electronic album, sound data is combined with image data displaying music data while displaying image data. There is something to play as.

In such a conventional electronic album,
When displaying a combination of music and images, the phrase "play this music when displaying this image"
The timing of giving a presentation such as "displaying a plurality of images sequentially while playing this song" has been realized by the operator manually determining and giving an instruction for all.

[0004] By the way, music always has a rhythm including elements such as time signature and tempo, and human beings have a feeling that they may feel an aesthetic appearance in an operation that matches this rhythm.

In Japanese Patent Application Laid-Open No. 10-143151, a user draws a figure using an input device such as a mouse, and detects a tempo or a beat from the movement of the figure to give a voice or image expression. A command device is described.

[0006]

However, in the above-mentioned conventional electronic album, the operator determines the reproduction timing of music and images.
Images are not always played at the timing that matches the rhythm of the music, and if there is a discrepancy in the timing, it may cause discomfort, and it is desired that improvements be made so that more appropriate playback can be performed. I have.

[0007] The present invention has been made in view of the above circumstances, and has as its object to provide an image processing apparatus and a recording medium that can reproduce an image that matches the rhythm of music.

[0008]

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention comprises a music input unit for inputting music data and a rhythm information included in the input music data. And timing control means for generating timing information, image input means for inputting image data, and control means for changing a display based on the input image data in synchronization with the timing information.

Further, an image processing apparatus according to a second aspect of the present invention
In the image processing apparatus according to the first aspect, the timing generating means analyzes a frequency of the input music data to extract a low frequency component, and converts the timing information based on rhythm information included in the low frequency component. To generate.

Further, in the image processing apparatus according to a third aspect of the present invention, in the image processing apparatus according to the first aspect, the timing generation means extracts a pattern by analyzing a pattern of the input music data, and The timing information is generated based on the included rhythm information.

[0011] An image processing apparatus according to a fourth aspect of the present invention is the image processing apparatus according to the first aspect, further comprising recording means for recording the timing information generated by the timing generating means.

According to a fifth aspect of the present invention, in the image processing apparatus according to the first aspect, the control means changes the display by switching a display based on one image data to a display based on another image data. It is to let.

An image processing apparatus according to a sixth aspect of the present invention is the image processing apparatus according to the first aspect, wherein the image processing for processing the image data so as to substantially continuously change the display correlation based on the image data. Means for controlling a continuous change of the display based on the image data in synchronization with the timing information to change the display based on the input image data.

According to a seventh aspect of the present invention, in the image processing apparatus according to the first aspect, the control means can control interval reproduction for switching display based on image data at appropriate time intervals, The display by the interval reproduction is changed in synchronization with the timing information.

An image processing apparatus according to an eighth aspect of the present invention is the image processing apparatus according to the first aspect, further comprising an image display device for visually displaying the timing information generated by the timing generating means. .

The image processing apparatus according to a ninth aspect of the present invention is the image processing apparatus according to the eighth aspect, wherein the sequence for changing the display based on the image data executed by the control means includes the timing information visually displayed. And a recording means for recording the sequence created by the sequence creating means.

A recording medium according to a tenth aspect of the present invention includes a step of inputting music data, a step of generating timing information based on rhythm information included in the input music data, a step of inputting image data, Changing the display based on the input image data in synchronization with the timing information.

[0018]

Embodiments of the present invention will be described below with reference to the drawings. FIGS. 1 to 5 show an embodiment of the present invention. FIG. 1 is a block diagram mainly showing an electrical configuration of an electronic album. FIG. 2 is a timing chart showing timing information obtained by analyzing music data. FIG. 3 is a flowchart showing a rhythm extraction process, FIG. 4 is a flowchart showing a link information input process for associating image data with timing information, and FIG. 5 is a timing chart showing music to be reproduced based on the link information. 9 is a flowchart showing a process of displaying an image in accordance with.

In this embodiment, the image processing apparatus of the present invention is applied to an electronic album.

As shown in FIG. 1, this electronic album is
For example, a music input terminal 1 serving as music input means for inputting music output from a microphone or other music playback device, and music for converting an analog electric signal input from the music input terminal 1 into digital data. A / D
Circuit 2 and a FIFO for temporarily buffering music data output from the music A / D circuit 2
A memory 3; a FIFO memory 4 for storing music data output via the bus; and a FIFO memory 4
A music D / A circuit 5 for converting digital music data output from the
A / Speaker 6 for playing back music based on the output of circuit 5
And a recording means connected to a second CPU 22 to be described later via a bus, for recording the memory 7 as a recording medium, and for recording various data including image data and music data. A recording medium 8 serving as a recording means including a detachable memory card or the like;
A recording medium interface (I / F) 9 serving as music input means for transmitting and receiving data to and from the recording medium 8 and also serving as an image input means, and for storing image data output via the bus. FIFO memory 10 and this F
A video encoding circuit 11 for performing various processes on image data output from the IFO memory 10, a TFT liquid crystal driving circuit 12 for controlling a TFT panel 13 based on the output of the video encoding circuit 11,
A TFT panel 13 as an image display device for visually displaying an image based on the control of the FT liquid crystal drive circuit 12;
A backlight unit 14 for illuminating the T panel 13 from the back side, a video output circuit 15 for generating a video output signal based on the output of the video encoding circuit 11, and a video generated by the video output circuit 15. A video output terminal 16 for outputting an output signal to an external device or the like, and the above-mentioned FIFO memories 3, 4, 10, and memory 7, and a recording medium I / F 9 are connected via the above-mentioned bus to provide a music reproducing function. A second CPU 22 which is a timing generation unit having various image processing functions, an image decompression function, a music (rhythm) analysis function, a memory control function, a recording medium access function and the like and an image processing unit, and a control unit for controlling an LCD panel 18 described later. LCD display circuit 1
7, an LCD panel 18 driven by the LCD display circuit 17 to display various information related to the electronic album, and external data for exchanging data between the electronic album and an external electronic device or the like. An I / F 19, a key matrix 20 for a user to input information to the electronic album, and a battery 23 serving as a power supply of the electronic album
A power supply circuit 24 for supplying power supplied from the battery 23 to each circuit in the electronic album; a music D / A circuit 5, a video encoding circuit 11, a TFT liquid crystal driving circuit, and a backlight unit 14. , LCD display circuit 17, external data I / F 19, key matrix 20,
In addition to controlling the power supply circuit 24, the second CPU 22
And a first CPU 21 which is a sequence creating means and a control means having a system control function for performing system control and the like by controlling each function in a comprehensive manner.

Next, in such an electronic album,
A process of analyzing rhythm information including information such as tempo and beat from music data will be described with reference to FIG.

This rhythm extraction process is performed by the first CPU 2
The control is performed by the second CPU 22 under the control of 1.

The premise of performing this processing is that music data is input from the music input terminal 1 and the second CP is input.
It is assumed that one or more music data has already been recorded on the recording medium 8 under the control of U22.

When the rhythm extraction process is started, music data is read from the recording medium 8 via the recording medium I / F 9 (step S1).

Then, rhythm analysis processing of the read music data is performed (step S2). In this rhythm analysis, timing information is generated from rhythm information included in music data by performing frequency analysis and pattern analysis.

That is, music often has a specialized part for engraving a rhythm called a "rhythm part", and a specific instrument such as "drum" or "guitar" is assigned to this part. Many. Therefore, by performing frequency analysis, the constituent frequencies of the musical instrument are extracted from the music, and the rhythm can be extracted by extracting the low-frequency portion of the constituent frequencies. Note that a frequency of, for example, about 1 Hz to 10 Hz is often used as the rhythm.

In music, besides the rhythm, "bars"
There is a small break called. This is mainly because it can be recognized as a repetition of a similar phrase. Therefore, by pattern analysis (searching for a repetition of a similar pattern and recognizing the repetition timing as one delimiter), It can analyze music bars and thus rhythm.

By performing such a rhythm analysis,
It generates timing information including information on the timing of performing image switching and image processing.

It is determined whether or not the rhythm analysis performed in this manner is successful (step S3). If the rhythm analysis is successful, timing information obtained by analyzing the rhythm is recorded on, for example, the recording medium 8 (step S4). On the other hand, if it is determined that the rhythm analysis has failed, a display to that effect is made on the LCD panel 18, the TFT panel 13, or the like (step S5), and this processing ends.

Next, a sequence is created which shows how an image is displayed in association with such music data. The process related to link information input for executing this sequence will be described with reference to FIG. 2 and along FIG.

In this process, first, it is specified which music data among the music data recorded on the recording medium 8 is to be subjected to the creation of link information (step S11).

Then, for the specified music data,
The rhythm extracting process described with reference to FIG. 3 is performed, and rhythm analysis is performed to generate timing information (Step S).
12).

Then, as shown in FIG. 2, a display is made on the display screen 13a of the TFT panel 13 so as to be easily understood visually (step S13).

That is, a music data rhythm display field 34 for displaying rhythm information of music data is provided at, for example, an upper portion of the display screen 13a. For example, a horizontal axis is used as a time axis 35 for indicating time. The timing obtained as a result of the analysis is displayed on the time axis 35 as a plurality of vertical bar-shaped partitions 36.

Further, a plurality of image data to be displayed in combination with the music data are displayed on a lower portion of the display screen 13a, for example, a first image data display 31, a second image data display 32, and a third image data display. They are displayed side by side as a data display 33.

Then, the timing for displaying these image data is registered manually or automatically in relation to the timing information displayed in the music data rhythm display field 34 (step S14).

As a method at this time, in the case of performing the operation manually, for example, the image data for specifying the timing is moved by a pointer or the like, and linked by dropping the desired portion of the music data to be displayed. Can be.

When linking automatically,
In consideration of the length of the music and the number of images to be displayed, calculation is performed so that an appropriate interval is generated between the images to be displayed, and the image data and the music data are linked.

At this time, the image data is linked so as to match any one of the quantized and discontinuous timings, and a link is established at an indeterminate portion between the timings. Never.

It is determined whether or not such registration has been completed (step S15). If there is still unregistered image data, the process returns to step S14 to perform registration, and all images to be displayed are displayed. When the data has been linked to the music data, link information for linking a series of image data and music data is stored in the recording medium 8 or the like (step S16), and the process ends.

FIG. 5 shows a process of displaying an image at the same time as reproducing music data in accordance with the sequence based on the link information as described above.
This will be described with reference to FIG.

First, the link information recorded on the recording medium 8 is read (step S21), and based on the read link information, whether or not music is to be reproduced or music data to be reproduced is changed. It is determined whether or not this is the case (step S22).

Here, when performing reproduction or change, the user selects which music data of the music data recorded on the recording medium 8 is reproduced along the link information, and reproduces or reproduces the music. If a change process is performed (step S23) and no reproduction or change is performed, the process of step S23 is skipped and the process proceeds to the next process.

Further, based on the read link information,
It is determined whether or not to reproduce the image, or whether to change the display based on the image data to be reproduced (step S24).

Here, when performing reproduction or change, it is necessary to select which image data of the image data recorded on the recording medium 8 to reproduce along the link information, and to reproduce or change the image. A change process is performed (step S25).

In the process of reproducing the image and changing the display based on the image data, for example, the following process is performed.

That is, for example, by switching from image data representing one scene to image data representing another scene, an image displayed based on the image data is switched and changed.

Alternatively, there is an example in which image processing is performed by changing the display correlation based on image data almost continuously.

As an example of such a process, when an image is switched, such as a fade-out or a fade-in, a process for giving a visual characteristic by performing a special process and gradually performing the switching is performed. No.

Another example is a case where the image is continuously zoomed in a direction of enlarging or reducing an image,
Examples include a case where an image is rotated in a screen, and a case where an image is moved in a screen.

On the other hand, if the reproduction or change of the image data is not performed in step S24, the process in step S25 is skipped and the process proceeds to the next process.

Then, it is determined whether or not to perform a wait process for waiting for the next process (step S26). If it is determined that the wait process is to be performed, the wait process is started (step S2).
7) If not performed, skip the process of step S27 and proceed to the next process.

By referring to the link information in this way, an image is displayed at a timing that matches the rhythm included in the music.

Then, it is determined whether or not the reproduction of a series of music data and image data has been completed (step S28). If not completed, the process returns to step S22 to repeat the above-described processing. If the reproduction has been performed up to this point, the processing ends.

When the reproduction is performed by such a process, if the link information is automatically generated as described above, so-called interval reproduction is performed.

The image processing program as shown in FIGS. 3, 4 and 5 is stored in a memory 7 as a recording medium.
The image processing program is stored in a ROM or the like provided in the electronic album, and is loaded into the memory 7 at the time of execution.
The ROM is also a recording medium for recording the image processing program. Further, the recording medium for recording the image processing program may be the recording medium 8 or another recording medium.

According to such an embodiment, image switching processing, fade-in,
In order to perform a process such as fade-out, it is possible to reproduce an image that matches the rhythm of music. Then, the music can be automatically matched to the rhythm of the music even in the interval reproduction.

Further, low-frequency components and patterns are extracted by frequency analysis and pattern analysis of music data,
The rhythm information contained therein can be easily obtained.

Further, by recording the generated timing information on a recording medium or the like, it is possible to cope with a case where real-time processing cannot be performed.

Further, since it is possible to create a sequence for displaying an image while visually confirming the timing information displayed on the TFT panel, there is an advantage that the operation is facilitated.

It should be noted that the present invention is not limited to the above-described embodiment, and it is needless to say that various modifications and applications are possible without departing from the gist of the invention.

[0062]

As described above, according to the image processing apparatus of the first aspect of the present invention, in order to change the display based on the image data in synchronization with the timing information, the reproduction of the image in accordance with the music rhythm is performed. Can be performed.

Further, according to the image processing apparatus of the present invention according to the second aspect, based on the rhythm information included in the low frequency component extracted by analyzing the frequency of the music data, the same as the first aspect of the present invention. The effect can be achieved.

Further, according to the image processing apparatus of the present invention, the music data is analyzed based on the rhythm information included in the extracted pattern.
The same effect as the invention described in (1) can be obtained.

According to the image processing apparatus of the present invention according to the fourth aspect, the same effects as those of the first aspect can be obtained, and real-time processing cannot be performed by recording the generated timing information. Can also be accommodated.

According to the fifth aspect of the present invention, the same effects as those of the first aspect can be obtained, and the switching from the display based on one image data to the display based on another image data can be achieved. It can be performed according to the rhythm of music.

According to the image processing apparatus of the present invention, the same effects as those of the first aspect can be obtained, and the continuous change of the display based on the image data can be adjusted in accordance with the rhythm of music. It is possible to do.

According to the image processing apparatus of the present invention according to the seventh aspect, the same effect as that of the first aspect can be obtained, and the interval reproduction can be performed according to the rhythm of music.

According to the image processing apparatus of the present invention, the same effects as those of the first aspect are obtained, and the timing information is visually displayed. Can be recognized.

According to the image processing apparatus of the ninth aspect of the present invention, the same effect as that of the eighth aspect of the present invention is obtained, and a sequence for changing the display while viewing the displayed timing information is easily created. can do.

According to the recording medium of the present invention, the display based on the image data can be changed in synchronization with the timing information by the recorded image processing program, and the reproduction of the image in accordance with the rhythm of music can be performed. Can be performed.

[Brief description of the drawings]

FIG. 1 is a block diagram mainly showing an electrical configuration of an electronic album according to an embodiment of the present invention.

FIG. 2 is a diagram showing an example in which timing information obtained by analyzing music data is displayed on a TFT panel in the embodiment.

FIG. 3 is a flowchart showing a rhythm extraction process in the embodiment.

FIG. 4 is a flowchart showing link information input processing for associating image data with timing information in the embodiment.

FIG. 5 is a flowchart showing a process of displaying an image in time with music to be reproduced based on link information in the embodiment.

[Explanation of symbols]

Reference Signs List 1 music input terminal (music input means) 6 speaker 7 memory (recording means, recording medium) 8 recording medium (recording means) 9 recording medium interface (music input means, image input means) 13 TFT panel ( Image display device) 21: First CPU (sequence creation unit, control unit) 22: Second CPU (timing generation unit, image processing unit)

Claims (10)

[Claims] 1. Music input means for inputting music data; timing generation means for generating timing information based on rhythm information included in the input music data; image input means for inputting image data; Control means for changing a display based on the input image data in synchronization with the information. 2. The apparatus according to claim 1, wherein said timing generation means analyzes a frequency of the input music data to extract a low frequency component, and generates said timing information based on rhythm information included in said low frequency component. The image processing apparatus according to claim 1, wherein: 3. The timing generating means extracts a pattern by analyzing a pattern of the input music data, and generates the timing information based on rhythm information included in the pattern. Claim 1
An image processing apparatus according to claim 1. 4. The image processing apparatus according to claim 1, further comprising recording means for recording timing information generated by said timing generation means. 5. The image processing apparatus according to claim 1, wherein the control unit changes the display by switching a display based on one image data to a display based on another image data. . 6. An image processing means for processing the image data so as to substantially continuously change the correlation of the display based on the image data, wherein the control means controls the continuous display based on the image data. The image processing apparatus according to claim 1, wherein a change based on the input image data is changed by controlling a temporal change in synchronization with the timing information. 7. The control means can control interval reproduction for switching display based on image data at appropriate time intervals, and changes the display by the interval reproduction in synchronization with the timing information. The image processing apparatus according to claim 1, wherein: 8. The image processing apparatus according to claim 1, further comprising an image display device that visually displays the timing information generated by the timing generation unit. 9. A sequence generating means for generating a sequence for changing a display based on image data, which is executed by the control means, according to the visually displayed timing information; The image processing apparatus according to claim 8, further comprising: recording means for recording the created sequence. 10. A step of inputting music data; a step of generating timing information based on rhythm information included in the input music data; a step of inputting image data; Changing a display based on the input image data; and recording an image processing program comprising: