JP6268287B2 - Moving image generating apparatus, moving image generating method, and program - Google Patents

Description

  The present invention relates to a moving image generating apparatus, a moving image generating method, a program, and an information storage medium storing the program, which generate a moving image showing how a character object moves according to music.

  Conventionally, techniques for analyzing the characteristics of music by detecting the pitch, intensity, type of beat, and the like of sounds included in music have been studied (for example, Patent Document 1 and Patent Document 2).

JP-A-2005-274708 JP 2008-102405 A

  One of the methods for allowing users to enjoy music is to display a moving image that shows how a character object that imitates a person or animal moves during the music playback. . In the past, such a moving image was manually created by the creator, which was very time-consuming.

  The present invention has been made in consideration of the above situation, and one of its purposes is a moving image generating apparatus capable of easily generating a moving image showing a state in which a character object moves in accordance with music, To provide a moving image generation method, a program, and an information storage medium storing the program.

  The moving image generating apparatus according to the present invention uses an acquisition unit that acquires timing information indicating the timing of sounds constituting the target music, and the timing information according to the timing of the character object according to the timing of the sounds. And a generation unit that generates a moving image showing a state of operation.

  In the moving image generation method according to the present invention, the step of acquiring timing information indicating the timing of the sound constituting the target music, and the state where the character object operates according to the timing of the sound using the timing information. Generating a moving image indicating the above.

  The program according to the present invention includes a step of obtaining timing information indicating the timing of sounds constituting the target music, and a moving image showing how the character object operates in accordance with the timing of the sounds using the timing information. A program for causing a computer to execute an image generating step. This program may be stored in a computer-readable information storage medium.

1 is a configuration block diagram of a moving image generation device according to an embodiment of the present invention. It is a functional block diagram of the moving image generation device according to the embodiment of the present invention. It is a figure which shows an example of a mode that a character object performs separate operation | movement. It is a figure which shows an example of the relationship between a music and the production | generation pattern of a moving image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a configuration block diagram of a moving image generation apparatus 1 according to an embodiment of the present invention. The moving image generating apparatus 1 is an information processing apparatus such as a personal computer or a home game machine, and includes a control unit 11 and a storage unit 12 as shown in FIG. The moving image generating apparatus 1 is connected to the input unit 13, the display unit 14, and the audio output unit 15.

  The control unit 11 is a CPU or the like, and performs various types of information processing according to programs stored in the storage unit 12. The storage unit 12 includes a memory element such as a RAM, and stores a program executed by the control unit 11 and data to be processed by the program.

  The input unit 13 receives an operation input from the user of the moving image generation apparatus 1. The input unit 13 may be a keyboard or a mouse, for example. Moreover, a touch panel, an operation button, etc. may be included. An operation signal indicating the content of the operation input by the user to the input unit 13 is transmitted to the control unit 11. The display unit 14 is, for example, a liquid crystal display panel or the like, and displays a moving image generated by the moving image generation device 1 on the screen under the control of the control unit 11. The sound output unit 15 is a speaker, a headphone, or the like, and sounds a sound according to the control of the control unit 11.

  The moving image generating apparatus 1 generates a moving image that shows how a character object moves in accordance with the music M as a target music (hereinafter referred to as music M). Hereinafter, functions realized by the moving image generation apparatus 1 in the present embodiment will be described. As shown in FIG. 2, the moving image generation device 1 is functionally configured to include a feature data acquisition unit 21, a moving image generation unit 22, and a moving image reproduction unit 23. These functions are realized when the control unit 11 executes a program stored in the storage unit 12. This program may be provided to the moving image generation apparatus 1 via a communication network such as the Internet, or may be provided by being stored in various computer-readable information storage media such as an optical disk.

  The feature data acquisition unit 21 acquires data indicating the features of the music M (hereinafter referred to as feature data). In particular, in the present embodiment, the feature data includes ringing timing information indicating the timing at which each of a plurality of sounds constituting the music M is played. For example, the ringing timing information may be information indicating the elapsed time from the start of the performance of the music M to the timing at which the corresponding sound rings. With this ringing timing information, the timing at which the sound is played in the music M can be specified. In addition to the ringing timing information, the feature data includes, for each of a plurality of sounds constituting the music M, beat type information indicating the pitch of the sound, the intensity (level) of the sound, and the type of beat (beat), etc. May be included. Here, the beat type information is information indicating the relationship between the target sound and the beat of the music M, and whether the sound matches the basic beat of the music M (hereinafter referred to as a beat sound), or This indicates whether the sound does not match the basic beat such as the back beat (hereinafter referred to as non-beat sound). Furthermore, when the sound is a beat sound, the beat type information includes information indicating whether the sound is a strong beat (beat such as the beginning of a measure) or a weak beat (beat such as the second beat of the measure). Information indicating detailed beat types may be included. In addition, the feature data includes not only information related to the individual sounds constituting the music M, but also information related to the key of the music M (major or minor) and time signature (3 time, 4 time, 2 time, etc.), 1 bar length Information on the entire music M such as sheath and tempo information may be included. Information indicating the start timing of each measure may also be included.

  This feature data can be generated, for example, by performing frequency analysis on audio data of music M recorded in a format such as PCM. Methods for specifying the pitch, intensity, timing, beat type, and the like of a sound included in music M by frequency analysis are disclosed in, for example, Patent Document 1, Patent Document 2, and the like. The feature data acquisition unit 21 may generate the feature data by performing such frequency analysis on the sound data of the music M itself. Alternatively, feature data generated by another information processing apparatus may be received from the outside via a communication network or the like. The feature data may be obtained by extracting music M audio data described in a format such as MIDI, music M score data, or the like. Here, it is assumed that the feature data acquisition unit 21 acquires feature data of the entire music M before generation of a moving image described later is started. Hereinafter, the process of the procedure in which the feature data acquisition unit 21 acquires the feature data of the entire music M before starting the generation of the moving image A in this way is referred to as static processing.

  The moving image generation unit 22 uses the feature data acquired by the feature data acquisition unit 21 to generate a moving image (hereinafter, referred to as a moving image A) that shows how the character object moves along with the music M. The character object is an object representing a character imitating a person or an animal, and may be a 2D object or a 3D object. The character object is composed of a plurality of parts connected to each other, and each part operates within a range that satisfies a predetermined restriction condition with respect to each other's positional relationship. As a specific example, in the following, a character object is a humanoid object and is composed of parts such as a head, a torso, arms, and legs. The object data of the character object is stored in the storage unit 12 in advance. In addition, the moving image generation unit 22 may paste a user's face image captured by a camera (not shown) connected to the moving image generation device 1 on the head part of the character object. The moving image generating unit 22 generates the moving image A by moving at least some parts (for example, arms and legs) of the character object in accordance with the sounding timing of the sound included in the music M. Thereby, for example, a moving image A in which a person is dancing to the music M can be generated. A specific example of the method in which the moving image generation unit 22 generates the moving image A will be described later.

  The moving image reproduction unit 23 reproduces the moving image A generated by the moving image generation unit 22 in accordance with the music M. Specifically, the moving image reproduction unit 23 separately acquires audio data of music M and causes the audio output unit 15 to reproduce the audio. Then, while synchronizing with the reproduction of the music M, the moving image A generated by the moving image generating unit 22 is displayed on the screen of the display unit 14. As a result, the user can view how the character object in the moving image A moves according to the music M. For example, the moving image reproduction unit 23 may reproduce the moving image A together with the music M as a karaoke image or a music M promotion image.

  Hereinafter, some specific examples of the method in which the moving image generation unit 22 generates the moving image A will be described. First, as a first example, a method for generating a moving image A using a predefined routine will be described. In this example, a standard action constituted by a series of movements of the character object is defined in advance. This standard operation is assumed to be executed in accordance with a phrase composed of one or more bars of the music. However, since the standard action is created in advance regardless of the music M, the movement does not always match the rhythm of the music M. Therefore, the moving image generation unit 22 adjusts the reproduction speed of the standard operation so that the reproduction time of the standard operation matches the time determined according to the ringing timing information.

  More specifically, the moving image generation unit 22 uses the ringing timing information included in the feature data, and the length (required time) of the unit phrase of music M (a unit of music M composed of one or more bars). ). Then, the playback speed of the standard operation is adjusted to be a part of the moving image A so that the time required for one standard operation matches the length of the unit phrase. Further, the start timing of the standard operation in the moving image A is determined so as to coincide with the start timing of the unit phrase in the music M. In this way, it is possible to generate a moving image A that shows how the character object performs a standard action in accordance with the reproduction of the unit phrase.

  A plurality of types of routine operations may be prepared. In this case, the moving image generation unit 22 operates the character objects continuously by combining the standard operations selected according to the given conditions. Moreover, it is good also as preparing several types of fixed operation | movement according to the kind of time signature, such as 3 time signature and 4 time signature. In this case, the moving image generation unit 22 determines the type of routine operation used for the moving image A in accordance with the type of the time signature of the music M included in the feature data.

  Next, as a second example, a method for moving a character object in accordance with individual sounds constituting music M will be described. In this example, the parts constituting the character object are moved at a timing corresponding to the timing at which the sound is produced, in accordance with all or part of the plurality of sounds constituting the music M. For example, the moving image generation unit 22 refers to the beat type information included in the feature data, and extracts a sound of a predetermined beat type (for example, a beat sound). Then, the moving image A is generated so that a specific part constituting the character object is moved at a timing corresponding to the timing at which the extracted sound is generated in the music M. Note that the moving image playback unit 22 moves the character object in accordance with timings such as a strong beat, a weak beat, or a back beat identified from the timing at which the sound is generated, not only at the timing at which the sound is played. Also good. In the following, the action that the character object performs at the timing according to the timing of each sound in the second example is referred to as an individual action.

  When causing the character object to execute the individual motion, the moving image generation unit 22 may determine the content of the individual motion according to the pitch and intensity of the corresponding sound, the beat type, and the like. As a specific example, when moving the arm of the character object at a timing according to the timing of the sound in the music M, the moving image generation unit 22 changes the magnitude of the movement of the arm according to the strength of the sound. May be. Further, for example, the moving image generation unit 22 may move different character object parts depending on the beat type, such as an arm at a timing when a beat sound is sounded and a wrist at a timing when a non-beat sound is sounded. .

  FIG. 3 is a diagram showing an example of the individual operation in the second example. In this figure, an example of moving the right arm of the character object in accordance with the beat sound is shown as a specific example of the individual action. In the example of this figure, the music M is assumed to be a 4-beat music piece. In the figure, P0 indicates the reference position of the right hand (the tip portion of the right arm), and P1 to P4 indicate the position of the right hand (target position) at the timing from the first beat to the fourth beat, respectively, within one measure. ing. The right hand of the character object starts from the reference position P0, reaches the target position P1 at the timing of the first beat, the target position P2 at the timing of the second beat, the target position P3 at the timing of the third beat, and the fourth beat To the target position P4. As a result, the character object performs an operation of rotating the arm in accordance with the music M in the moving image A.

  Note that the position of the right hand in the above-described example may be changed according to the strength (level) of each beat sound. In this case, for example, when the beat sound level is the maximum value, the right hand is moved to the target positions P1 to P4. When the beat sound level is the minimum value (0), the right hand is not moved from the reference position P0. I will do it. When the level of the beat sound is between the minimum value and the maximum value, the higher the sound level is on the straight line connecting the reference position P0 and the target positions P1 to P4, the higher the target positions P1 to P4. The right hand is moved so that the smaller the position is, the closer the position is to the reference position P0. Thereby, the louder the sound, the larger the character object can be operated. In this way, when moving a character object according to the timing of a certain sound, by moving the parts that make up the character object by an amount corresponding to the sound level, natural motion expression that matches the sound level It can be performed.

  The first example and the second example described above may be used in combination. For example, a part corresponding to the first one bar of music M is generated by the method of the first example, and a part corresponding to the next bar is generated by the method of the second example. Alternatively, the moving image A corresponding to the music M may be generated by another method. FIG. 4 shows the transition along the time series of the moving image A generated by such a method. In FIG. 4, the horizontal axis indicates the passage of time, the arrow extending upward from the horizontal axis indicates the timing at which the beat sound of the music M sounds, and the arrow extending downward indicates the timing at which the non-beat sound sounds. ing. In the example of this figure, in the time corresponding to the first measure of the music M, the character object executes one standard action with the time adjusted. In addition, at the time corresponding to the second measure, the character object executes an individual action in accordance with each sound included in the measure. The white circles and black circles in the figure indicate the execution timing of individual movements that match the sound of music M. The white circles indicate movements that match the beat sound (for example, movements of the shoulders and elbows), and the black circles match the non-beat sounds. It is shown that the movement (for example, wrist movement) is performed.

  The moving image generation unit 22 may generate a moving image corresponding to one unit phrase by a method combining the first example and the second example. As a specific example, the moving image generating unit 22 causes a character object to perform a fixed movement in accordance with a unit phrase, and a moving image to perform individual movements according to individual sounds (particularly non-beat sounds) between the fixed movements. Is generated. Alternatively, a moving image that causes a part of the character object (for example, a foot or a body) to move in accordance with a standard movement, while the other part (for example, an arm) performs a movement determined by the second example. A may be generated.

  In the above description, the moving image generation unit 22 generates the moving image A that shows how a single character object operates. However, the present invention is not limited to this, and the moving image A may show a state in which a plurality of character objects move according to the music M. In this case, the character objects may perform the same action with each other, or may perform different actions determined by different methods. The moving image A may include other objects that operate asynchronously with the music M in addition to the character object that operates in accordance with the music M.

  Furthermore, the objects included in the moving image A may interact with each other. As a specific example, when two character objects arranged in a virtual space are performing movements that match music M independently of each other, the parts of each character object (for example, each other's parts) Hand) may approach within a predetermined distance. When such an event occurs, for example, two character objects may hold hands and perform a predetermined action for a certain period of time, and then resume the action in accordance with the music M. In addition, the character object may perform an operation such as grabbing or throwing another object moving in the virtual space.

  Further, the moving image generation unit 22 may generate the moving image A based on a user instruction. In this case, for example, for each unit phrase constituting the music M, the user selects a standard action that is actually executed by the character object from among a plurality of standard action candidates, or is specific to the sound included in the music M. The moving image A is generated by designating the contents of individual individual operations. In this case as well, the user only needs to specify the contents of the routine action or the individual action, and the timing for executing these actions is determined by the moving image generation unit 22 based on the ringing timing information included in the feature data. Thereby, compared with the case where a user designates the timing of each operation | movement, the user can create the moving image A according to the music M easily. For this reason, for example, presenting a self-made moving image A to other users, or comparing a plurality of self-made moving images A according to the same music M, and performing a contest to compete for the result. It can be easily realized.

  Furthermore, the moving image generation unit 22 receives an operation input to the input unit 13 by the user during the reproduction of the moving image A by the moving image reproduction unit 23, and changes the content of the moving image A according to the content of the operation input. Also good. In this case, the moving image generation unit 22 determines the action of the character object in real time during the reproduction of the moving image A by the moving image reproduction unit 23 and reflects it in the subsequent reproduction of the moving image A.

  In the above description, the moving image generating apparatus 1 executes the static process of generating the moving image A after acquiring the feature data of the entire music M in advance. However, the present invention is not limited to this, and the moving image generation apparatus 1 may generate the moving image A while acquiring feature data in real time during the reproduction of the music M. Hereinafter, the processing of such a procedure is referred to as real-time processing. A specific example of real-time processing will be described below. In this processing example, the moving image generating apparatus 1 is connected to a sound collecting device (not shown) such as a microphone, and generates a moving image A while recording music M reproduced by an external sound source with the sound collecting device. I will do it.

  In this processing example, the feature data acquisition unit 21 performs analysis processing such as frequency analysis on the sound data recorded by the sound collecting device in real time to identify the timing of the sound included in the music M. Further, the feature data acquisition unit 21 estimates information related to the periodically repeated rhythm cycle of the music M, such as the time signature of the music M, the length of the measure, and the timing of the beginning of the measure, from the timing of the specified sound. Also good. By performing such a rhythm cycle estimation, the beat type of a newly detected sound can be specified. As a specific example, it is assumed that the sound detected at time t1 is a beat sound (strong beat) at the beginning of a measure and the length of one measure is estimated as Tb from the timing of the sound detected so far. . In this case, when the feature data acquisition unit 21 detects that a new sound has been produced at time (t1 + Tb) thereafter, the feature data acquisition unit 21 determines that the sound is also the first beat sound of the measure.

  If the feature data acquisition unit 21 can detect the sound ringing timing in this way, the moving image generation unit 22 performs the character object in accordance with the sound timing by various methods as in the case of the static processing described above. It is possible to generate a moving image A showing how the camera operates. Specifically, for example, when sound generation is detected, the moving image generation unit 22 generates a moving image A that causes the character object to execute a predetermined individual operation immediately in accordance with the timing. At this time, if a series of processes from detection of sound to determination of the action content of the character object and display of the moving image A for performing the determined individual action on the display unit 14 are performed in a sufficiently short time, Even if the character object moves with a slight delay from the generation of the sound, it appears to the user's eyes that the character object is moving according to the sound in the music M.

  In addition, if the rhythm cycle is estimated as described above, the moving image generation unit 22 can cause the character object to perform a standard action in accordance with the unit phrase. Specifically, the moving image generation unit 22 uses one or n rhythm cycles (n is a predetermined natural number) detected by the feature data acquisition unit 21 as a unit phrase so as to match the length of the unit phrase. Causes the character object to perform a standard action with adjusted speed. Thereby, similarly to the example of the static processing described above, it is possible to generate a moving image A that shows a state in which the character object performs a standard action according to the unit phrase.

  In this real-time processing, the user may instruct the beat timing of the music M. Specifically, the user may instruct the beat timing of the music M by, for example, clapping. By collecting this clapping sound with a sound collector and detecting it by frequency analysis, the feature data acquisition unit 21 can more accurately calculate the rhythm cycle of the music M than when using only the music M sound. Can be estimated. In addition, the user may explicitly perform an operation input for instructing the beat timing to the input unit 14. In this case, the user performs an operation input to the operation buttons and the touch sensor in accordance with the beat timing of the music M being reproduced. The feature data acquisition unit 21 specifies the timing of this operation input as the beat timing of the music M. Thereby, the rhythm cycle of the music M can be specified with high accuracy.

  The embodiments of the present invention are not limited to those described above. For example, in the above description, the moving image A generated by the moving image generating device 1 is reproduced by itself. However, the present invention is not limited to this, and other moving devices can be distributed by distributing the moving image A via a communication network. It is good also as making it reproduce. Also in this case, the state in which the character object operates in accordance with the music M can be reproduced by reproducing the moving image A in synchronization with the music M on the reproduction apparatus side. In this case, the moving image generating apparatus 1 may generate moving image data by synthesizing the moving image A data and the audio data of the music M, and may be a distribution target to the outside. Further, the data of the moving image A and the audio data of the music M obtained individually on the playback device side may be played back synchronously.

Claims (7)

Based on the sound of the user's clapping collected by the sound collecting device, a specifying unit that specifies the timing of the sound that constitutes the target music;
An estimation unit for estimating a rhythm cycle of the music from the identified timing;
A character whose movement is adjusted to match the length of the unit phrase with one or n rhythm cycles as a unit phrase at the timing when a new sound is detected based on the rhythm cycle. A generation unit that generates a moving image to be executed by the object;
A moving image generating apparatus comprising: The moving image generating apparatus according to claim 1,
The character object is composed of a plurality of parts,
The generating unit generates the moving image by moving any of the plurality of parts. The moving image generating device according to claim 1 or 2,
The generating unit generates a moving image in which the character object executes a predetermined motion that is determined in advance according to the unit phrase. In the moving image generation device according to claim 2 or 3,
The estimation unit estimates the sound intensity together,
The generation unit generates the moving image by moving any one of the plurality of parts by an amount corresponding to the intensity of the sound. In the moving image generation device according to any one of claims 1 to 4,
The specifying unit also specifies the beat type of the sound,
The moving image generating apparatus, wherein the generating unit changes a content of an action according to the rhythm cycle to be executed by the character object according to a beat type of the sound. Identifying the timing of the sound constituting the target music based on the clapping sound of the user collected by the sound collecting device ;
Estimating the rhythm cycle of the music from the identified timing;
A character whose movement is adjusted to match the length of the unit phrase with one or n rhythm cycles as a unit phrase at the timing when a new sound is detected based on the rhythm cycle. Generating a moving image to be executed by the object;
A moving image generating method comprising: Identifying the timing of the sound constituting the target music based on the clapping sound of the user collected by the sound collecting device ;
Estimating the rhythm cycle of the music from the identified timing;
A character whose movement is adjusted to match the length of the unit phrase with one or n rhythm cycles as a unit phrase at the timing when a new sound is detected based on the rhythm cycle. Generating a moving image to be executed by the object;
A program that causes a computer to execute.

Images