JP2014109770A - Speech processing unit, speech recognition system, speech processing method, and speech processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a novel and improved speech processing unit capable of extracting speech sounds more vividly while reducing the influence of noise to the speech signal that changes corresponding to the fluctuation of vocal sound.SOLUTION: The present invention includes a detecting part which detects a period of a speaker's behavior, a specification part which specifies gain change along a time series based on the period of the detected behavior, and a signal processing part which adjusts the amplitude of an input signal including a speech signal in accordance with the time series based on the specified gain change.

Description

  The present invention relates to a voice processing device, a voice recognition system, a voice processing method, and a voice processing program.

  A speech recognition technique is widely used as a technique for determining the operation content and input characters by analyzing a voice uttered by a speaker (hereinafter simply referred to as “speaker”) acquired as an input.

JP 2004-163458 A

  In such voice recognition technology, it is necessary to correctly recognize the voice uttered by the speaker, but noise such as noise other than voice can be cited as a factor that hinders this. Patent Document 1 discloses an example of a technique for removing such noise.

  On the other hand, the sound emitted by a speaker and collected by a predetermined microphone has a unique fluctuation for each speaker. On the other hand, noise such as noise is not affected by the operation of the speaker. That is, the magnitude of the influence of noise on the audio signal may change according to the phase of the fluctuation. On the other hand, the influence of fluctuation for each speaker is also reflected in the rhythm of the movement of the speaker, such as the movement of the utterance such as the movement of the mouth, the movement of the head and body, the change of the facial expression, and the like.

  Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to influence the influence of noise on a voice signal that changes according to the fluctuation of the voice based on the operation of the speaker. It is an object of the present invention to provide a new and improved speech processing apparatus that can reduce speech and extract speech more clearly.

  In order to solve the above problems, according to an aspect of the present invention, a detection unit that detects a period of a speaker's behavior and a change in gain along a time series are identified based on the detected period of the behavior. And a signal processing unit that adjusts the amplitude of the input signal including the audio signal in time series based on the specified change in the gain. The

  With such a configuration, the fluctuation of the voice signal is estimated as the period of the speaker's behavior, and a difference occurs in the amplitude of the input signal between the desired timing and the other timing in the period of the speaker's behavior. Let Thereby, the input signal (and voice signal) at the desired timing is emphasized in accordance with the period of the speaker's behavior (and thus fluctuation of the voice signal).

  A synchronization operation unit that identifies a synchronization timing between the period of the speaker's behavior and the input signal, and the signal processing unit adjusts the amplitude of the input signal based on the identified synchronization timing May be determined.

  Thereby, it is possible to synchronize the input signal and the process related to the amplitude adjustment along the time series for the input signal in accordance with the synchronization timing.

  The synchronization calculation unit, based on the information indicating the operation of the speaker's utterance detected along the time series, and the change in the amplitude of the input signal along the time series, the period of the speaker's behavior and A shift amount along the time series with the input signal may be calculated, and the synchronization timing may be specified based on the shift amount.

  The synchronization calculation unit may use a difference between a timing at which the amplitude of the input signal increases by a predetermined amount or more and a timing at which the speech operation is started as the shift amount.

  The detection unit may detect the period of the behavior based on image information indicating a speaker's operation.

  The detection unit may detect an operation period of a predetermined part of each part of the speaker from the image information, and specify the period of the behavior based on the detected period.

  The detection unit detects the period of movement of a plurality of parts as the action of the predetermined part, and specifies a period of the behavior by applying a predetermined statistical process to each period of the plurality of parts. May be.

  With such a configuration, it is possible to more accurately specify the period of the speaker's behavior that changes according to the combination of movements of a plurality of parts.

  The detection unit may weight the periods of the plurality of parts as the statistical process, and specify the period of the behavior by taking an average of the weighted periods.

  Based on a Bayesian network created in advance based on a causal relationship between the period of each of the plurality of parts and the period of the behavior as the statistical process, the detection unit calculates the period from each of the detected parts of the plurality of parts. The period of the behavior may be specified.

  The specifying unit may specify the change in the gain so as to synchronize with the detected period of the behavior.

  The specifying unit may specify the gain change such that a peak position of the gain change deviates from a peak position of the behavior period by a predetermined time width.

  The specifying unit changes the gain so that the phase is shifted by a predetermined time width with respect to the period of the behavior so that the gain is amplified at a predetermined phase in the detected period of the behavior. You may specify.

  In order to solve the above-described problem, according to another aspect of the present invention, a sound collection unit that collects an input signal including an audio signal, an image acquisition unit that acquires a motion of a speaker as a moving image, A detection unit that detects a period of a speaker's behavior based on the moving image, a specification unit that specifies a change in gain along a time series based on the detected period of the behavior, and a change in the specified gain A speech recognition apparatus comprising: a signal processing unit that adjusts the amplitude of an input signal including a speech signal along a time series; and a speech recognition unit that performs speech recognition based on the input signal whose amplitude is adjusted. Provided.

  In order to solve the above problem, according to another aspect of the present invention, a detection step for detecting a period of a speaker's behavior, and a gain along a time series based on the detected period of the behavior. An audio processing method comprising: a specifying step for specifying a change; and a signal processing step for adjusting the amplitude of an input signal including the audio signal along a time series based on the specified change in the gain. Provided.

  In order to solve the above problem, according to another aspect of the present invention, a detection process for detecting a period of a speaker's behavior and a gain along a time series based on the detected period of the behavior. An audio processing program that executes a specifying process for specifying a change, and a signal process for adjusting the amplitude of an input signal including an audio signal along a time series based on the specified change in the gain. Provided.

  As described above, according to the present invention, it is possible to reduce the influence of noise on a voice signal, which changes according to the fluctuation of the voice, based on the action of the speaker, and to extract the voice more clearly. Become.

It is a conceptual diagram for demonstrating an example of the application scene of the speech recognition system which concerns on embodiment of this invention. It is the block diagram which showed the structure of the speech recognition system which concerns on the same embodiment. It is the block diagram which showed the structure of the audio | voice processing unit concerning the embodiment. It is a figure for demonstrating the change of the amplitude along the time series of each signal contained in an input signal. It is a figure for demonstrating the identification method of the period of a speaker's behavior. It is a figure for demonstrating the gain control based on the period of a speaker's behavior. It is a figure for demonstrating the synchronization with the process based on the cycle of an input signal and a speaker's behavior. It is the flowchart which showed a series of operation | movement of a speech processing unit. It is a figure for demonstrating the gain control in the one aspect | mode of the audio | voice processing unit which concerns on a modification. It is a figure for demonstrating the gain control in the one aspect | mode of the audio | voice processing unit which concerns on a modification.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  First, an example of an application scene of the speech recognition system according to the present embodiment will be described with reference to FIG. 1A. FIG. 1A is a conceptual diagram for explaining an example of an application scene of the speech recognition system according to this embodiment. As an example, FIG. 1A includes a main body M11 and a display M13. A PC (Personal Computer) including a microphone M12 that collects the voice of the user U1 as an input interface and a camera M131 that captures the appearance of the user U1. ).

  The speech recognition and stem according to the present embodiment is applied to a PC as shown in FIG. 1A, for example. This voice recognition system acquires the voice of a speaker (user U1) as an input signal at a sound collection unit such as a microphone M12, analyzes the input signal, and instructs the speaker's instructions (for example, operation contents and input characters). Etc.). The voice recognition system then executes a process corresponding to the determined instruction content, for example, an execution unit (for example, a CPU of the main body M11) via an OS (Operating System) installed in the main body M11. ). Hereinafter, the configuration of the voice recognition system will be described.

[Voice recognition system]
First, the configuration of the speech recognition system according to the present embodiment will be described with reference to FIG. 1B. FIG. 1B is a block diagram showing the configuration of the speech recognition system according to the present embodiment. As shown in FIG. 1B, this voice recognition system includes a sound collection unit 101, an image acquisition unit 102, a voice processing unit 11, a voice recognition unit 12, and an operation control unit 13. This voice recognition system corresponds to a “voice recognition device”.

  The sound collection unit 101 acquires a voice uttered by a speaker as an input signal. This input signal includes a voice signal indicating the voice uttered by the speaker and noise such as noise. A specific example of the sound collection unit 101 is the microphone M12 illustrated in FIG. 1A. The sound collection unit 101 outputs the acquired input signal InA to the sound processing unit 11.

  The image acquisition unit 102 captures the appearance of the speaker along a time series, and acquires a moving image showing the operation. A specific example of the image acquisition unit 102 is the camera M131 illustrated in FIG. 1A. The image acquisition unit 102 outputs the acquired moving image InB to the sound processing unit 11. The moving image InB corresponds to “image information indicating the operation of the speaker”.

  The sound processing unit 11 acquires the input signal InA from the sound collection unit 101. As described above, the input signal InA may include an audio signal. This voice signal has a unique fluctuation for each speaker. That is, the amplitude of the audio signal changes along the time series. The influence of this fluctuation is also reflected in the movement of the utterance such as the movement of the mouth and the movement of the speaker such as the movement of the neck and hand (hereinafter, these are collectively referred to as “the behavior of the speaker”). .

  Therefore, the voice processing unit 11 acquires the moving image InB from the image acquisition unit 102, analyzes the moving image InB, and specifies the behavior of the speaker. The speech processing unit 11 estimates the change in the amplitude of the speech signal along the time series based on the behavior of the identified speaker, and adjusts the amplitude of the input signal in accordance with the estimated change in the amplitude. Emphasize the audio signal in the input signal. The details of the operation principle, the configuration of the audio processing unit 11 and the contents of the processing will be described later.

  The voice processing unit 11 outputs the input signal Out whose amplitude is adjusted along the time series to the voice recognition unit 12. The voice processing unit 11 corresponds to a “voice processing apparatus”.

  The voice recognition unit 12 receives an input signal Out whose amplitude is adjusted along the time series from the voice processing unit 11. The voice recognition unit 12 performs voice recognition processing on the input signal Out, and specifies a sentence uttered as a voice by the speaker. In this voice recognition process, for example, the frequency component of the input signal is analyzed, and each character constituting the sentence may be specified based on the distribution of the frequency component. The voice recognition process is not limited to the above method as long as a sentence can be specified based on the input signal Out.

  The voice recognition unit 12 performs syntax analysis (for example, lexical analysis, semantic analysis, etc.) on the specified sentence, and recognizes the meaning of the sentence. The voice recognition unit 12 instructs the operation control unit 13 to perform processing indicated by the sentence based on the sentence whose meaning is recognized. The syntax analysis is not limited to the above method as long as the meaning of the sentence can be identified based on the input signal Out.

  In addition, since the amplitude of a part of audio signal in the input signal Ont is small, the character may not be specified. In that case, the voice recognition unit 12 estimates the character that could not be specified based on the other part of the character that could be specified by either or both of the voice recognition processing and the syntax analysis. It may be operated.

  The operation control unit 13 schematically shows a control unit that controls the operation of each unit (not shown) constituting the apparatus or system in which the speech recognition system is introduced. The operation control unit 13 receives an instruction to execute processing based on the input signal Out from the voice recognition unit 12. Based on this instruction, the operation control unit 13 controls the operation of each unit constituting this apparatus or system. A specific example of the operation control unit 13 is the CPU of the main body M11 illustrated in FIG. 1A.

[Audio processing unit 11]
Next, the operation principle, detailed configuration and processing contents of the audio processing unit 11 will be described.

(Operating principle)
First, with reference to FIG. 2, the mode of each signal included in the input signal and the principle of operation of the audio processing unit 11 based thereon will be described. FIG. 2 is a diagram for explaining the change in amplitude along the time series of each signal included in the input signal. The vertical axis in FIG. 2 indicates the amplitude A of each signal, and the horizontal axis indicates time t. 2 indicates an input signal, and the input signal f10 includes an audio signal f110 indicating the voice of the speaker and a noise f130 such as noise. The audio signal f110 includes a signal having a period due to, for example, a sentence, a clause, a phrase, a clause, a syllable, a phoneme, and the like. In general, the audio signal f110 and the noise f130 are superposed, but in the example of FIG. 2, the audio signal f110 and the noise f130 are separated and shown separately in a simulated manner.

  As described above, the voice signal f110 is fluctuated according to the speaker, and the influence of the fluctuation is the behavior of the utterance of the speaker (or the accompanying sentence, clause, phrase, phrase, etc.). This also appears in the speaker's behavior such as the movement of the neck and hand. That is, as shown in FIG. 2, it can be said that the amplitude of the audio signal f110 changes along the time series in a cycle corresponding to the behavior of the speaker. This period of fluctuation, that is, the period of change in amplitude along the time series of the audio signal is hereinafter referred to as period f20.

  On the other hand, since the noise f130 does not depend on the behavior of the speaker, unlike the audio signal f110, fluctuation according to the speaker does not occur. Therefore, when the amplitude of the audio signal f110 is small (at a small phase), the ratio of the amplitude of the noise f130 to the amplitude of the audio signal f110 is large, and the noise f130 becomes dominant. On the other hand, when the amplitude of the audio signal f110 increases (in a phase that increases), the ratio of the amplitude of the noise f130 to the amplitude of the audio signal f110 decreases, and the audio signal f110 becomes dominant compared to the former. .

  Therefore, the speech processing unit 11 according to this embodiment estimates the period f20 in which the amplitude of the speech signal changes along the time series based on the behavior of the speaker, and calculates the amplitude of the input signal based on the estimated period. Adjust along the series. This estimated period is hereinafter referred to as “speaker behavior period”.

  As a specific example, the speech processing unit 11 attenuates the amplitude of the input signal f10 as the amplitude of the speech signal f110 decreases based on the period of the speaker's behavior. By operating in this way, the audio processing unit 11 has the amplitude of the input signal f10 as the timing at which the ratio of the amplitude of the noise f130 to the amplitude of the audio signal f110 increases (that is, the noise f130 becomes dominant). Will be further attenuated. As a result, the timing at which the ratio of the amplitude of the noise f130 to the amplitude of the audio signal f110 is small (that is, the timing at which the audio signal f110 is dominant) is emphasized, and the accuracy of the speech recognition processing by the subsequent speech recognition unit 12 is improved. Let Note that the input signal f10 in FIG. 2 corresponds to the aforementioned input signal InA.

(Constitution)
Next, the configuration of the audio processing unit 11 will be described with reference to FIG. 1C. FIG. 1C is a block diagram showing the configuration of the audio processing unit 11. As shown in FIG. 1C, the audio processing unit 11 includes an operation detecting unit 111, a gain specifying unit 112, an audio signal acquiring unit 113, a delay processing unit 114, a synchronization calculating unit 115, and a signal processing unit 116. Consists of including. The signal processing unit 116 includes an amplifier 1161 and a gain control unit 1162 with good luck. Hereinafter, the operation of each of these units will be described.

  The motion detection unit 111 receives the moving image InB from the image acquisition unit 102 as an input. In this moving image InB, the appearance of the speaker is taken in time series, and the operation of the speaker is shown. The motion detection unit 111 performs image analysis processing on the moving image InB to identify the period of the speaker's behavior. A specific example of the operation will be described below.

  The motion detection unit 111 identifies positions of one or more predetermined parts among the parts constituting the speaker's body from the frame images constituting the moving image InB. In the following, the part where the position is specified may be particularly referred to as “target part”. Such specification of the position may be performed, for example, by extracting the shape feature of the target part and based on the position and orientation of the shape feature.

  Reference is now made to FIG. FIG. 3 is a diagram for explaining a method for specifying the period of the speaker's behavior, and schematically shows each part constituting the speaker U11. For example, in the example shown in this figure, a part U111 corresponding to “head” and a part U112 corresponding to “arm” are set as target parts. The position and orientation of the part U112 corresponding to “arm” is the position of the feature point P12 indicating the shape feature of this part, such as “joint” such as “elbow” or “shoulder”, “hand”, etc. And can be specified as orientation. Note that by extracting a plurality of feature points from each part, it is possible to improve the accuracy of specifying the position and orientation of the part. Similarly, the movement of the part U111 corresponding to the “head” can be specified as the position and orientation of the feature point P11 indicating the shape feature such as “eyes”, “nose”, “ear”, and the like. . In this way, the motion detection unit 111 identifies the position and orientation of each target part.

  Next, the motion detection unit 111 monitors the motion of each target part, that is, the change in the position of each target part for a predetermined number of frames, and specifies the cycle of the motion for each target part. . Therefore, it is desirable that the number of frames for monitoring related to the specification of this cycle is a sufficient period for specifying the movement cycle of the target part, and is determined in advance according to the type of the target part. In addition, in order to specify the period of each part, it is investigated beforehand by experiment etc. how many frames should be provided with the monitoring period. Hereinafter, the time width corresponding to the predetermined number of frames is set as h111. Further, hereinafter, when “the cycle of the target part” is described, it indicates the operation cycle of the target part.

  The motion detection unit 111 performs a predetermined statistical process on the cycle of each identified target part, thereby obtaining one cycle in which the amplitude changes along the time series based on the motion of each target part. , Specified as the cycle of the speaker's behavior. An example of this statistical process is a weighted average process. Specifically, weighting predetermined for each part is performed on the period of each target part, and the average of the periods of each target part after weighting is specified as the period of the speaker's behavior. The degree of weighting for each part is obtained in advance by experimentation or the like to obtain a causal relationship between the movement of each part and the change in amplitude along the time series of the audio signal f110 (that is, the period f20). Make a decision.

  As another example of this statistical processing, estimation by a Bayesian network may be used. Specifically, a Bayesian network is created in advance based on the causal relationship between the movement of each part and the period of the speaker's behavior (and consequently the causal relation with the period f20). The motion detection unit 111 may apply the Bayesian network using the specified period of each target region as an input and set the output as the period of the speaker's behavior. In this way, by specifying the period of movement for multiple target parts and performing statistical processing, the period of speaker behavior that changes according to the combination of movements of multiple parts can be specified more accurately Is possible.

  The period of the speaker's behavior is not limited to information that does not explicitly specify a time series such as a so-called period. For example, information indicating a change in amplitude along the time series during the monitoring period (time width h111). As described above, the information may clearly indicate time-series positions. By using such information, for example, the start timing of the monitoring period and the start timing of the speaker's behavior cycle (timing at which the amplitude starts to change) do not necessarily have to match. As a specific example, this may be the case when the speaker starts to operate after a certain time has elapsed since the speaker started speaking. In the following, these will be included and described simply as the “speaker behavior cycle”.

  The motion detection unit 111 outputs the specified speaker behavior period to the gain specification unit 112.

  In addition, the motion detection unit 111 identifies the positions and orientations of the feature points P20a and P20b of the part indicating the motion of the speaker's speech such as “mouth” for each predetermined frame (for example, every frame). The motion detection unit 111 sequentially outputs information indicating the positions and orientations of the identified feature points P20a and P20b to the synchronization calculation unit 115.

  The gain specifying unit 112 receives information indicating the cycle of the speaker's behavior from the motion detection unit 111. Based on the period of the speaker's behavior, the gain specifying unit 112 controls the control signal for changing the amplitude of the input signal InA along the time series, that is, changes the gain with respect to the input signal InA along the time series. The control signal shown is generated. As a specific example, the gain specifying unit 112 according to the present embodiment generates a control signal so as to attenuate the input signal InA in synchronization with the period of the speaker's behavior received from the motion detection unit 111. As a result, the control signal is generated so that the input signal InA is attenuated at the timing (phase) at which the amplitude decreases in the period of the speaker's behavior.

  For example, FIG. 4 is a diagram for explaining gain control based on the period of the speaker's behavior in this embodiment. F50 in FIG. 4 is a graph that schematically shows a change in gain along a time series based on the generated control signal. The example in FIG. 4 shows a case where gain control based on the generated control signal is applied in synchronization with the specified period of the speaker's behavior, and ideally along this time series. Gain control is synchronized with the period f20. Details of gain control based on this control signal will be described later. Further, hereinafter, the processing time related to the series of operations of the above-described gain specifying unit 112 is assumed to be h112. In this manner, gain control along the time series (that is, gain change along the time series) is specified by the gain specifying unit 112 as a control signal.

  The gain specifying unit 112 outputs the generated control signal to the gain control unit 1162.

  The audio signal acquisition unit 113 sequentially receives the input signal InA including the audio signal f110 from the sound collection unit 101 as an input. The audio signal acquisition unit 113 sequentially outputs the input signal InA to the delay processing unit 114.

  The audio signal acquisition unit 113 monitors the amplitude of the input signal InA and sequentially outputs at least information indicating the amplitude to the synchronization calculation unit 115.

  The delay processing unit 114 sequentially receives the input signal InA from the audio signal acquisition unit 113. The delay processing unit 114 performs a delay process so that the input signal InA is delayed by a predetermined delay amount h114. The delay amount h114 at this time is determined in advance in consideration of the processing time of the motion detection unit 111 and the gain specifying unit 112. Specifically, the delay amount h114 is obtained by adding at least the time width h111 for the number of frames required for the motion detection unit 111 to specify the period of the speaker's behavior and the processing time h112 of the gain specifying unit 112. The time width h111 + h112 is provided. That is, the delay amount h114 is determined in advance so as to satisfy the condition of h114 ≧ h111 + h112. This is because the control signal for adjusting the amplitude of the input signal InA along the time series is output with a delay of at least the time width h111 + h112 with respect to the input signal InA.

  The delay processing unit 114 outputs the input signal InA subjected to the delay processing to the amplifier 1161.

  The synchronization calculation unit 115 sequentially receives information indicating the positions and orientations of the feature points P20a and P20b of the part indicating the operation of the speaker's utterance from the motion detection unit 111. Based on this information, the synchronization calculation unit 115 monitors changes in the positions and orientations of the feature points P20a and P20b, and detects the timing tB when the speaker's speech operation is started.

  Further, the synchronization calculation unit 115 sequentially receives information indicating the amplitude of the input signal InA from the audio signal acquisition unit 113. The input signal InA includes only the noise f130 at the timing when the speaker is not speaking. That is, the amplitude of the input signal InA at this timing is the amplitude of the noise f130. On the other hand, at the timing when the speaker speaks, the amplitude of the audio signal f110 is superimposed on the amplitude of the noise f130. That is, the amplitude of the input signal InA increases at the timing when the speaker starts speaking. Therefore, the synchronization calculation unit 115 monitors the change in the amplitude of the input signal InA based on this information, and detects the timing tA at which the amplitude of the input signal InA changes (increases) by a predetermined amount or more.

  The input signal InA acquired through the sound collection unit 101 and the moving image InB acquired through the image acquisition unit 102 are not necessarily strictly synchronized. In other words, the period of the speaker's behavior specified from the input signal InA and the moving image InB (and processing based on the period of the speaker's behavior) is not necessarily strictly synchronized. Note that the processing based on the period of the speaker's behavior indicates time-series gain control by the gain control unit 1162 using the control signal generated by the gain specifying unit 112. Details of the gain control unit 1162 will be described later.

  For example, FIG. 5 is a diagram for explaining the synchronization between the input signal InA subjected to the delay process and the process based on the period of the speaker's behavior. The horizontal axis in FIG. 5 indicates time t. F10a to f10c in FIG. 5 schematically illustrate the input signal InA in which the audio signal f110 is superimposed on the noise f130 at different timings. Further, t10a to t10c indicate start timings of the input signals f10a to f10c (that is, timings at which the amplitude of the input signal InA changes (increases) by a predetermined amount or more). Further, t30a to t30c indicate timings when the speaker's speech operation is started. Also, f50a to f50c schematically show processing (that is, gain control along a time series) based on the period of the speaker's behavior at different timings. Further, t50a to t50c indicate the start timings of the processes f50a to f50c based on the period of the speaker's behavior.

  As described above, the start timings t50a to t50c of the processes f50a to f50c based on the period of the speaker's behavior are synchronized with the start timings t30a to t30c of the speech operation. This is because the start timings t50a to t50c and the start timings t30a to t30c are specified from the same moving image InB. On the other hand, the input signal InA and the moving image InB are not strictly synchronized. In other words, the start timings t10a to t10c are not always synchronized with the start timings t30a to t30c and t50a to t50c. That is, one of the start timings t10a to t10c and the start timings t30a to t30c and t50a to t50c may be delayed.

  Therefore, in order to synchronize the input signal InA and the period of the speaker's behavior, strictly speaking, the processing based on the period of the speaker's behavior, the synchronization calculation unit 115 shifts between them along the time series. A quantity (ie time difference) Δt is calculated. Specifically, as shown in FIG. 5, the synchronization calculation unit 115 synchronizes the start timings t10a to t10c of the input signals f10a to f10c with the start timings t30a to t30c of the speakers. These deviation amounts Δt are calculated. Here, the start timings t10a to t10c are indicated by the detected timing tA. The start timings t30a to t30c are indicated by the detected timing tB. That is, the synchronization calculation unit 115 calculates the deviation amount Δt = tA−tB as the difference between the detected timings tA and tB.

  The synchronization calculation unit 115 performs the delay process by controlling the process based on the input signal InA or the period of the speaker's behavior to be shifted (delayed) along the time series by this Δt. The input signal InA and the processing based on the period of the speaker's behavior are synchronized.

  Specifically, when the input signal InA is delayed with respect to the moving image InB (Δt <0), the input signal InA after the delay process is delayed with respect to the process based on the period of the speaker's behavior. Will do. In this case, the synchronization calculation unit 115 notifies the gain control unit 1162 of the shift amount Δt. In response to this deviation amount Δt, the gain control unit 1162 delays the start timing of its own processing by this deviation amount Δt.

  When the moving image InB is delayed with respect to the input signal InA (Δt> 0), the processing based on the period of the speaker's behavior is delayed with respect to the input signal InA after the delay processing. It will be. In this case, the synchronization calculation unit 115 notifies the delay processing unit 114 of the shift amount Δt. When receiving this notification, the delay processing unit 114 further delays the input signal InA by Δt in addition to the delay amount h114. The notification of Δt to the delay processing unit 114 needs to be performed before the delay processing by the delay processing unit 114 is completed. Therefore, the delay amount h114 needs to be set sufficiently longer than the processing time related to the calculation of Δt. However, in many cases, the process related to the calculation of Δt is sufficiently shorter than the delay amount h114. At this time, the synchronization calculation unit 115 may notify Δt = 0 to the gain control unit 1162, and the gain control unit 1162 may immediately start processing.

  When the input signal InA and the moving image InB are synchronized (Δt = 0), the input signal InA after the delay process and the process based on the period of the speaker's behavior are synchronized. Therefore, in this case, the synchronization calculation unit 115 does not perform the process related to the notification of the deviation amount Δt, or notifies either or both of the gain control unit 1162 and the delay processing unit 114 of Δt = 0. That's fine.

  As described above, the synchronization calculation unit 115 specifies the synchronization timing between the input signal InA and the speaker's behavior based on the deviation amount Δt, and matches the input signal InA or the speaker's behavior according to the synchronization timing. The processing based on the period of behavior is shifted (delayed) along the time series. Thereby, the input signal InA and the processing based on the period of the speaker's behavior are synchronized.

  In the above description, the synchronization calculation unit 115 performs processing based on the input signal InA subjected to delay processing based on the shift amount Δt between the input signal InA and the moving image InB and the period of the speaker's behavior. The synchronization timing was indirectly specified. As another method, the synchronization calculation unit 115 receives the input signal InA subjected to delay processing based on the detected timings tA and tB and the above-described time width h111, processing time h112, and delay amount h114. The synchronization timing with the processing based on the period of the speaker's behavior may be directly specified. In this case, for example, the timing corresponding to the input signal InA subjected to the delay process is represented by tA + h114. The timing at which processing based on the speaker behavior cycle can be started is expressed as tB + h111 + h112. Therefore, by calculating the shift amount Δt based on Δt = (tA + h114) − (tB + h111 + h112), the synchronization timing between the input signal InA subjected to the delay process and the process based on the period of the speaker's behavior is specified. be able to.

  In the above description, the synchronization calculation unit 115 specifies the timing tA at which the amplitude of the input signal InA has changed (increased) by a predetermined amount or more. However, the audio signal acquisition unit 113 may perform this. In this case, the audio signal acquisition unit 113 may notify the synchronization calculation unit 115 of information indicating the specified timing tA. Similarly, the motion detection unit 111 may identify the timing tB when the speaker's speech motion is started. In this case, the operation detection unit 111 may notify the synchronization calculation unit 115 of information indicating the specified timing tB.

  The amplifier 1161 receives the input signal InA subjected to the delay process from the delay processing unit 114. The gain Gain of the amplifier 1161 is controlled based on the gain control unit 1162. That is, the amplifier 1161 adjusts (amplifies or attenuates) the amplitude of the input signal InA according to the gain control unit 1162. The amplifier 1161 outputs the input signal InA whose amplitude is adjusted to the voice recognition unit 12 (see FIG. 1B) located at the subsequent stage.

  The gain control unit 1162 receives a control signal indicating a gain change with respect to the input signal InA in time series from the gain specifying unit 112. Based on this control signal, gain control section 1162 controls gain Gain of amplifier 1161 in time series (this operation corresponds to “gain control”).

  Further, the gain control unit 1162 receives a notification of the shift amount Δt from the synchronization calculation unit 115. When receiving the notification of the shift amount Δt, the gain control unit 1162 delays its own process, that is, the gain control start timing by the shift amount Δt. Thereby, gain control along the time series by the gain control unit 1162 (that is, processing based on the period of the speaker's behavior) is synchronized with the input signal InA subjected to the delay processing. This aspect will be described below with reference to FIG.

  As described above, the period of the speaker's behavior corresponds to an estimation of the period f20 at which the amplitude of the voice signal changes along the time series. Therefore, by synchronizing the gain control along the time series by the gain control unit 1162 with the input signal InA subjected to the delay processing, ideally, along the time series in the period f20 as shown in FIG. The gain control (corresponding to the graph f50) is synchronized.

  In this case, timings t501 and t503 at which the gain Gain decreases and the signal is attenuated coincide with timings t201 and t203 at which the amplitude of the audio signal f110 in the input signal f10 (that is, the input signal InA) decreases. Also, the timing t502 at which the gain is increased and the signal is not attenuated (or amplified) coincides with the timing t202 at which the amplitude of the audio signal f110 in the input signal f10 increases.

  Thereby, the higher the ratio of the amplitude of the noise f130 to the amplitude of the audio signal f110, the greater the attenuation of the input signal f10. That is, the input signal f10 is further attenuated at the timing when the noise f130 is dominant, and the amplitude of the input signal f10 is emphasized at the timing when the audio signal f110 is dominant.

  It should be noted that the synchronization calculation unit 115 outputs the audio signal f110 as the input signal InA when the positions and orientations of the feature points P20a and P20b of the part indicating the speaker's utterance operation have not changed for a predetermined time or more. May be determined not to be included. In this case, for example, the synchronization calculation unit 115 temporarily stops gain control processing along the time series by the gain control unit 1162, and when the positions and orientations of the feature points P20a and P20b change again, The processing by the gain control unit 1162 may be started again. At this time, the motion detection unit 11 may specify the speaker's behavior cycle again, and the gain specification unit 112 may regenerate the control signal based on the speaker's behavior cycle.

  In addition, when it is not necessary to strictly synchronize the input signal InA subjected to the delay processing and the processing based on the period of the speaker's behavior, the synchronization between the input signal InA and the moving image InB is guaranteed. In such a case, the synchronization calculation unit 115 may be omitted. In this case, the timing tB at which the speaker's speech operation is started is the timing at which the audio signal f110 starts to be superimposed on the noise f130, that is, the timing tA at which the amplitude of the input signal InA changes (increases) by a predetermined amount or more. It will be treated as equal. In the case of such a configuration, the gain control unit 1162 may start processing immediately without waiting for a notification from the synchronization calculation unit 115.

  Also, in the above description, the processing in the case of a single speaker has been described. However, for a plurality of speakers, one of them is selectively operated as a processing target. May be. In this case, the motion detection unit 111 may identify a speaker to be processed by applying a technique for identifying an individual such as face recognition, for example.

  Note that the selection criterion for the target speaker may be specified by an operator (for example, a speaker) for each predetermined process, or the target speaker may be determined in advance. In addition, for each identified speaker, a parameter of statistical processing applied to the period of each part, for example, how to apply a weight in weighted average processing, or applied Bayesian network may be switched. These pieces of information may be created in advance and stored in a place where the motion detection unit 111 can appropriately read.

  By operating in this way, for example, even when a plurality of speakers are speaking at the same time, the cycle of the speaker's behavior is specified according to the speech of the target speaker, and the input signal InA is based on this cycle. Is adjusted along the time series. As a result, the speech of the target speaker is more easily emphasized than the speech of other speakers, and even when multiple speakers speak, The accuracy can be improved.

[A series of processes of the audio processing unit 11]
Next, a series of processes of the audio processing unit 11 will be described with reference to FIG. FIG. 6 is a flowchart showing a series of operations of the audio processing unit 11.

(Step S11)
The motion detection unit 111 receives the moving image InB from the image acquisition unit 102 as an input. In this moving image InB, the appearance of the speaker is taken in time series, and the operation of the speaker is shown.

  The motion detection unit 111 identifies the positions and orientations of the feature points P20a and P20b of the part indicating the speaker's utterance motion, such as “mouth”, for each predetermined frame (for example, every frame). The motion detection unit 111 sequentially outputs information indicating the positions and orientations of the identified feature points P20a and P20b to the synchronization calculation unit 115.

  Further, the motion detection unit 111 identifies positions of one or more predetermined parts among the parts constituting the body of the speaker from the frame images constituting the moving image InB.

(Step S12)
Next, the motion detection unit 111 monitors the motion of each target part, that is, the change in the position of each target part for a predetermined number of frames, and specifies the cycle of the motion for each target part. .

  The motion detection unit 111 performs a predetermined statistical process on the cycle of each identified target part, thereby obtaining one cycle in which the amplitude changes along the time series based on the motion of each target part. , Specified as the cycle of the speaker's behavior. As an example of this statistical processing, weighted average processing or estimation using a Bayesian network can be cited.

  The motion detection unit 111 outputs the specified speaker behavior period to the gain specification unit 112. The operation related to the specification of the speaker's behavior period corresponds to the “detection step”.

  The gain specifying unit 112 receives information indicating the cycle of the speaker's behavior from the motion detection unit 111. Based on the period of the speaker's behavior, the gain specifying unit 112 controls the control signal for changing the amplitude of the input signal InA along the time series, that is, changes the gain with respect to the input signal InA along the time series. The control signal shown is generated. As a specific example, the gain specifying unit 112 according to the present embodiment generates a control signal so as to attenuate the input signal InA in synchronization with the period of the speaker's behavior received from the motion detection unit 111. As a result, the control signal is generated so that the input signal InA is attenuated at the timing (phase) at which the amplitude decreases in the period of the speaker's behavior.

  The gain specifying unit 112 outputs the generated control signal to the gain control unit 1162. The operation relating to the generation of the control signal corresponds to a “specific step”.

(Step S21)
The audio signal acquisition unit 113 sequentially receives the input signal InA including the audio signal f110 from the sound collection unit 101 as an input. The audio signal acquisition unit 113 sequentially outputs the input signal InA to the delay processing unit 114.

  The audio signal acquisition unit 113 monitors the amplitude of the input signal InA and sequentially outputs at least information indicating the amplitude to the synchronization calculation unit 115.

(Step S22)
The delay processing unit 114 sequentially receives the input signal InA from the audio signal acquisition unit 113. The delay processing unit 114 performs a delay process so that the input signal InA is delayed by a predetermined delay amount h114. The delay amount h114 at this time is determined in advance in consideration of the processing time of the motion detection unit 111 and the gain specifying unit 112.

  The delay processing unit 114 outputs the input signal InA subjected to the delay processing to the amplifier 1161.

(Step S30)
The synchronization calculation unit 115 sequentially receives information indicating the positions and orientations of the feature points P20a and P20b of the part indicating the operation of the speaker's utterance from the motion detection unit 111. Based on this information, the synchronization calculation unit 115 monitors changes in the positions and orientations of the feature points P20a and P20b, and detects the timing tB when the speaker's speech operation is started.

  Further, the synchronization calculation unit 115 sequentially receives information indicating the amplitude of the input signal InA from the audio signal acquisition unit 113. The synchronization calculation unit 115 monitors the change in the amplitude of the input signal InA based on this information, and detects the timing tA at which the amplitude of the input signal InA changes (increases) by a predetermined amount or more.

  The synchronization calculation unit 115 calculates a deviation amount Δt = tA−tB as the difference between the detected timings tA and tB. This shift amount Δt indicates the shift amount (time difference) along the time series between the input signal InA and the processing based on the period of the speaker's behavior.

(Step S41)
The synchronization calculation unit 115 performs the delay process by controlling the process based on the input signal InA or the period of the speaker's behavior to be shifted (delayed) along the time series by this Δt. The input signal InA and the processing based on the period of the speaker's behavior are synchronized.

  Specifically, when the input signal InA is delayed with respect to the moving image InB (Δt <0), the synchronization calculation unit 115 notifies the gain control unit 1162 of the shift amount Δt. In response to this deviation amount Δt, the gain control unit 1162 delays the start timing of its own processing by this deviation amount Δt.

  Further, when the moving image InB is delayed with respect to the input signal InA (Δt> 0), the synchronization calculation unit 115 notifies the delay processing unit 114 of the shift amount Δt. When receiving this notification, the delay processing unit 114 further delays the input signal InA by Δt in addition to the delay amount h114. At this time, the synchronization calculation unit 115 may notify Δt = 0 to the gain control unit 1162, and the gain control unit 1162 may immediately start processing.

  When the input signal InA and the moving image InB are synchronized (Δt = 0), the input signal InA after the delay process and the process based on the period of the speaker's behavior are synchronized. Therefore, in this case, the synchronization calculation unit 115 does not perform the process related to the notification of the deviation amount Δt, or notifies Δt = 0 to one or both of the gain control unit 1162 and the delay processing unit 114. .

(Step S42)
The gain control unit 1162 receives a control signal indicating a gain change with respect to the input signal InA in time series from the gain specifying unit 112. Based on this control signal, gain control section 1162 controls gain Gain of amplifier 1161 in time series (this operation corresponds to “gain control”).

  Further, the gain control unit 1162 receives a notification of the shift amount Δt from the synchronization calculation unit 115. When receiving the notification of the shift amount Δt, the gain control unit 1162 delays its own process, that is, the gain control start timing by the shift amount Δt.

  The amplifier 1161 receives the input signal InA subjected to the delay process from the delay processing unit 114. The gain Gain of the amplifier 1161 is controlled based on the gain control unit 1162. That is, the amplifier 1161 adjusts (amplifies / attenuates) the amplitude of the input signal InA according to the gain control unit 1162. The amplifier 1161 outputs the input signal InA whose amplitude is adjusted to the voice recognition unit 12 (see FIG. 1B) located at the subsequent stage. The operation related to the adjustment of the amplitude of the input signal InA along this time series corresponds to a “signal processing step”.

  The series of operations described above is configured by a program for causing a CPU of a device (for example, the main body M11 in FIG. 1A) that operates the voice processing unit 11 (or the voice recognition system including the voice processing unit 11) to function. can do. This program may be configured to be executed via an OS (Operating System) installed in the apparatus (for example, the main body M11). Moreover, as long as the apparatus which operates the audio | voice processing unit 11 can read this program, the position memorize | stored will not be limited. For example, this program may be stored in a recording medium connected from the outside of the apparatus. In this case, the recording medium may be connected to the apparatus so that the CPU of the apparatus executes the program.

[Modification]
Next, the operation of the audio processing unit 11 according to the modification will be described. The voice recognition processing and syntax analysis executed by the voice recognition unit 12 cannot recognize some voices and recognize the voices (characters) that can be recognized when characters corresponding to the voices are missing. There may be a process of estimating the portion. This processing is missing depending on which part of the set of characters in a predetermined unit such as a sentence, clause, phrase, etc. could be recognized, depending on the characteristics of speech recognition processing and parsing (eg, algorithm). There may be a difference in the ease of estimating the portion. As a specific example, the recognition rate may be higher when the first half of the set of characters of a predetermined unit can be recognized than when the middle part can be recognized. In the speech processing unit 11 according to the modification, based on the period of the speaker's behavior, the position along the time series of the set of characters of the predetermined unit (or the break of sentences between sentences, clauses, phrases, etc.) Is adjusted, and the portion to be emphasized in the input signal is adjusted, thereby improving the recognition rate of the speech recognition process.

  Therefore, in the audio processing unit 11 according to the modification, a predetermined period (for example, 1/2 period or 1 period) of the period of fluctuation of the audio signal, that is, the period of the specified speaker's behavior, is provided. It is assumed that the position is synchronized with the position along the time series of a set of characters in a predetermined unit. In addition, the speech processing unit 11 gains so as to emphasize a predetermined position (timing) in a time series among the set of characters of the predetermined unit according to the characteristics of voice recognition processing and syntax analysis. To control.

  An example thereof will be described below with reference to FIGS. 7A and 7B. FIG. 7A is a diagram for describing gain control based on a period of a speaker's behavior in one aspect of a voice processing unit according to a modification. FIG. 7B shows an aspect different from FIG. 7A of the speech processing unit according to the modification, and is a diagram for describing gain control based on the period of the speaker's behavior in this aspect. In the example shown in FIGS. 7A and 7B, the first half of a set of characters in a predetermined unit is emphasized. In the following, the description will be made with attention paid to the gain specifying unit 112 that is different in processing from the above-described embodiment, and other configurations are the same as those in the above-described embodiment, and thus detailed description thereof is omitted.

  First, the example shown in FIG. 7A will be described. 7A corresponds to the input signal f10 shown in FIG. 4 (that is, the input signal InA), and f20 corresponds to the period f20 shown in FIG. Further, f50 is a graph that schematically shows a change in gain along a time series, and corresponds to f50 in FIG. In addition, f51 in FIG. 7A is a graph that schematically shows a change in gain along the time series in this example.

  The gain identifying unit 112 according to this modification example shifts the phase of the control signal indicating the gain change from the period of the speaker behavior received from the motion detection unit 111 by a predetermined time width h51 along the time series. Generate to deviate. For example, in the example illustrated in FIG. 7A, the graph f51 is shifted in phase forward (front side in time series) by the time width h51 compared to the graph f50. A predetermined constant value may be used as the time width h51, or a relative value with respect to the length of the speaker's behavior cycle (for example, the length of one cycle) may be used. Which of these should be used may be determined according to, for example, the characteristics of speech recognition processing or syntax analysis. In this manner, the gain specifying unit 112 adjusts the phase of the gain change along the time series so that the gain is amplified at a desired phase in the period of the speaker's behavior.

  As a result, as shown in FIG. 7A, the peak position of the gain change along the time series shown by the graph f51 is shifted forward by the time width h51. Therefore, for example, in the portion of the audio signal f110 indicated between timings t201 and t203, the front signal remains without being attenuated. That is, the front part is emphasized between the timings t201 and t203, and the recognition rate of a set of characters of a predetermined unit corresponding to this period can be improved.

  In the above description, the gain specifying unit 112 adjusts the phase of the control signal indicating the gain change. However, for example, the synchronization calculation unit 115 takes the time width h51 into consideration in time series by the gain control unit 1162. You may adjust the start timing of the gain control along.

  Next, the example shown in FIG. 7B will be described. In the example shown in FIG. 7A, by adjusting the phase of the control signal indicating a change in gain, the gain is set so that a predetermined position (timing) along a time series is emphasized in a set of characters in a predetermined unit. Was controlling. In the example illustrated in FIG. 7B, control is performed so that a desired portion is emphasized by shifting the peak position of the gain change along the time series instead of adjusting the phase.

  7B corresponds to the input signal f10 (that is, the input signal InA) shown in FIGS. 4 and 7A, and f20 corresponds to the period f20 shown in FIGS. 4 and 7A. Further, f50 is a graph that schematically shows a change in gain along the time series, and corresponds to f50 in FIGS. 4 and 7A. Further, f52 in FIG. 7B is a graph that schematically shows a change in gain along the time series in this example.

  The gain specifying unit 112 according to this modified example has a predetermined time width h52 along the time series in which the peak position of the control signal indicating the change in gain is relative to the period of the speaker's behavior received from the motion detecting unit 111. It generates so that it may shift only. For example, in the example illustrated in FIG. 7B, the peak position of the graph f52 is shifted to the front side (front side in time series) by the time width h52 compared to the graph f50. As the time width h52, a predetermined constant value may be used, or a relative value with respect to the length of the speaker's behavior cycle (for example, the length of one cycle) may be used. Which of these should be used may be determined according to, for example, the characteristics of speech recognition processing or syntax analysis.

  As a result, as shown in FIG. 7B, the peak position of the gain change along the time series shown by the graph f52 is shifted forward by the time width h52. Therefore, for example, in the portion of the audio signal f110 indicated between timings t201 and t203, the front signal remains without being attenuated. That is, the front part is emphasized between the timings t201 and t203, and the recognition rate of a set of characters of a predetermined unit corresponding to this period can be improved. In the example illustrated in FIG. 7B, the timing at which the amplitude of the audio signal f110 decreases most (for example, t201 and t203) matches the timing at which the gain Gain decreases most (for example, t501 and t503). Therefore, it is possible to attenuate the signal at the timing at which the ratio of the amplitude of the noise f130 to the amplitude of the audio signal f110 is the largest, that is, the timing at which the noise f130 is dominant, and emphasize other portions.

  In the embodiment and the modification described above, an example in which a signal at another timing is attenuated so that the input signal InA is emphasized at a desired timing based on the period of the speaker's behavior has been described. However, the method is not limited to this method as long as it is possible to make a difference in signal amplitude between the emphasized portion of the input signal InA and another portion different from the emphasized portion. For example, after the entire input signal InA is amplified, the amplitude of the signal of the other part may be attenuated. Further, control may be performed so as to amplify the signal of the emphasized portion. For example, the gain specifying unit 112 may generate the control signal by adjusting the gain Gain at each timing in accordance with the desired control.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 101 Sound collecting part 102 Image acquisition part 11 Audio | voice processing unit 111 Operation | movement detection part 112 Gain specification part 113 Audio | voice signal acquisition part 114 Delay processing part 115 Synchronization calculating part 116 Signal processing part 1161 Amplifier 1162 Gain control part 12 Voice recognition part 13 Operation control Part

Claims (15)

A detector that detects the period of the speaker's behavior;
Based on the detected period of the behavior, a specifying unit that specifies a change in gain along a time series;
A signal processing unit that adjusts the amplitude of an input signal including an audio signal along a time series based on the identified change in the gain;
An audio processing apparatus comprising: A synchronization calculation unit for specifying a synchronization timing between the speaker's behavior period and the input signal;
The audio processing apparatus according to claim 1, wherein the signal processing unit determines a timing for adjusting an amplitude of the input signal based on the identified synchronization timing.   The synchronization calculation unit, based on the information indicating the operation of the speaker's utterance detected along the time series, and the change in the amplitude of the input signal along the time series, the period of the speaker's behavior and The audio processing apparatus according to claim 2, wherein a shift amount along a time series with the input signal is calculated, and the synchronization timing is specified based on the shift amount.   The said synchronous calculating part uses the difference between the timing when the amplitude of the said input signal increased more than predetermined amount, and the timing when the operation | movement of the said speech is started as said deviation | shift amount. Voice processing device.   The speech processing apparatus according to claim 1, wherein the detection unit detects a period of the behavior based on image information indicating a speaker's operation.   The detection unit detects a period of movement of a predetermined part of each part of the speaker from the image information, and specifies the period of the behavior based on the detected period. The speech processing apparatus according to claim 5.   The detection unit detects the period of movement of a plurality of parts as the action of the predetermined part, and specifies a period of the behavior by applying a predetermined statistical process to each period of the plurality of parts. The speech processing apparatus according to claim 6, wherein:   The said detection part weights the period of each of these some site | part as said statistical process, and specifies the period of the said behavior by taking the average of the said weighted said period, The said characteristic is characterized by the above-mentioned. Voice processing device.   Based on a Bayesian network created in advance based on a causal relationship between the period of each of the plurality of parts and the period of the behavior as the statistical process, the detection unit calculates the period from each of the detected parts of the plurality of parts. The speech processing apparatus according to claim 7, wherein a period of the behavior is specified.   The speech processing apparatus according to claim 1, wherein the specifying unit specifies the change in the gain so as to synchronize with the detected period of the behavior.   The said specific | specification part specifies the said gain change so that the peak position of the said gain change may shift | deviate only the predetermined time width from the peak position of the said behavior period. Audio processing device.   The specifying unit changes the gain so that the phase is shifted by a predetermined time width with respect to the period of the behavior so that the gain is amplified at a desired phase in the detected period of the behavior. The voice processing apparatus according to claim 1, wherein the voice processing apparatus is specified. A sound collection unit for collecting an input signal including an audio signal;
An image acquisition unit for acquiring the motion of the speaker as a moving image;
A detection unit for detecting a period of a speaker's behavior based on the moving image;
Based on the detected period of the behavior, a specifying unit that specifies a change in gain along a time series;
A signal processing unit that adjusts the amplitude of an input signal including an audio signal along a time series based on the identified change in the gain;
A speech recognition unit for performing speech recognition based on the input signal whose amplitude is adjusted;
A speech recognition device comprising: A detection step for detecting the period of the speaker's behavior;
A specific step of identifying a change in gain along a time series based on the detected period of the behavior;
A signal processing step of adjusting the amplitude of the input signal including the audio signal along a time series based on the specified change in the gain;
A speech processing method comprising: A detection process that detects the period of the speaker's behavior;
Based on the detected period of the behavior, a specific process for specifying a change in gain along a time series;
Signal processing for adjusting the amplitude of the input signal including the audio signal along a time series based on the specified change in the gain;
A voice processing program characterized by executing

Images