JP2006333224A - Voice signal packet communication method, multipoint mixing method, voice signal packet receiving method, and system and apparatus using them - Google Patents

Abstract

In a method of performing packet loss concealment processing in a multipoint mixing unit, an excessive load may be applied to the multipoint mixing unit.
In the present invention, when an upstream packet loss to a multipoint mixing unit occurs, the multipoint mixing unit does not perform packet loss concealment processing, A flag indicating that a packet loss has occurred is incorporated and transmitted to the receiving side. On the receiving side, packet loss concealment processing is performed when a packet loss occurs in the downstream from the multipoint mixing unit and when the flag indicating that “packet loss has occurred in the upstream” is detected from within the voice packet. I do.
Also, a packet loss flag is incorporated and transmitted to the receiving side only when a packet that was the main speaker is lost immediately before packet loss occurs among a plurality of transmission sources.
[Selection] Figure 6

Description

  The present invention provides a countermeasure for packet loss when digitalized audio signals such as voice and music (hereinafter collectively referred to as “voice signals”) are transmitted via a packet communication network such as the Internet. The present invention relates to a communication method, a multipoint mixing method, a reception method, and a system and apparatus thereof.

  Services that transmit voice signals using Voice over IP (Internet Protocol) technology are becoming popular. As shown in FIG. 1, a voice signal from an input terminal 11 is converted into a voice packet by a voice signal transmitter 12 and transmitted to a voice signal receiver 14 by a packet communication network 13 such as an IP network. 14 decodes the voice packet and outputs the reproduced voice to the output terminal 15. When this is performed in real time, depending on the state of the communication network 13, a packet loss (loss) occurs in the middle of the communication network, which causes a problem of quality degradation such that the reproduced voice is interrupted. In particular, in the case of a communication service called “best effort” such as the Internet, this problem is particularly noticeable when the communication network is congested because packet loss is allowed.

  Therefore, when voice signals are communicated via a packet communication network, a technique called packet loss concealment is used. This method is used when a packet is lost in the middle of a communication path or when it has not arrived at the receiving side within a limited time due to a delay in the communication path (hereinafter referred to as “lost packet” or “lost packet”). "). The method of estimating and compensating the speech signal in the section corresponding to") on the receiving side is used. FIG. 2 is a general configuration example of the audio signal transmission unit 12 in FIG. The input voice is stored in the input buffer 21 and sent to the voice packetization unit 22 after being divided at regular intervals called frames. The voice packetization unit 22 converts the framed voice into a voice code using a voice coding method, generates a voice packet, and sends the voice packet to the packet sending unit 23. The packet sending unit 23 sends a voice packet to the packet communication network. Note that the time length of one frame is generally about 10 to 20 milliseconds in many cases. In addition, the ITU-T G. An arbitrary encoding method such as the 711 method may be used, and a PCM (pulse code modulation) signal format may be used.

  FIG. 3 is a general configuration example of the audio signal receiving unit 14 in FIG. Voice packets received by the packet receiver 31 from the packet communication network are stored in a reception buffer 32 also called a fluctuation absorbing buffer. For a frame in which a packet is correctly received, a voice packet is extracted from the reception buffer and decoded into a voice signal by the voice packet decoding unit 33. For a lost packet, a packet loss concealment process is performed by the lost signal generation unit 34. A sound signal is generated, and the generated sound signal is output. When the information of the pitch period (the length on the time axis corresponding to the fundamental frequency of voice) is used for packet loss concealment processing, the output voice signal is stored in the output voice buffer 35 and the pitch extraction unit 36 is used. And the obtained pitch period value is supplied to the lost signal generator 34. The signal generated by the lost signal generation unit 34 is output to the output terminal 15 through the changeover switch 37, and when there is no packet loss, the decoded audio signal from the audio packet decoding unit 33 is output to the output terminal 15 through the changeover switch 37. . Here, the lost signal generation unit 34, the output audio buffer 35, and the pitch extraction unit 36 constitute a packet loss concealment. Note that a communication terminal that performs voice communication in both directions includes both a transmission unit and a reception unit in each terminal.

As a typical method of packet loss concealment, there are methods shown in Non-Patent Document 1 and Non-Patent Document 2. The method shown in Non-Patent Document 1 is a well-known method, and uses the pitch period of voice for packet loss concealment.
Packet loss concealment is an effective method for making it difficult to perceive deterioration in the audibility of the playback sound on the receiving side when packet loss occurs. For example, in the form of FIG. 1, the function is implemented in the audio signal receiving unit, or the audio signal transmitting unit adds redundant information in advance, and the audio signal receiving unit implements processing with less deterioration of the reproduced audio. Or In general, mounting the packet loss concealment in the form of FIG. 1 does not cause a problem in terms of processing amount.

Next, a case where multipoint communication is performed between a plurality of voice communication apparatuses will be described. In general, in the case of multipoint communication, a multipoint mixing unit 100 is prepared as shown in FIG. 4, and transmission signals from the respective voice communication apparatuses 200 </ b> A to 200 </ b> D are combined by the multipoint mixing unit 100. Send to voice communication devices 200A-D. FIG. 5 shows a flow when the voice signal receiving unit 400D of the voice communication device 200D receives voice packets transmitted from the voice signal transmission units 300A to 300C of the voice communication devices 200A to 200C. Each of the audio signal transmission unit 300A, the audio signal transmission unit 300B, and the audio signal transmission unit 300C sends an audio packet to the multipoint mixing unit 100 (also referred to as a multipoint mixing server) via the IP communication network 13. The multipoint mixing unit 100 decodes the voice packet A, the voice packet B, and the voice packet C, performs mixing (addition of the decoded voice), re-encodes the voice packet A, and sends the voice signal reception unit 400D via the IP communication network 13. Voice packet M. In this case, the packet loss between the audio signal transmission unit 300A, the audio signal transmission unit 300B, or the audio signal transmission unit 300C and the multipoint mixing unit 100 (hereinafter referred to as “packet loss in the upstream to the multipoint mixing unit”). .) Occurs. Further, a packet loss (hereinafter referred to as “packet loss in the downstream from the multipoint mixing unit”) also occurs between the multipoint mixing unit 100 and the audio signal receiving unit 400D. Conventionally, a packet loss concealment function has been implemented in the multipoint mixing section 100 as a countermeasure against upstream packet loss up to the multipoint mixing section, and a voice response has been implemented as a countermeasure against downstream packet loss from the multipoint mixing section. A packet loss concealment function was implemented in the signal receiving unit 400. FIG. 5 shows an example of three points of the audio signal transmission units 300A to 300C, but there may be multipoint voice communication of four or more points.
ITU-T Recommendation G.711 Appendix I, “A high quality low-complexity algorithm for packet loss concealment with G.711”, pp.1-18, 1999. Omuro Naka, et al. “Improvement of burst packet loss tolerance by parallel transmission of voice features”, IEICE Technical Report (Institute of Electronics, Information and Communication Engineers), SP2004-77, pp.35-40, 2004.

  In the conventional method in which packet loss concealment processing is performed in the multipoint mixing unit (server) when packet loss in the upstream to the multipoint mixing unit occurs, for example, when packets from multiple points are lost simultaneously In addition, there is a problem that the load of packet loss concealment processing applied to the multipoint mixing server increases. That is, assuming that the load of packet loss concealment processing in the case of one-to-one communication shown in FIG. Such a load is N. As the number of multipoint connection points increases and the number of multipoint connection groups (also called the number of multipoint conferences) processed by one server increases, the load on the multipoint mixing unit (server) increases. There was a possibility.

  In the present invention, when an upstream packet loss to the multi-point mixing unit occurs, the multi-point mixing unit (server) does not perform the packet loss concealment process, A flag indicating that “loss has occurred” (hereinafter referred to as “packet loss flag”) is incorporated and transmitted to the receiving side. On the receiving side, when a packet loss occurs in the downstream from the multipoint mixing unit, and when the flag indicating that “packet loss has occurred in the upstream” is detected from within the voice packet, the packet is Packet loss concealment processing is performed on the assumption that the packet is lost.

  Also, a packet loss flag is incorporated and transmitted to the receiving side only when a packet that was the main speaker is lost immediately before packet loss occurs among a plurality of transmission sources.

  According to the present invention, even when a packet loss occurs up to the multi-point mixing unit, the multi-point mixing unit (server) does not increase the load, and the reception side reproduces sound with less sound quality deterioration on the perception. can do. In addition, when a packet of a transmission source that is not the main speaker is lost, the packet loss concealment process is not performed on the receiving side, so that it is possible to reduce the process with little influence on the call quality.

The present invention can be executed as a computer main body and a computer program, or can be realized by being mounted on a digital signal processor or a dedicated LSI.
[First Embodiment]
In FIG. 6, the structural example of the multipoint mixing part of this invention is shown. Moreover, the processing flow of a multipoint mixing part is shown in FIG. The multipoint mixing unit 100 includes a packet reception unit 31 that receives voice packets, a reception buffer 32 that temporarily stores received packets, a packet loss detection unit 110 that detects occurrence of packet loss, and a received voice packet. A main speaker determination unit 120 that determines which packet of a speaker is mainly speaking, a packet loss flag generation unit 140 that generates a packet loss flag, and a voice waveform that decodes a voice code included in a voice packet into a voice waveform From the decoding unit 130, the adding unit 150 for adding the audio signal, the audio waveform encoding unit 160 for encoding the audio waveform, the packetizing unit 170 for incorporating the audio code and the packet loss flag into the packet, and the packet sending unit 23 for transmitting the packet It is configured. FIG. 6 shows an example in which voice packets are received from three points and mixed, and the voice packet resulting from the mixing is transmitted to one point, as in FIG. It may be two or more points or the same point as the transmission source. In general, in a multipoint audio communication conference, the mixing result is returned to the transmission source.

  The voice packets A to C are received by the respective packet receivers 31 (S31) and stored in the reception buffer 32 (S32). The stored audio signals are decoded into audio signals by the audio waveform decoding units 130A to 130C in accordance with the packet order (S130). The adder 150 adds the decoded speech signals (S150), and the speech waveform encoder 160 encodes them again. When an upstream packet loss up to the multipoint mixing unit occurs, each speech waveform decoding unit outputs a zero signal (silence). This is equivalent to decoding and adding only the voice at a point where no packet loss occurs.

  The packet loss detection unit 110 monitors the reception state of voice packets from each point and determines whether a packet loss has occurred (S111). The main speaker determination unit 120 uses the information contained in the voice packets A to C to determine which speaker at the transmission point A to C at that time is mainly speaking (S121). “Mainly speaking” means that, for example, if only the speaker at the point B is speaking and the speakers at the other two points are not speaking, the point where the speaker is mainly speaking is B, and the main voice packet Is a voice packet B. If two or more people are speaking at the same time, it is determined which is the main speaker from the volume of the voice and the continuation state of the speech.

  Note that “using information contained in voice packets A to C” may be determined by analyzing a voice code contained in the packet. Alternatively, if the voice packets A to C include additional information for determining the main speaker from the transmission side in advance, the determination may be made using the additional information. From the viewpoint of server load reduction, the latter is preferable. There is a method in which additional information for determining a main speaker in advance on the transmission side, for example, an identifier of a voice section or a non-voice section, or power information is incorporated into a packet.

  In the packet loss flag generation unit 140, when the packet loss detection unit 110 detects a loss in one or a plurality of packets of the voice packets A to C (S112), the packet loss is detected at the immediately preceding time. When the detected point is the main speaker (S122) (for example, when the point B is the main speaker at the previous time and it is determined that the voice packet B is lost at the next time) For example, 1 is set as the packet loss flag (S141), and 0 is set otherwise (S142). Other examples include the case where the main speaker is the point A and the voice packet C is lost, and the case where all the packets at all points are not lost. In voice packet communication, since it is common to divide voice into sections called frames of about 10 milliseconds to 20 milliseconds, encode the voice and then send it as a packet. Indicates a time order in units of frames. The time order is generally recorded as a time stamp in the header of the packet.

The packetizing unit 170 generates a single packet by combining the speech code output from the speech waveform encoding unit 160 and the packet loss flag information output from the packet loss flag generating unit 140 (S170). The unit 23 transmits the voice packet M (S23).
FIG. 8 shows a configuration example of the audio signal receiving unit in the present invention. FIG. 9 shows a processing flow of the audio signal receiving unit. The audio signal receiving unit 400 includes a packet receiving unit 31 that receives audio packets, a reception buffer 32 that temporarily stores received packets, a packet loss detecting unit 410 that detects the occurrence of packet loss, and the occurrence of packet loss. A packet loss flag detection unit 420 that detects a packet loss flag shown, a packet loss concealment 430, a speech waveform decoding unit 440, a switch unit 460 that selects output speech, and an OR unit 450 that controls the switch. The received voice packet M is received by the packet receiving unit 31 (S31) and stored in the reception buffer unit 32 (S32). The voice code included in the accumulated voice packet is decoded into a voice waveform by the voice waveform decoding unit 440 in accordance with the order of the time stamp of the packet (S440). The switch unit 460 is normally set on the voice waveform decoding unit 440 side (S461), and the output voice of the voice waveform decoding unit 440 is output as the output of the voice signal receiving unit.

The packet loss detection unit 410 monitors the reception state of the voice packet (S411), and determines whether or not a packet loss has occurred (S412). The packet loss flag detection unit 420 monitors the packet loss flag incorporated in the voice packet (S421), and determines whether it indicates a packet loss state (S422). The OR unit 450 switches the switch unit 460 when either the packet loss detection unit 410 detects a packet loss or the packet loss flag detection unit 420 detects a packet loss flag indicating a packet loss state from the packet. Switch to the packet loss concealment side (S462).

  The packet loss concealment 430 may be the same method as the conventional method shown in FIG. 3 (S430). That is, it may be configured from the lost signal generation unit 34, the output audio buffer 35, and the pitch extraction unit 36. As conventional techniques of packet loss concealment, there are cases where output voice is stored and used in a buffer, or redundant information embedded in an input voice packet M is extracted and used.

  As described above, by using the configuration of the multipoint mixing unit and the audio signal receiving unit in the present invention, even when packet loss in the upstream to the multipoint mixing unit occurs, the multipoint mixing unit It just adds the information that it has occurred. The packet loss concealment process is performed by the audio signal receiving unit (receiving terminal) regardless of whether a packet loss occurs in the upstream or a packet loss occurs in the downstream. Therefore, even when a packet loss occurs in the uplink without increasing the load on the multipoint mixing unit, it is possible to reproduce sound with little deterioration in sound quality on hearing.

  In the present invention, the packet loss concealment process is not performed when an uplink packet other than the point determined as the “main utterance point” by the multipoint mixing unit is lost. This is because even if a packet at a point where speech is not made is lost, the sound quality on hearing is not greatly affected.

  Usage forms for performing voice communication on an IP communication network have become widespread, and by applying the present invention, low-cost and highly reliable multipoint voice communication (multipoint voice communication conference) can be realized.

The figure which shows the structure of the one-to-one audio | voice communication via an IP communication network. The figure which shows the general structural example of an audio | voice signal transmission part. The figure which shows the general structural example of an audio | voice signal receiving part. The figure which shows the structure of the multipoint audio | voice communication via an IP communication network. The figure which shows the flow of the audio | voice signal in multipoint audio | voice communication. The figure which shows the structural example of the multipoint mixing part of this invention. The figure which shows the processing flow of the multipoint mixing part of this invention. The figure which shows the structural example of the audio | voice signal receiving part of this invention. The figure which shows the processing flow of the audio | voice signal receiving part of this invention.

Claims (10)

When performing multipoint voice packet communication between two or more communication devices including at least a transmission unit and one or more communication devices including at least a reception unit,
In the transmitter,
A process of generating a frame audio signal by dividing the audio signal at regular intervals called frames,
Converting the frame audio signal into an audio code, storing it in a packet and transmitting it;
In the multi-point mixing section,
Storing received packets in the receive buffer;
The process of specifying the frame number to be extracted;
A loss detection process for determining whether or not a packet including a voice code corresponding to the frame number to be taken out is stored in the reception buffer;
Decoding speech code into speech waveform;
Adding the decoded speech waveform; and
Encoding the added speech waveform to generate an added speech code;
In the receiver,
Storing received packets in the receive buffer;
The process of specifying the frame number to be extracted;
A loss detection process for determining whether or not a packet including a voice code corresponding to the frame number to be taken out is stored in the reception buffer;
A voice signal packet communication method comprising:
In the multipoint mixing section,
A packet loss flag generation process for generating a flag indicating packet loss (hereinafter referred to as a “packet loss flag”) when it is determined that a packet loss has occurred in the loss detection process;
A packetizing process for storing the packet loss flag and the added voice code in a packet and transmitting the packet;
In the receiving unit,
A flag determination process for determining whether the packet loss flag is not included in the received packet;
When it is determined in the loss detection process that a packet including a voice code corresponding to the frame number to be extracted is stored in the reception buffer, and in the flag determination process, the packet loss flag is included in the packet. If it is determined that it is not, a voice packet decoding process for extracting a voice code from the packet stored in the reception buffer and decoding it into a voice waveform to be a frame output voice signal;
When it is determined in the loss detection process that a packet including a frame audio signal corresponding to the frame number to be extracted is not stored in the reception buffer, or in the flag determination process, the packet loss flag is included in the packet If it is determined that the packet has been lost, a loss process that performs packet loss concealment, and
A voice signal packet communication method comprising: connecting and outputting frame output voice signals output from the voice packet decoding step or the loss processing step. The voice signal packet communication method according to claim 1,
In the multipoint mixing section,
A main speaker determination process for determining a main speaker from the extracted voice codes from the plurality of transmitters,
Only when it is determined that a packet loss has occurred in the loss detection process and in the main speaker determination process, it is determined that the packet is from the transmitter that was the main speaker in the frame immediately before the occurrence of the packet loss. A voice signal packet communication method comprising: the packet loss flag generation step of generating a loss flag. When performing multipoint voice packet communication between two or more communication devices including at least a transmission unit and one or more communication devices including at least a reception unit,
In the multi-point mixing section,
Storing received packets in the receive buffer;
The process of specifying the frame number to be extracted;
A loss detection process for determining whether or not a packet including a voice code corresponding to the frame number to be taken out is stored in the reception buffer;
Decoding speech code into speech waveform;
Adding the decoded speech waveform; and
In a multipoint mixing method for voice signal packets, the method comprising: encoding a summed voice waveform to generate a summed voice code;
In the multipoint mixing section,
A packet loss flag generation process for generating a packet loss flag when it is determined that a packet loss has occurred in the loss detection process;
A multipoint mixing method, comprising: a packetization process in which the packet loss flag and the added voice code are stored in a packet and transmitted. A multipoint mixing method for voice signal packets according to claim 3,
In the multipoint mixing section,
A main speaker determination process for determining a main speaker from the extracted voice codes from the plurality of transmitters,
Only when it is determined that a packet loss has occurred in the loss detection process and in the main speaker determination process, it is determined that the packet is from the transmitter that was the main speaker in the frame immediately before the occurrence of the packet loss. And a packet loss flag generation process for generating a loss flag. When performing multipoint voice packet communication between two or more communication devices including at least a transmission unit and one or more communication devices including at least a reception unit,
In the receiver,
Storing received packets in the receive buffer;
The process of specifying the frame number to be extracted;
A loss detection process for determining whether or not a packet including a voice code corresponding to the frame number to be taken out is stored in the reception buffer;
In a voice signal packet receiving method having:
In the receiving unit,
A flag determination process for determining whether a packet loss flag is included in the received packet;
When it is determined in the loss detection process that a packet including a voice code corresponding to the frame number to be extracted is stored in the reception buffer, and in the flag determination process, the packet loss flag is included in the packet. If it is determined that it is not, a voice packet decoding process for extracting a voice code from the packet stored in the reception buffer and decoding it into a voice waveform to be a frame output voice signal;
When it is determined in the loss detection process that a packet including a frame audio signal corresponding to the frame number to be extracted is not stored in the reception buffer, or in the flag determination process, the packet loss flag is included in the packet If it is determined that the packet has been lost, a loss process that performs packet loss concealment, and
A voice signal packet receiving method comprising: connecting and outputting frame output voice signals output from the voice packet decoding step or the loss processing step. When performing multipoint voice packet communication between two or more communication devices including at least a transmission unit and one or more communication devices including at least a reception unit,
In the transmitter,
Means for generating a frame audio signal by dividing the audio signal at regular intervals called frames;
Means for converting the frame audio signal into an audio code, storing it in a packet and transmitting it;
In the multi-point mixing section,
Means for storing received packets in a reception buffer;
Means for specifying the frame number to be extracted;
Lost detection means for determining whether or not a packet including a voice code corresponding to the frame number to be extracted is stored in the reception buffer;
Means for decoding a speech code into a speech waveform;
Means for adding the decoded speech waveform;
Means for encoding the added speech waveform to generate an added speech code;
In the receiver,
Means for storing received packets in a reception buffer;
Means for specifying the frame number to be extracted;
In the voice signal packet communication system, comprising: loss detection means for determining whether or not a packet including a voice code corresponding to the frame number to be taken out is stored in the reception buffer.
In the multipoint mixing section,
A packet loss flag generating means for generating a packet loss flag when it is determined by the loss detecting means that a packet loss has occurred;
Packetizing means for storing the packet loss flag and the added voice code in a packet and transmitting the packet;
In the receiver,
Flag determining means for determining whether the packet loss flag is not included in the received packet;
When it is determined by the loss detection means that a packet including a voice code corresponding to the frame number to be taken out is stored in the reception buffer, and the packet loss flag is included in the packet by the flag determination means If it is determined that the packet is not, a voice packet decoding unit that extracts a voice code from the packet stored in the reception buffer, decodes the voice code into a voice waveform, and generates a frame output voice signal;
The packet loss flag is included in the packet when the loss detection unit determines that the packet including the frame audio signal corresponding to the frame number to be taken out is not stored in the reception buffer, or the flag determination unit includes If it is determined that the packet is lost, a loss processing means for performing packet loss concealment,
A voice signal packet communication system comprising: a frame output voice signal output from the voice packet decoding means or the loss processing means. The voice signal packet communication system according to claim 6,
In the multipoint mixing section,
A main speaker determination means for determining a main speaker from the extracted voice codes from the plurality of transmitters;
Only when it is determined that a packet loss has occurred in the loss detection means and the main speaker determination means determines that the packet is from the transmitter that was the main speaker of the frame immediately before the occurrence of the packet loss, the packet The voice signal packet communication system, comprising: the packet loss flag generation means for generating a loss flag. Means for storing received packets in a reception buffer;
Means for specifying the frame number to be extracted;
Lost detection means for determining whether or not a packet including a voice code corresponding to the frame number to be extracted is stored in the reception buffer;
Means for decoding a speech code into a speech waveform;
Means for adding the decoded speech waveform;
A multipoint mixing apparatus for voice signal packets, comprising: means for encoding the added voice waveform to generate an added voice code;
A packet loss flag generating means for generating a packet loss flag when it is determined by the loss detecting means that a packet loss has occurred;
A multipoint mixing apparatus comprising: packetizing means for storing the packet loss flag and the added voice code in a packet and transmitting the packet. The multipoint mixing apparatus according to claim 8, wherein
A main speaker determination means for determining a main speaker from the extracted voice codes from the plurality of transmitters;
Only when it is determined that a packet loss has occurred in the loss detection means and the main speaker determination means determines that the packet is from the transmitter that was the main speaker of the frame immediately before the occurrence of the packet loss, the packet The multipoint mixing apparatus, comprising: the packet loss flag generation means for generating a loss flag. Means for storing received packets in a reception buffer;
Means for specifying the frame number to be extracted;
In the voice signal packet receiving apparatus, comprising: loss detection means for determining whether or not a packet including a voice code corresponding to the frame number to be taken out is stored in the reception buffer.
Flag determining means for determining whether or not a packet loss flag is included in the received packet;
When it is determined by the loss detection means that a packet including a voice code corresponding to the frame number to be taken out is stored in the reception buffer, and the packet loss flag is included in the packet by the flag determination means If it is determined that the packet is not, a voice packet decoding unit that extracts a voice code from the packet stored in the reception buffer, decodes the voice code into a voice waveform, and generates a frame output voice signal;
The packet loss flag is included in the packet when the loss detection unit determines that the packet including the frame audio signal corresponding to the frame number to be taken out is not stored in the reception buffer, or the flag determination unit includes If it is determined that the packet is lost, a loss processing means for performing packet loss concealment,
A voice signal packet receiving apparatus comprising: a means for concatenating and outputting frame output voice signals output from the voice packet decoding means or the loss processing means.

Images