JP2006074555A - Audio / video adjustment method for multimedia gateway - Google Patents

Abstract

The present invention relates to an audio / video adjustment method in a multimedia gateway provided at a connection point between an IP network and a multimedia mobile terminal network, and relates to audio and video from an IP network in a frame having a constant cycle of 64 Kbps. The purpose is to adjust the audio and video to the optimum rate when multiplexed and transmitted.
A multiplexing processing unit includes a scheduler that performs control for generating multiplexed data, and a multiplexing table that stores data of a combination of an audio rate and a moving image rate within each fixed length. The multiplexing table is set so that the audio rate is the lowest rate and the remaining capacity is for video and overhead for transmission to the mobile terminal, and the scheduler multiplexes using the audio rate data and video rate of the multiplexing table. Configured to generate structured data.
[Selection] Figure 1

Description

  The present invention relates to an audio / video adjustment method in a multimedia gateway provided at a connection point between an IP network and a multimedia portable terminal network capable of audio / video transmission / reception on 64Kbps-based 3G-324M (3GPP).

  In recent years, with the advancement of mobile phone technology, the transition from the second-generation mobile communication, which is mainly a conventional voice, to the third-generation communication that combines voice and video is progressing. Among them, carriers that employ W-CDMA (Wideband Code Division Multiple Access) as a standard by the standardization project of the third generation (3G) mobile communication system are 3GPP (3rd Generation Partnership Project) The standard 3G-324M is adopted as the audio visual communication standard established by the 3GPP standard. In the 3G-324M standard, video data is MPEG-4 (Moving Picture Experts Group: encoding technology for low-speed lines defined by the International Organization for Standardization ISO and an encoding method corresponding to advanced video formats) or H.264. 263 (compressed video encoding for low-speed video conferences and videophones recommended by ITU-T) is used, and audio data is encoded by AMR (Adaptive Multi Rate). The method is adopted.

  Multimedia communication using 3GPP is in the spotlight as a more efficient multimedia communication technology because communication is possible by multiplexing audio and moving images within a limited band of 64 Kbps.

  Furthermore, with the explosive spread of IP networks in recent years, multimedia communication using inexpensive IP networks is also flourishing, and multimedia services connecting these IP networks with mobile networks using 3G-324M are also signs of widespread use. Has been showing. However, a problem with connection between an IP network and a mobile communication network using 3G-324M is that the G. of 64 Kbps on the IP network side. Speech converted to 711 (one of the recommendations of the ITU-T speech coding system, which is a PCM system that samples speech at 8 KHz, each sample is represented by 8 bits, and realizes a transmission rate of 64 Kbps) MPEG4 or H.264 How to multiplex a moving picture encoded by H.263 into 64Kbps 3G-324M while maintaining a high level of sound quality and image quality.

  In general, on 3G-324M, AMR, which is a compressed audio coding method, is used for audio, and MPEG4 or H.264 is used for moving images. It is common to use H.263. Since AMR is a variable encoding system consisting of a plurality of rates, in order to multiplex audio and video within 64 Kbps, the maximum rate of 3G-324M, the video rate is increased when the audio is at the lowest rate, and the audio is at the highest rate. Sometimes the video rate is reduced, which means that the audio and video quality conflict.

  As described above, in the conventional method, since the limited audio / video rate of 64 Kbps is set at a predetermined rate, there is a problem that if one is increased, the other is decreased.

  FIG. 13 shows the configuration of a system including a conventional multimedia gateway. In the figure, 80 is a multimedia portable terminal, 81 is a multimedia portable terminal network that transmits audio and moving images by an encoding method using 3G-324M or H263, and 82 is a multimedia portable terminal network 81 and an IP network. A multimedia gateway 83, which is provided in between and transmits multimedia information between two network terminals or servers, SIP (Session Initiation Protocol: signaling for starting, editing and ending a session on an IP network) A call agent (CA: Call Agent) that controls audio and video calls over an IP network by controlling a protocol processing server) and MEGACO (Media Gateway Control: a device that performs protocol processing to control a media gateway) ), 84 is an I for performing communication by IP (Internet Protocol). A P network 85 is a multimedia distribution server that is connected to the IP network and distributes multimedia information such as voice and moving images.

  As shown in FIG. 13, the multimedia portable terminal 80 has a function of bidirectionally transmitting and receiving not only voice but also a moving image, and the multimedia portable terminals 80 accommodated in the multimedia portable terminal network 81 are based on the 3G-324M standard. Thus, mutual communication by voice and video is performed. In addition, a service for distributing audio and video as multimedia information from the multimedia distribution server 85 connected to the IP network 84 to the multimedia portable terminal 80 is realized. In this case, audio and video information is transmitted from the multimedia distribution server 85 to the IP network 84 in a form of separate audio packets 85a and video packets 85b. The voice packet 85a includes IP / UDP (includes partner application information in the upper User Datagram Protocol of IP) / RTP (Realtime Transport Protocol) / G. 711 (encoding system corresponding to a 64 Kbps signal obtained by sampling and digitizing voice at 8 KHz) and a packet corresponding to the encoding system. The video packet 85b uses the IP / UDP / RTP protocol in common with audio (the header includes information for identifying audio and video), but MPEG4 or H263 is used as the encoding method. . These voice and video packets enter the multimedia gateway 82 alternately through the same route of the IP network 84.

  FIG. 14 shows the configuration of a multimedia gateway that is the subject of the present invention. FIG. 14 includes a mechanism for transmitting a downlink signal transmitted from the IP network to the multimedia terminal and a mechanism for transmitting an uplink signal transmitted from the multimedia terminal to the IP network. In the figure, the downstream mechanism 820a is IP / UDP / RTP / G. 802b is a video packet termination unit for terminating a video packet from the IP network, 821a and 821b are audio data, a jitter buffer for the video data, and 822 is a voice CODEC process. 823 is an audio buffer, and 824 is a 3G-324M processing unit that multiplexes the audio data of the audio buffer 823 and the moving image data of the jitter buffer 821b and converts them into frames of the 3G-324M standard (64 Kbps). An upstream side mechanism 825 is a 3G-324M termination unit that terminates and separates frames including audio and moving images from the multimedia terminal, and 826a is an H.264 standard. H.245 that processes call control information according to H.245 (end-to-end communication control protocol). 245 processing unit, 826b is a voice extracting unit that extracts voice data, 826c is a moving image extracting unit that extracts moving image data, and 827 is a call by communication with a CA (83 in FIG. 13) that receives call control information and uses MEGACO. The CPU 828 that performs control receives the audio data (AMR encoding) on the 3G-324M side, A CODEC 829 for converting into a signal encoded with 711 inputs audio data from the CODEC 828 and moving image data from the moving image extraction unit 826c, and inputs IP / UDP / RTP / G. 711 standard voice (RTP) packets and IP / UDP / RTP / MPEG4 or H.264 standard. This is an IP packetizing unit that divides and transmits the moving image (RTP) packet according to the H.263 standard.

  64 Kbps 3G-324M data input from the multimedia portable terminal network (81 in FIG. 13) is transmitted to a 3G-324M termination unit 825, with audio (AMR) 826b, moving image (MPEG4 or H.263) 826c, data (H . 245 compatible call control data) 826a, and the audio is converted from the AMR compressed code to the G. Data (H.245) is terminated with NSRP (Numbered Simple Retransmission Protocol) and CCSRL (Control Channel Segmentation and Reassembly Layer). Then, it is transferred to the CPU 827. The CPU cooperates with the control content from MEGACO to 245 negotiations are performed. G. The voice converted to 711 is converted into RTP packets (IP / UDP / RTP) by the IP packetization unit 829 and then transferred onto the IP network. In addition, the moving image is not subjected to codec processing but is converted into RTP packets (IP / UDP / RTP) by the IP packetization unit 829 and transferred onto the IP network.

  On the other hand, the audio and moving images distributed from the multimedia distribution server 85 have G. The video is encoded with MPEG4 or H.264. 263, each transmitted as an IP network 84 as a packet, terminated at the IP / UDP / RTP terminator 820a, 820b of the multimedia gateway shown in FIG. 14, and the header of the IP / UDP / RTP protocol is removed respectively. In order to absorb fluctuations, encoded data of audio and moving images is stored in jitter buffers 821a and 821b, and encoded data of moving images in MPEG4 (or H.263) is input to the 3G-324M processing unit 824, G. The encoded data 711 is AMR encoded at the AMR rate determined by the CODEC processing unit 822 and stored in the audio buffer 823, and the output is input to the 3G-324M processing unit 824. The 3G-324M processing unit 824 combines (multiplexes) the AMR audio data in the audio buffer 823, the MPEG4 moving image data, and the data (call control data) from the CPU 827, and has a frame configuration having a bit number corresponding to a rate of 64 Kbps. Multiplexed data (MUX-PDU: Multiplex Protocol Data Unit) is generated and transmitted to the multimedia portable terminal 80 via the multimedia portable terminal network 81. Conventionally, a multiplexing table in which a rate (allocation amount) of data amount of voice, video, and control information (H.245) is set in advance is transmitted from the information transmission side to the other side, and the transmission side In accordance with the multiplexing, the separation process on the receiving side can be performed using the multiplexing table.

  In the case of downlink transmission, the multiplexing table is sent from the multimedia gateway (3G-324M processing unit) to the multimedia portable terminal, and in the case of uplink, the multiplexing table is sent from the multimedia portable terminal to the multimedia gateway. It is done. At this time, in the multiplexing table, the audio and moving image rates are set at a predetermined rate within a limited range of 64 Kbps.

Conventionally, a means for transmitting a coded bit stream by multiplexing moving images, still images, audio, etc. via the Internet network and a cellular phone network, and a scene description for synthesizing and reproducing each coded data as one scene There is a technique of a multimedia information transmitting apparatus that transmits a stream of encoded bitstream transmitting means and a scene description of scene description transmitting means at different transmission paths or at different timings (see Patent Document 1). . In addition, a mobile phone technology having a function of transmitting and receiving moving images according to the MPEG4 standard and a function of receiving terrestrial digital broadcasting is known (see Patent Document 2).
JP 2003-299044 A JP 2003-153111 A

  The technique of Patent Document 1 is a technique for transmitting a multiplexed encoded bit stream and a scene description, and Patent Document 2 is a technique of a mobile phone equipped with a function of transmitting / receiving MPEG4 images and receiving digital terrestrial broadcasting, This does not suggest that the audio / video rate is efficiently controlled within a fixed bandwidth (64 Kbps).

  In the conventional system described in FIGS. 13 and 14, it is necessary to multiplex audio / video within 64 Kbps, and it is difficult to multiplex both with high quality.

  Also, the video and audio rates are set to be constant by a predetermined multiplexing table for all terminals connected to the gateway. To change according to the situation, the audio and video conditions are detected and an algorithm is used. Therefore, it is necessary to constantly repeat a series of processing to calculate the ratio and perform notification to change the table, which increases the load, and therefore corresponds to the characteristics of each mobile terminal and the environment on the mobile terminal side. However, depending on the mobile terminal, there is a problem in connectivity (cannot be connected because only some rates are supported depending on the model).

  In addition, the video output from the distribution server is the maximum rate to the last, and if the average rate is much lower than this, the line resources are not effectively used. For example, when the video output rate is set to 50 Kbps, the possible audio output rate is 4.75 Kbps considering the overhead of 3G-324M (eg management bits), but the video output rate shows the maximum value It is assumed that the actual average rate is about 40 Kbps. In this case, data corresponding to 10 Kbps is wasted, which is inefficient.

  Furthermore, since only a certain audio / video rate determined in advance from the distribution server is dropped from the distribution server, there is no means for recognizing the reception status on the mobile terminal side and no means for feeding it back. The stability of can not be ensured. There is no way to check the reception status on the mobile terminal side, so the stability of sound quality and image quality cannot be ensured.

  To solve this problem, a video translator is provided for each channel on the multimedia distribution server or multimedia gateway, and the audio rate can be freely output by adjusting the video rate variably with the translator. Although it is possible, implementing a video translator with heavy processing capability will lead to a decrease in channel density, an increase in the size of the device, and an increase in the device price.

  The present invention maintains the quality of video by maximizing the video rate of G-324M audio and video multiplexed data (MUX-PDU) corresponding to a multimedia portable terminal network, and Provide an audio / video adjustment method for multimedia gateways that can adjust the audio / video to an appropriate rate with a limited amount of data, such as by assigning that amount to the quality of the audio when there is room in the rate For the purpose.

  FIG. 1 shows the principle configuration of the present invention. In this basic configuration, a multimedia gateway receives audio packets and video packets as multimedia information from the IP network (84 in FIG. 13) (information from the multimedia distribution server 85 etc. in FIG. 13). Only the configuration of the multiplexing processing unit that generates the downlink 3G-324M multiplexed data to be transmitted to the mobile terminal of the terminal network is shown. The configuration for generating the uplink signal and the configuration for call control are as follows: The illustration is omitted. In the figure, 1a is provided in the multimedia gateway, a multiplexing processing unit for multiplexing the downstream multimedia signal from the IP network to the multimedia portable terminal network into 3G-324M, 10 is a multiplexed data storage unit, 11 is a scheduler, 11a is a multiplexing table initial setting means, 11b is a monitoring means for monitoring a moving image rate, 11c is a multiplexing processing means, 12 is a multiplexing table, 2a is a termination unit, 2b is a jitter buffer, 2c is a codec ( CODEC) processing unit, 2d is an audio buffer, 3a is a termination unit, and 3b is a jitter buffer.

  The scheduler 11 of the multiplexing processing unit 1a uses the lowest rate (for example, the rate of audio and video of multimedia information to be transmitted to the multimedia portable terminal (80 in FIG. 13) connected by the initial setting unit 11a (for example, The number of bytes corresponding to 4.7 Kbps), the maximum rate of the moving image (for example, the number of bytes corresponding to 50 Kbps), and the remaining multiplexing pattern assigned to overhead (for maintenance management) in the multiplexing table 12 Set for the channel assigned for terminal communication. IP / UDP / RTP / G.P input from the IP network (84 in FIG. 13). 711 voice packet is terminated at the termination unit 2a, and the header of the frame corresponding to RTP is removed. The audio data encoded by 711 is supplied to the codec processing unit 2c through the jitter buffer 2b. Under the control of the scheduler 11, the codec processing unit 2 c performs G. The code 711 is converted into an AMR code (by the lowest rate) and stored in the audio buffer 2d. The moving image packet from the IP network is terminated at the termination unit 3a, and the MPEG4 data excluding the header is stored in the jitter buffer 3b.

  The multiplex processing means 11c of the scheduler 11 refers to the multiplex table 12 for the audio / video rate set for the channel assigned to the communication of the mobile terminal, identifies that the audio is the lowest rate, The data in the buffer 2d is converted into the lowest rate AMR code by the codec processing unit 2c, then transferred to the multiplexed data storage unit 10 by the scheduler, and the moving image data (MPEG4) stored in the jitter buffer 3b is set in the multiplexing table 12 The data is converted into a data length corresponding to the rate, and stored in the multiplexed data storage unit 10. The overhead data is set in the multiplexed data storage unit 10 with a predetermined fixed length. If the frame length of the multiplexed data is constant, the data amount is fixed corresponding to 64 Kbps.

  By performing communication at the initially set rate (the audio code data is set to the lowest rate), the quality of the moving image can be maintained to the maximum. When operating in this way, the monitoring means 11b of the scheduler 11 monitors the data length of the moving image data 10b of the multiplexed data storage unit 10, and if a free space is generated (an area where the amount of moving image data corresponds to the maximum rate) If it is smaller, the vacant amount is detected and notified to the multiprocessing means 11c. The multiplex processing means 11c performs conversion by increasing the speech coding rate in accordance with the notified empty amount (a pattern in which the speech coding rate is increased is also registered in the multiplex table 12 in advance). Change the pattern number for setting). Thereafter, when the amount of moving image data is increased by the monitoring unit 11b, the multiprocessing unit 11c performs control so as to decrease the audio coding rate.

  In the above description, the audio / video rate is set to the lowest rate in the above description. However, the optimum audio rate is set in the multiplexing table 12 for each channel of each mobile terminal without setting the minimum rate. Also good.

  Furthermore, by setting a threshold value in the video buffer, if the threshold value is exceeded, a multiplexing table with a higher multiplexing density is selected, or the overhead of the header (maintenance and management) is reduced to increase the video rate. Multiplexed data can be output on the 3G-324M line while maintaining a high rate within a possible range for both audio and moving images.

  Also, when multiplexing audio / video, it is possible to multiplex the data for audio evaluation and image evaluation, and to maintain a high level of sound quality / image quality by feeding back these evaluation results from the mobile terminal. Good.

  According to the principle of the present invention and the first embodiment, in the multimedia gateway, the image quality can be maintained at a certain level or higher by setting the audio / video rate audio to the lowest rate and maximizing the video rate.

  In addition, according to the configuration of the second embodiment described later, it is possible to effectively use free resources of multiplexed data by monitoring the moving image rate, and improve the sound quality by increasing the AMR rate. In addition, it is possible to transmit an appropriate audio rate for each terminal by holding a table for managing the audio rate to be transmitted for each channel according to Example 3 described later.

  In addition, according to a fourth embodiment to be described later, when a threshold is provided in the moving image buffer and the threshold is exceeded, a multiplexing table having a higher multiplexing rate is selected, or an adaptation layer of multiplexed data (MUX-PDU) (AL) Reduce the header overhead by eliminating the header option, increase the multiplexing efficiency, and multiplex on 3K-324M of 64Kbps while maintaining the high rate within the possible range for both audio and video. Is possible.

  Furthermore, when multiplexing according to Example 5 described later, the voice evaluation data is multiplexed, and the evaluation result is fed back from the mobile terminal, whereby the multimedia gateway can set the voice rate to an appropriate value. . Further, according to the sixth embodiment, in addition to the voice evaluation data, the image evaluation data is multiplexed and transmitted to the mobile terminal, and the evaluation result is fed back from the mobile terminal to the distribution server, so that an appropriate image rate is obtained from the distribution server. It is possible to maintain a high level of sound quality and image quality by transmitting moving image data.

  FIG. 2 shows the configuration of the first embodiment. Example 1 in FIG. 2 shows only the configuration in the direction (downward direction) of transmitting multimedia information (voice and video) from the IP network in the multimedia gateway to the mobile terminal of the multimedia mobile terminal network. Is omitted (uplink transmission is the same as in the prior art). In FIG. 2, reference numeral 1 denotes a multimedia gateway, and the multiplexing processing unit 1 a (configured by reference numerals 10 to 12) shown in FIG. 1 is included in the multimedia gateway 1. Reference numerals 10 to 12, 2a to 2d, 3a, and 3b in FIG. 1 have the same names as the same reference numerals in the principle configuration shown in FIG.

  The multiplexed data storage unit 10 in FIG. 2 has a capacity for storing multiplexed data (MUX-PDU: Multiplex Protocol Data Unit) corresponding to 3G-324M for one frame. 2 shows MUX-PDU (multiplexed data) of two frames included. In each MUX-PDU (multiplexed data), 10a is audio data compressed by AMR, 10b is MPEG4 or H.264. 263 (MPEG4 in the following description) is shown. In the first embodiment, one frame composed of audio data and moving image data has a length of 20 ms and a transmission rate of 64 Kbps. Therefore, one frame is composed of 160 B (bytes), and is a combination of 3G-324M audio data (AMR compression code), moving image data (MPEG4), and a certain length of overhead (maintenance management data). The scheduler 11 compresses voice compressed data G.A. In the codec processing unit 2c, control is performed so that the code 711 is converted into an AMR code at a rate set in the multiplexing table 12.

  Each combination is set in the multiplexing table 12 by changing the audio rate and the moving image rate corresponding to each number (referred to as a multiplexing table number), and the scheduler 11 transmits a frame when transmitting multimedia information. A multiplexing table number (referred to as MC) is set in the header (not shown) of (MUX-PDU) and transmitted. When this multiplex table number is received by the mobile terminal, the mobile terminal also has a table similar to the multiplex table 12 of the scheduler 11, and the multiplex table number is determined from the header of the frame sent from the scheduler 11. Are extracted, the audio rate and the moving image rate are identified, and the multiplexed data is separated into audio and moving image data and decoded.

  In the first embodiment, if the audio rate is set in advance in the multiplexing table 12 at a rate corresponding to the minimum speed of 4.75 Kbps (corresponding to 13 bytes / frame), the remaining 147 bytes of the entire frame (160 bytes) Of these, the video rate that can be arranged is the amount excluding the MUX-PDU (multiplexed data) layer and the AL (adaptation layer) overhead. The size of the overhead varies depending on the multiplexing pattern from time to time. However, if the audio as the adaptation layer 2 and the moving image as the adaptation layer 3 are arranged as individual multiplexed data (MUX-PDU) during the 20 ms interval, 18 It becomes a byte. In this case, the video size that can be placed is the remaining 51.6 Kbps (corresponding to approximately 129 bytes / frame), a multiplexing table number is set, and the table number ( (Including the audio and image rates corresponding to the number) and set it in the multiplexing table of the portable terminal. Thus, by selecting the lowest rate as audio AMR encoded data, the image quality level of moving images is kept high.

  FIG. 3 is a processing flow of the first embodiment, and this processing is executed by the scheduler 11. The scheduler determines whether it is the head (voice data) every fixed period of 20 ms (S1 in FIG. 3). If it is the head, the AMR data is output from the voice buffer (2d in FIG. 2), and the multiplexed data (MUX) -PDU) (S2 in FIG. 3). Next, the moving image data is output from the jitter buffer (3b in FIG. 2), arranged next to the AMR data of the multiplexed data (MUX-PDU) (S3 in FIG. 3), and the processing ends. Although the overhead arrangement is omitted in the flow of FIG. 3, it goes without saying that the overhead is arranged in the MUX-PDU of each frame.

  In the first embodiment, it is possible to multiplex audio / moving images at a given rate relatively easily, but it is not possible to change each rate to an arbitrary form and maintain the best sound quality and image quality. In order to cope with this, configurations of the second to seventh embodiments will be described below.

  FIG. 4 shows the configuration of the second embodiment. In the figure, reference numerals 1, 10 to 12, 2a to 2d, 3a, and 3b have the same names as the same reference numerals in the configuration of the first embodiment shown in FIG. Reference numeral 100 denotes a frame imaging process in the scheduler 11.

  In this second embodiment, the codec processing unit checks the amount of moving picture data and confirms that there is a free space in the fixed allocation area, so that the rate corresponding to the free area is allocated to increase the audio rate. 2c (FIG. 4) is controlled.

  In FIG. 4, the scheduler 11 performs the following imaging process before mapping on the multiplexed data storage unit 10 in practice. First, the voice rate is mapped at a fixed rate of 20 ms on the MUX-PDU at a minimum rate of 4.75 Kbps. The rest, excluding overhead, is placed in the video. At that time, the remaining size of the video buffer is checked, and the video is placed within the corresponding fixed interval, leaving a size that remains even after the next fixed interval. . At this time, when there is still an empty area at a fixed interval, the scheduler 11 controls the codec processing unit 2c to increase the audio rate by the amount that satisfies the empty area. When there is no free space, the conversion rate of the codec processing unit 2c remains 4.75 Kbps.

  The audio is converted into a rate for filling the empty area in the codec processing unit 2c, and then arranged in the actual MUX-PDU of the multiplexed data storage unit 10 together with the moving image data extracted from the moving image buffer. In the example of FIG. 4, the AMR conversion rate corresponding to the empty area of frame 1 of the imaging process 100 is 7.95 Kbps, and the AMR conversion rate corresponding to the empty area of frame 2 of the imaging process 100 is 5.90 Kbps. ing.

  Since video data is normally transmitted from the distribution server at the maximum (MAX) value set rate, the average area is lower than the maximum value, and considering the burstiness, if AMR is considered at the minimum rate, the free space is Always occurs.

  FIG. 5 is a processing flow of the second embodiment. In the scheduling process, it is first determined whether or not it is the head (voice data) every fixed period of 20 ms (S1 in FIG. 5), and if it is the head, AMR data is output from the voice buffer and multiplexed data (MUX-PDU) It is arranged at the head (S2 in the same), and then the moving image data is output from the jitter buffer (3b in FIG. 4), and is arranged next to the AMR data of the multiplexed data (MUX-PDU) (S3 in FIG. 5). Next, it is determined whether or not there is an empty area in the MUX-PDU (S4 in FIG. 5). If there is not, the process ends. If there is, the AMR data is re-encoded at a high rate for the empty area (S5 in FIG. 5). , AMR is rearranged at the head, and the moving image is arranged at the next position of AMR (S6). Note that the codec processing unit 2c executes the process of re-encoding AMR data at a high rate.

  As described above, according to the second embodiment, it is possible to improve the voice quality by effectively using the free resources in the MUX-PDU and increasing the voice rate. As a result, the second embodiment can achieve the same multiplexing rate. Regardless, the level of voice quality can be improved.

  FIG. 6 shows the configuration of the third embodiment. In the figure, the reference numerals 1, 11, 12, 2a to 2d, 3a, 3b have the same names as the parts having the same reference numerals in the first and second embodiments shown in FIGS. . Reference numeral 20 provided in the third embodiment is a management table for each channel used in the codec processing unit 2c, and is a management table for each channel for managing the audio rate transmitted to the mobile terminal that uses the channel for each channel. 10-0, 10-1,... Represent the state in which the multiplexed data storage unit 10 (shown in FIGS. 2 and 4) changes corresponding to channel 0, channel 1,. The time division processing is performed for each channel 0, 1, 2,.

  The channel-specific management table 20 is held based on the line information set for each channel in advance and the negotiation result at the time of initial setting from the portable terminal.

  In the third embodiment, the voice packet terminated at the IP / UDP / RTP termination unit 2a is set in correspondence with the channel based on the contents of the channel-specific management table 20 in the codec processing unit 2c via the jitter buffer 2b. After being converted to the rate, it is stored in the audio buffer 2d. Thereafter, the optimum multiplexing table for each channel is selected by the scheduler and mapped as multiplexed data 10-0, 10-1,.

  FIG. 7 is a processing flow of the third embodiment. The scheduler determines whether it is the head (audio data) every fixed period of 20 ms (S1 in FIG. 7), and if it is the head, the codec processing unit refers to the management table for each channel and executes codec conversion ( In step S2), the AMR data obtained by the conversion is output from the audio buffer and placed at the head of the multiplexed data (MUX-PDU) (step S3). Next, a moving image is output from the jitter buffer (3b in FIG. 6), arranged next to the AMR of the MUX-PDU (S4 in FIG. 7), and the process ends.

  According to the third embodiment, a constant audio rate has been transmitted to all the mobile terminals connected to the multimedia gateway until now, but an appropriate audio rate can be transmitted to each mobile terminal. Become. This clears the problem of connectivity specific to the mobile terminal regarding the audio rate.

  FIG. 8 shows the configuration of the fourth embodiment. In the figure, reference numerals 1, 10 to 12, 2a to 2d, 3a, 3b, and 10-0, 10-1, and 20 have the same names as the parts having the same reference numerals in the third embodiment shown in FIG. Is omitted. 30 in the fourth embodiment represents an individual moving image buffer for each channel provided in the moving image jitter buffer 3b, and 31 represents a boundary for changing the multiplexing table by increasing the moving image rate for each channel. A multiplex change threshold value holding unit 32 for holding a moving image rate threshold value, and a boundary for changing the multiplex table beyond the multiplex change threshold value and then returning the original multiplex table to a lower moving image rate. A release threshold value holding unit that holds a threshold value of the moving image rate.

  In the fourth embodiment, the channel-specific management table 20 is provided as in the third embodiment (FIG. 6), and the multiplexing table number corresponding to the combination of the audio rate and the moving image rate is set for each channel, and the jitter buffer 3b. For each individual video buffer 30 corresponding to each channel, a multiplexing change threshold value (a value representing a position corresponding to a specific rate of the video buffer 30) is stored in the multiplexing change threshold holding unit 31 and a cancellation threshold value (video buffer). A value representing a position corresponding to a rate lower than the change threshold value of 30) is set in the release threshold value holding unit 32.

  The scheduler 11 determines the audio rate for each channel in accordance with the channel-specific management table 20 and multiplexes. However, the video rate (data amount) increases even if the video is multiplexed at the remaining audio rate. If the rate does not fit in the remaining rate, for example, if the audio rate is 12.2 Kbps at the maximum but the video rate is the maximum at 50.4 Kbps, the remaining buffer of the video (data that cannot be output) increases. Thus, the threshold of the multiplexing change threshold holding unit 31 is exceeded corresponding to the channel. At this time, the scheduler 11 tries to keep the selection within the 64 Kbps by changing the selection of the multiplexing table to a setting with a higher multiplexing rate than usual. However, if only the change of the multiplexing table still exceeds the threshold value of the individual moving image buffer 30, for example, when the audio rate is 12.2 Kbps and the moving image rate is 50.4 Kbps, the scheduler 11 further When audio data and moving picture data are multiplexed, an adaptation layer header, serial number, error check data, and the like are added. These data can be selected as an option. In the fourth embodiment, when the threshold value of the multiplexing change threshold value holding unit 31 is exceeded, the overhead of this option is reduced as much as possible to increase the moving image rate. Specifically, in terms of the above audio / video rate, if both audio and video are multiplexed without an adaptation layer 2 (AL2) and MUX-PDU option header within an interval of 40 ms = 320 bytes, the header overhead is 160 bytes. It will be 3 bytes in length, and 50.4 Kbps video will be stored.

  The release of exceeding the threshold value of the individual buffer is detected when the data amount of the moving image buffer 30 is set in the release threshold value holding unit 32 and falls below the threshold value. In this case, the scheduler 11 returns the adaptation layer 2 (AL2) and MUX-PDU headers to normal, and in some cases, returns the setting of the multiplexing table to normal. As a result, the header option is restored and the multiplexing rate is lowered to maintain the error occurrence / detection frequency to the normal level.

  FIG. 9 is a processing flow of the fourth embodiment. The scheduler determines whether it is the head (audio data) every fixed period of 20 ms (S1 in FIG. 9), and if it is the head, determines whether the video buffer has exceeded a threshold value (threshold value of the multiplexing change threshold value holding unit 31). (S2). If not exceeded, the video buffer determines whether the threshold release condition is satisfied (below the threshold value of the release threshold value holding unit 32) (S3 in FIG. 9). Is restored, the option header is restored and restored (S4). If it is determined in step S2 that the moving image buffer has exceeded the threshold, the MUX-PDU multiplexing efficiency is increased by selecting the optimum multiplexing table and deleting the option header (S5 in FIG. 9). A PDU is generated (S6).

  FIG. 10 shows the configuration of the fifth embodiment. In the figure, reference numerals 1, 10 to 12, 2a to 2d, 3a, and 3b have the same names as the respective parts having the same reference numerals in Examples 1 to 4 in FIGS. Description is omitted. Reference numeral 13 denotes a voice evaluation data generation unit for storing voice evaluation data set in the multiplexed data storage unit 10 and sent to a portable terminal as a transmission destination. Reference numeral 14 denotes 3G- (upward) transmitted from the portable terminal. A 3G-324M termination unit that terminates a 324M signal, 15 is a feedback result analysis unit, and 16 is a CPU that performs call processing by transmitting and receiving call control data via MEGACO.

  In the fifth embodiment, the scheduler 11 adds predetermined data for audio quality evaluation during transmission of real-time audio / video data, transmits it, receives the evaluation result in the portable terminal, and is fed back. The audio rate is adjusted in the codec processing unit 2c according to the evaluation result. The voice evaluation of the fifth embodiment uses a method called PSQM (Perceptual Speech Quality Measure) recommended by ITU-T (International Telecommunication Union Telecommunication Standardization Sector). This method compares the audio sample used for testing with the audio on the receiving side that has arrived via the network, and quantifies the degree of degradation of the receiving audio relative to the sample.

  In FIG. 10, the scheduler 11 arranges audio at a fixed interval of 20 ms at the AMR minimum rate of 4.75 Kbps, and the remaining area (apart from the overhead) is used for video data and a predetermined audio evaluation for audio evaluation. Data for voice (sample of voice used for test) is arranged from the voice evaluation data generator 13. Since the audio evaluation data is multiplexed for the area where video data should be placed, the video rate is somewhat reduced, but there is no need to multiplex the audio evaluation data within all fixed intervals. Since multiplexing is performed at intervals of a constant period t1 (t1 = 20 ms × n), the influence on the moving image is extremely small. When the voice evaluation data (voice sample) is received by the other party's mobile terminal, the evaluation is performed by using a voice evaluation algorithm that compares the received voice evaluation data with a previously stored voice sample. The multimedia information in the direction is sent to the multimedia gateway by in-channel such as adaptation layer 2 (AL2).

  The evaluation result is extracted by the 3G-324M termination unit 14 of the multimedia gateway, the feedback result is analyzed by the feedback result analysis unit 15, and if it is necessary to raise or lower the audio rate, the codec processing unit 2c is notified accordingly. The codec processing unit 2 c raises or lowers the AMR audio rate according to the analysis result from the feedback result analysis unit 15. At this time, moving image data may be affected by increasing the audio rate (increasing the amount of audio data). However, the amount of moving image data may be reduced by using a multiplexing table, or the fourth embodiment described above. This can be dealt with by using a method of controlling the moving image buffer of FIG. 8 and FIG. 9 with a threshold.

  FIG. 11 is a processing flow of the fifth embodiment. In this processing flow, when the feedback from the portable terminal is received, the scheduler determines whether the feedback result is worse than the previous speech evaluation value (S1 in FIG. 11). The compression rate is instructed to be increased (highly compressed) (S2), and if it is bad, the codec processing unit is instructed to decrease (lowly compress) the AMR compression rate (S3). When the moving image rate is compressed by the audio rate in accordance with the low compression instruction (the moving image data area is reduced), the moving image buffer control (the control shown in the fourth embodiment shown in FIGS. 8 and 9) is also used. (S4 in FIG. 11). However, the moving image buffer control in step S4 can be dealt with by simply changing the number of the multiplexing table to the combination number with the moving image rate lowered without depending on the processing of the fourth embodiment.

  FIG. 12 shows the configuration of the sixth embodiment. In the figure, reference numerals 1, 10 to 16, 2a to 2d, 3a, and 3b have the same names as those of the same reference numerals in the fifth embodiment shown in FIG. Reference numeral 17 denotes an image evaluation data generation unit, 18 denotes an image feedback analysis unit, and 4 denotes a SIP (Session Initiation Protocol) server.

  In the sixth embodiment, in addition to the predetermined data for voice evaluation used in the fifth embodiment (FIGS. 10 and 11), further predetermined image evaluation data is multiplexed and transmitted to the portable terminal. To do. The image evaluation of the sixth embodiment uses a marker embedded type absolute evaluation method. In this method, an invisible signal (marker) is embedded in the original image to be evaluated, and image quality estimation is performed on the receiving side based on the deterioration of the marker signal. As a result, the receiving side can perform absolute evaluation that does not require an image as a comparison source to be evaluated.

  Multiplexing is performed from the image evaluation data generation unit 17 at an interval of a certain period t2 (t2 = 20 ms × m). It can be sent independently from the voice evaluation data from the voice evaluation data generating unit 13. The other party's mobile terminal performs voice evaluation using the voice evaluation data, and at the same time, performs image evaluation using the image evaluation data, and uses the result as an adaptation layer 2 (AL2) or the like different from the voice evaluation result. To feed back to the multimedia gateway. The multimedia gateway separates the feedback result of the image evaluation from the moving image data by the 3G-324M terminal unit 14, analyzes the content by the image feedback analysis unit 18, and supplies the CPU 16 with the image rate increase / decrease information according to the feedback result. . This information is transferred from the CPU 16 to the SIP server 4 over the MEGACO protocol, and the evaluation information is fed back from the SIP server 4 to the multimedia distribution server (85 in FIG. 13). Convert and resubmit.

  According to the sixth embodiment, not only audio data but also a moving image can be transmitted while maintaining a high level of sound quality and image quality by feeding back an actual evaluation result on the mobile terminal side.

  (Supplementary note 1) IP-compatible audio from a distribution server provided at a connection point between a multimedia mobile terminal network accommodating mobile terminals capable of transmitting and receiving audio and video and an IP network to which the multimedia distribution server is connected Terminate packet and image packet, convert audio signal to AMR (adaptive variable rate) audio data of multimedia portable terminal network, terminate image packet and extract moving image data, and save audio data, moving image data and overhead In the audio / video adjustment method in the multimedia gateway having a multiplexing processing unit that assembles fixed-length multiplexed data at a rate of 64 Kbps and outputs the multiplexed data to the multimedia portable terminal network at a fixed period, the multiplexing processing unit A scheduler for controlling the generation of digitized data, and the voice rate and the dynamics within each fixed length. A multiplexing table storing rate combinations, and setting the multiplexing table in the multiplexing table so that the audio rate is the lowest rate and the remaining capacity is for video and overhead for transmission to each mobile terminal. , The scheduler generates multiplexed data using the audio rate data of the multiplexing table and the moving image rate, and the audio / video adjusting method in the multimedia gateway.

  (Additional remark 2) In additional remark 1, it has a means to detect the empty area of the multiplexed data storage part which stores the multiplexed data produced | generated by the said multiplexing process part, and detects an empty area by the means to detect the said empty area Then, the scheduler controls the means for converting to the AMR audio data of the multimedia portable terminal network so as to increase the audio rate of the minimum rate by an amount corresponding to the empty area. Audio / video adjustment method at the gateway.

  (Additional remark 3) In the additional remark 1, about the portable terminal which needs to hold | maintain a specific audio | voice rate according to an audio | voice reception characteristic or reception environment conditions, the said specific audio | voice rate is set to the multiplexing table according to the channel of this portable terminal. The scheduler adjusts according to the audio rate set in the corresponding channel of the multiplexing table when multiplexing the moving image at the predetermined intervals.

  (Supplementary note 4) The multiplexing change threshold value according to supplementary note 1, further comprising a channel-compatible moving image buffer for holding moving image data obtained by terminating an IP-compatible image packet from a distribution server, wherein a multiplexing change threshold value is set for each channel. When the scheduler detects that the video rate of each channel has increased and has exceeded the correspondingly set multiplexing change threshold, it reduces the adaptation layer option overhead and reduces the video rate. An audio / video adjustment method in a multimedia gateway characterized by increasing

  (Supplementary note 5) In Supplementary note 4, the scheduler includes a release threshold value holding unit in which a release threshold value is set in correspondence with a channel-compatible video buffer that holds video data obtained by terminating an IP-compatible image packet from a distribution server. The multimedia gateway characterized in that the adaptation layer overhead is returned to normal when the image rate decreases and falls below the release threshold after the image rate of the video buffer corresponding to the channel exceeds the multiplexing change threshold. Audio / video adjustment method in Japan.

  (Supplementary note 6) In any one of Supplementary note 1 or Supplementary note 4, the multiplexing processing unit is provided with a voice evaluation data generating unit that generates voice quality evaluation data, and the scheduler includes voice data, moving image data, When assembling the overhead, the data of the voice evaluation data generating unit is multiplexed and output at a rate of once every a plurality of times of the fixed period for outputting the multiplexed data. When a voice evaluation result using the voice evaluation data is received, a voice rate of the multiplexed data to the mobile terminal is adjusted according to the voice evaluation result. Adjustment method.

  (Supplementary note 7) In Supplementary note 6, the multiplexing processing unit is provided with an image evaluation data generating unit that generates image evaluation data, and the scheduler outputs the multiplexed data when assembling moving image data. The data of the image evaluation data generating unit is multiplexed and output at a rate of once per a plurality of times of a fixed period, and the scheduler is information on the result of image evaluation using the image evaluation data from the mobile terminal side. The image evaluation result information is transmitted to the multimedia distribution server connected to the IP network via a path for transferring call control information between the IP network and the multimedia gateway, and the multimedia distribution server According to the result of the image evaluation, the image rate is adjusted and transmitted to the IP network. Audio and video adjustment method in and breakfasts gateway.

It is a figure which shows the principle structure of this invention. 1 is a diagram illustrating a configuration of Example 1. FIG. FIG. 3 is a diagram illustrating a processing flow of Example 1. 6 is a diagram illustrating a configuration of Example 2. FIG. FIG. 6 is a diagram illustrating a processing flow of Example 2. 6 is a diagram illustrating a configuration of Example 3. FIG. FIG. 10 is a diagram illustrating a processing flow of Example 3. FIG. 10 is a diagram showing a configuration of Example 4. FIG. 10 is a diagram illustrating a processing flow of Example 4. FIG. 10 is a diagram showing a configuration of Example 5. FIG. 10 is a diagram illustrating a processing flow of Example 5. 10 is a diagram illustrating a configuration of Example 6. FIG. It is a figure which shows the structure of the system containing the conventional multimedia gateway. It is a figure which shows the structure of the multimedia gateway used as the object of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1a Multiplexing process part 10 Multiplexed data storage part 11 Scheduler 11a Initial setting means 11b Monitoring means 11c Multiprocessing means 12 Multiplexing table 2a Termination part 2b Jitter buffer 2c Codec (CODEC) processing part 2d Voice buffer 3a Termination part 3b Jitter buffer

Claims (5)

IP-compatible voice packets and image packets from the distribution server provided at the connection point between the multimedia portable terminal network containing portable terminals capable of transmitting and receiving voice and video and the IP network to which the multimedia distribution server is connected The audio signal is converted into AMR (adaptive variable rate) audio data of the multimedia portable terminal network, the video packet is terminated and the moving image data is taken out, and the audio data, moving image data and overhead are transmitted at a speed of 64 Kbps. In an audio / video adjustment method in a multimedia gateway having a multiplexing processing unit that assembles fixed-length multiplexed data and outputs it to the multimedia portable terminal network at a fixed cycle,
The multiplexing processing unit includes a scheduler that performs control to generate multiplexed data, and a multiplexing table that stores combinations of audio rates and video rates within each fixed length,
For the transmission to each mobile terminal, set the multiplexing table so that the audio rate is the lowest rate and the remaining capacity is for video and overhead in the multiplexing table,
The audio / video adjustment method in a multimedia gateway, wherein the scheduler generates multiplexed data using the audio rate data and the video rate of the multiplexing table. In claim 1,
Means for detecting an empty area of a multiplexed data storage unit for storing multiplexed data generated in the multiplexing processing unit;
When the vacant area is detected by the means for detecting the vacant area, the scheduler converts the lowest audio rate to the amount that satisfies the vacant area, with respect to the means for converting the audio data of the multimedia portable terminal network to AMR. An audio / video adjustment method in a multimedia gateway characterized by controlling to increase. In claim 1,
For a portable terminal that needs to hold a specific voice rate according to voice reception characteristics and reception environment conditions, the specific voice rate is set in a multiplexing table for each channel of the portable terminal,
The scheduler adjusts according to the audio rate set to the corresponding channel of the multiplexing table when multiplexing the moving image at the predetermined intervals, and adjusting the audio / moving image in the multimedia gateway. In claim 1,
A video buffer corresponding to a channel that holds video data obtained by terminating an IP-compliant image packet from a distribution server, and a multiplexing change threshold holding unit that sets a multiplexing change threshold corresponding to each channel;
When the scheduler detects that the video rate of each channel has increased and has exceeded the correspondingly set multiplexing change threshold, the scheduler can reduce the adaptation layer option overhead and increase the video rate. Audio / video adjustment method in the featured multimedia gateway. In claim 1,
The multiplexing processing unit is provided with a voice evaluation data generation unit for generating voice quality evaluation data,
When assembling audio data, moving image data, and overhead, the scheduler multiplexes and outputs the data of the audio evaluation data generating unit at a rate of once every a plurality of times of the fixed period for outputting the multiplexed data. ,
When the scheduler receives a result of voice evaluation using the voice evaluation data from the mobile terminal side, the scheduler adjusts a voice rate of the multiplexed data to the mobile terminal according to the result of the voice evaluation. Audio / video adjustment method for multimedia gateways.

Images