CN115691516B - Low-delay audio matrix configuration method and server - Google Patents

Abstract

The invention discloses a low-delay audio matrix configuration method and a server, wherein the method comprises the following steps: s1: driving the input equipment to establish connection with the output equipment, and performing audio data interaction between the input equipment and the output equipment; s2: the hardware operation rate, the current network delay condition and the audio data packet processing amount of each playing device are obtained, the audio data interception amount is calculated, and the output device is driven to intercept corresponding audio data and then sent to the playing device. In the invention, the hardware operation rate, the current network delay condition and the processing capacity of the audio data packet of each playing device are fully considered, and the quantity to be intercepted is calculated in real time, so that the data to be processed of the devices with different network delay conditions are dynamically intercepted to different degrees, a plurality of devices can realize synchronization, the requirement of synchronously playing the audio for the plurality of devices in a professional scene is met, and the delay sense caused by the network environment is eliminated.

Description

A low-latency audio matrix configuration method and server

technical field

The invention relates to the technical field of audio processing, in particular to a low-delay audio matrix configuration method and server.

Background technique

With the advancement of social informatization, all walks of life have higher and higher requirements for the audio output of audio matrices in professional scenes such as medium and large stage conferences, studios and studios, including but not limited to accurate output of output sources. performance, no delay, high-quality mixing effects and complex audio preemption logic.

At present, the existing technology is only aimed at splitting audio data packets, splitting large packets of audio data into multiple small packets, so as to reduce the delay to a certain extent, but in the network or device environment, there will be network devices and so on. The delay problem caused by the series environment, and the existing technology uses the device with the highest delay time as the adjustment standard, delays other data synchronization to the highest time, and sends the delay synchronously, which does not fundamentally solve the delay problem, resulting in In some professional scenarios, there will be a significant delay in audio playback.

Contents of the invention

In order to overcome the deficiencies of the prior art, one of the purposes of the present invention is to provide a low-latency audio matrix configuration method, which can solve the problem of obvious delay in the transmission of audio data under the influence of the network environment in professional scenarios.

The second object of the present invention is to provide a low-latency audio matrix configuration server, which can solve the problem of obvious delay in the transmission of audio data under the influence of the network environment in professional scenarios.

In order to achieve one of the above-mentioned purposes, the technical scheme adopted in the present invention is as follows:

A low-latency audio matrix configuration method, comprising the following steps:

S1: Drive the input device to establish a connection with the output device, and perform audio data interaction between the input device and the output device;

S2: Obtain the hardware operation rate of each playback device, the current network delay and the processing volume of audio data packets, calculate the amount of audio data interception, drive the output device to intercept the corresponding audio data, and then send it to the playback device.

Preferably, said S2 is specifically implemented by the following steps:

S21: Obtain the current network delay of each playback device;

S22: Determine whether the current network delay is greater than or equal to the preset delay value, if not, drive the output device to send the audio data to the playback device, if so, execute S23;

S23: Obtain the hardware computing rate and audio data packet processing volume of the playback device, and calculate the audio data interception volume;

S24: Drive the output device to intercept corresponding audio data, and then send it to the playback device.

Preferably, said S1 is specifically implemented by the following steps:

Send audio configuration information to the input device and output device, drive the input device and output device to establish a connection through the audio configuration information, and perform audio data interaction.

Preferably, the following steps are further included between S1 and S2:

A1: Drive the input device to remove the echo from the audio frame, and perform noise reduction processing, identify and filter background noise;

A2: Determine whether the input device currently obtains audio data, if not, encode to generate a blank encoded frame, and send it to the output device, if so, execute A3;

A3: Determine whether to perform audio mixing processing, if yes, attenuate the voice through the attenuation factor, if not, parse the audio data into PCM audio stream data, and perform noise detection processing.

In order to achieve two of the above-mentioned purposes, the technical scheme adopted in the present invention is as follows:

A low-latency audio matrix configuration server, including a memory and a processor;

a memory for storing program instructions;

The processor is configured to run the program instructions to execute the above-mentioned low-latency audio matrix configuration method.

Compared with the prior art, the beneficial effect of the present invention is that: by fully considering the hardware operation rate of each playback device, the current network delay situation and the processing volume of audio data packets, and calculating the quantity to be intercepted in real time, thereby different network delays Different degrees of dynamic interception of data to be processed by the device in different situations, so that multiple devices can be synchronized to meet the needs of multiple devices to play audio synchronously in professional scenarios, and eliminate the sense of delay caused by the network environment.

Description of drawings

FIG. 1 is a flow chart of the low-latency audio matrix configuration method described in the present invention.

Detailed ways

The preferred embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described here are only used to illustrate and explain the present invention, and are not intended to limit the present invention.

In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" etc. The indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, or in a specific orientation. construction and operation, therefore, should not be construed as limiting the invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and should not be construed as indicating or implying relative importance.

In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection. Connected, or integrally connected; it can be mechanically connected or electrically connected; it can be directly connected or indirectly connected through an intermediary, and it can be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention in specific situations.

Below, in conjunction with accompanying drawing and specific embodiment, the present invention is described further:

In the present invention, the host (equivalent to the server), the input device and the output device are constructed into an audio matrix, the input device can be a TX-8770 encoder, the output device can be a TX-8704 decoder, the host, Both the input device and the output device are connected to the same network.

Embodiment one:

As shown in Figure 1, a low-latency audio matrix configuration method includes the following steps:

S1: Drive the input device to establish a connection with the output device, and perform audio data interaction between the input device and the output device;

Specifically, by driving the input device to establish a connection with the output device, the input device transmits the audio data collected from the audio source to the output device, and the output device transmits the audio data to the playback device. In this embodiment, the S1 Specifically, it is realized by the following steps:

Send audio configuration information to the input device and output device, drive the input device and output device to establish a connection through the audio configuration information, and perform audio data interaction.

Preferably, after the host (server) service starts normally, it sends the UDP broadcast protocol in the network environment. After the input device and the output device receive the UDP broadcast protocol, they send the online request protocol to the host, and the host receives the equipment of the online protocol of each device. Finally, mark the device configuration information, and maintain the healthy heartbeat communication with each device, set the device information, matrix configuration, preemption logic configuration and audio configuration in the host, according to the address of the input device and output device, the host sends audio configuration information ( Equivalent to the configuration protocol signal) to the input device and output device, the input device configures according to the received audio configuration information, outputs the audio data to the output source device, and the output device configures, verifies and accepts the input device transmission according to the received audio configuration information The audio data, and then each channel of the output device outputs the received audio data to the playback device.

If the host is offline, the input device and output device can still transmit audio data according to the audio configuration information. If the input device and output device experience abnormal conditions such as power failure or network disconnection, after the device is restarted, the host will re-send the audio configuration information. Input The audio data interaction between the device and the output device returns to normal.

Further, the following steps are also included between S1 and S2:

A1: Drive the input device to remove the echo from the audio frame, and perform noise reduction processing, identify and filter background noise;

Specifically, conventional audio processing such as echo cancellation, noise suppression, and silence detection is performed on the audio data through the input device. Preferably, according to the audio frame just played, some operations similar to cancellation are performed in the collected audio frame, so that the echo Remove it from the collected audio frame; when performing noise reduction processing, identify the part that belongs to the background noise and filter it out from the audio frame.

A2: Determine whether the input device currently obtains audio data, if not, encode to generate a blank encoded frame, and send it to the output device, if so, execute A3;

Specifically, through the silence detection algorithm combined with the previous noise suppression algorithm, it can be identified that there is no audio source input to the input device. If not, the input device can encode a special encoded frame, such as a blank encoded frame, and send it to the output. Devices can save bandwidth and traffic.

A3: Determine whether to perform audio mixing processing, if yes, attenuate the voice through the attenuation factor, if not, parse the audio data into PCM audio stream data, and perform noise detection processing.

Specifically, when performing sound mixing, a normalized sound mixing algorithm (adaptive weighted sound mixing algorithm) is used for sound mixing processing, and more bits (32 bits) are used to represent a sample of audio data. Find a way to reduce its amplitude so that it is still distributed within the range that 16bit can represent. To avoid overflow, the speech is attenuated using a variable attenuation factor. This attenuation factor also represents the weight of the voice, and the attenuation factor changes with the change of audio data, so it is called adaptive weighted mixing. When there is overflow, the attenuation factor is small, so that the overflowed data can be within the critical value after attenuation, and when there is no overflow, the attenuation factor is slowly increased, so that the data changes more gently.

When not mixing, the AEC algorithm based on deep learning is adopted, and the combination of traditional signal processing and deep learning technology is used to perform further residual echo suppression after echo cancellation adaptive filtering. Inside the program, the audio is parsed into PCM audio stream data. Perform a series of noise detection processing, adaptively gain the sound quality to a stable frequency range, reduce the delay in transmission, and improve the audio quality.

S2: Obtain the hardware operation rate of each playback device, the current network delay and the processing volume of audio data packets, calculate the amount of audio data interception, drive the output device to intercept the corresponding audio data, and then send it to the playback device.

Specifically, before the audio data is sent, according to the delay between each playback device in the network and the device processing data rate, the audio data interception amount is calculated, and the real-time synchronous low-delay processing is sent. In this embodiment, the S2 Specifically, it is realized by the following steps:

S21: Obtain the current network delay of each playback device;

Specifically, the current network delay situation can be obtained by directly pinging the network delay degree of each terminal device (playing device) stored in the server.

S22: Determine whether the current network delay is greater than or equal to the preset delay value, if not, drive the output device to send the audio data to the playback device, if so, execute S23;

Specifically, whether the audio data needs to be processed is measured by judging whether the current network delay is greater than the preset delay value. Preferably, if the current network delay is less than 1 ms, it is considered normal, and the current second delay of each terminal is recorded If the current network delay is greater than 1ms, the audio data needs to be intercepted.

S23: Obtain the hardware computing rate and audio data packet processing volume of the playback device, and calculate the audio data interception volume;

Specifically, calculate and obtain the computing speed of the chip of the current device, because there is a specific private protocol between the device and the server, which will keep the heartbeat data in real time, and the heartbeat packet stores the basic information of the current device, such as network conditions, temperature, and current processing rate , the calculation formula of the rate is as follows: data amount (byte/second) = sampling rate (Hz) * sampling bit (bit) * chip sampling channel number / 8, where 1Byte = 8bit, 12-bit sampling bit, 4 channels, 200K sampling frequency AD chip 1 second data size is: 200000*12*4/8＝1200000Byte＝1171.875K＝1.14441M; That is to say, the transmission rate is 1200000Byte/s per second, 1200byte/ms per millisecond, such as at the same time After receiving other data, read and store the parameters of the remaining data that can be processed into the server, and then calculate the data packets per second of the audio, as follows: set audio sampling rate = 48000, sampling channel = 2, Bit depth = 16, sampling interval = 20ms, that is, the total amount of data in one second, the sampling interval is 20ms, 50 times per second (1s = 1000ms), then the total amount of data is calculated as:

The total amount of data in one second=44100*2*16/8=192000(Byte)

Audio data size per frame = 192000/50 = 3840 (Byte)

Number of samples per channel = 320/2 = 160 (Byte)

The amount of data per millisecond = 192000/1000 = 192 (Byte)

Then according to the hardware operation rate of the device playback device and the current network delay, for example, the current processing rate of playback device A and the remaining energy processing capacity: 5000byte/ms and the current network delay is 1ms, then the data processing rate of 5000byte/ms> 192byte /ms, normal data transmission;

If the current processing rate of playback device B is 100byte/ms and the current network delay is 2ms, that is, the current delay is 1ms, and the processing rate is 100byte<192byte, then the total data required to be processed is 192+192 -100=284byte;

If the current processing rate of the playback device C is the remaining capacity: 1200byte/ms and the current network delay is 3ms, that is, the current delay is 2ms, so the current audio data is 192*2=386byte, and the processing rate of the device itself is 1200byte /ms>192byte/ms, so the total number that should be processed is 386byte;

To sum up, the final sent data result is to split the 192000byte/s data into 512byte data into one packet, and send 375 times per second. In this embodiment, in the ordinary LAN environment, the data transmitted by udp is the largest 1472 bytes is the best (to avoid fragment reassembly). However, in network programming, routers in the Internet may have different values (less than the default value), and the standard MTU value on the Internet is 576, so the data length of UDP programming on the Internet is preferably 576-20-8=548 Within bytes, and because 512 is 2 to the 9th power, the computer is binary. Therefore, all multiplications and divisions related to 2 can be converted into shift operations, which can greatly reduce the computational burden.

S24: Drive the output device to intercept corresponding audio data, and then send it to the playback device.

In this embodiment, if playback device A sends 192000 byte data normally, the first packet of data is sent normally; playback device B sends 192000-284=191716 byte in one second, intercepts the data of 284 bytes in front of the data, and starts from the 284th byte Start counting to the 192000th data and send the first packet of data; the playback device C sends 192000-386=191614byte data in one second, intercepts the 386 bytes of data in front of the data, and counts from the 386th to the 192000th data A packet of data is sent 512-386=126byte; so after the audio data interception is completed, the time error of playback devices A, B and C is resolved, and their data is synchronized, and the sound playback and sound input are 1ms Error, so it does not affect the normal use, although it will cause the device to have a certain amount of audio data, but the sound fade-in effect is done, the effect is small to large, and the time is completely enough for the audio to be sent out synchronously without affecting the use. The audio data packets are adjusted and calculated in real time according to the network fluctuations and the processing of the device itself. In the professional studio scene, the delay between the sound source coming in and the sound coming out is kept within 1ms.

Furthermore, the audio data is uniquely identified and the data is encrypted with salt and data identification, so that the data will not be captured and intercepted during transmission, which will interfere with the continuous audio output of the target device) The preemption logic can be found on the client page Call the audio matrix system under different scenes. There are priorities in the scene. The one with high priority can preempt the one with low priority. When audio preemption is configured, it will be preempted according to the priority value of the input channel configuration, and the priority is high. The preemptible priority is low. When audio preemption is not configured, each audio channel will be mixed or unmixed.

In addition, the system considers that specific scenes need to switch audio at any time, output different audio, and also provide IO trigger to switch scenes, realize offline operation, configure each required scene matrix point on the host, corresponding to the device encoder (input Device) different IO trigger ports, and then offline use is convenient for users to switch. In a specific place, if you want to switch scenes in time, you can bypass the host and directly trigger the IO port of the device encoder to configure the device scene , switch the audio processing between different encoders and decoders, apply different audio source inputs, and output different audio. The process is as follows:

Scene 1: There are encoder 1, encoder 2, decoder 1, and decoder 2. After configuring the output channel and input channel corresponding to each matrix point, save it. In encoder 1, link the channel 1 triggered by IO Go to scene 2, save

Scene 2: There are encoder 1, encoder 3, decoder 1, and decoder 2. After configuring the output channel and input channel corresponding to each matrix point, save it. In encoder 1, link the channel 2 triggered by IO Go to scene 1, save.

At the project site, according to the actual needs, the IO channel port of the encoder 1 is short-circuited to trigger the switching of the saved scene or the scene switching is performed on the computer side, and different audio is applied to the scene, and the specified audio source is input and output.

Embodiment two:

A low-latency audio matrix configuration server, including a memory and a processor;

a memory for storing program instructions;

The processor is configured to run the program instructions to execute the low-latency audio matrix configuration method as described in Embodiment 1.

Those skilled in the art can make various other corresponding changes and deformations according to the above-described technical solutions and concepts, and all these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims (4)

1. A method of low delay audio matrix configuration comprising the steps of:

s1: driving the input equipment to establish connection with the output equipment, and performing audio data interaction between the input equipment and the output equipment;

s2: obtaining the hardware operation rate, the current network delay condition and the audio data packet processing amount of each playing device, calculating the audio data interception amount, driving the output device to intercept corresponding audio data, then sending to the playing device,

the step S2 is specifically realized by the following steps:

s21: acquiring the current network delay condition of each playing device;

s22: judging whether the current network delay condition is greater than or equal to a preset delay value, if not, driving the output equipment to send the audio data to the playing equipment, and if so, executing S23;

s23: the hardware operation rate q and the processing amount of the audio data packet of the playing device are obtained, the interception amount of the audio data is calculated,

wherein the processing amount m of the audio data packet represents the residual energy processing amount m of the playing device based on the current hardware operation rate q, the residual energy processing amount m represents the processable data amount in unit time, each playing device calculates and obtains the corresponding audio data interception amount based on the corresponding operation rate q, the residual energy processing amount m and the current network delay n,

if m > (n-1) q, the audio data interception amount is (n-1) q,1 represents 1 unit time,

if m is less than (n-1) q, the interception amount of the audio data is nq-m;

s24: and driving the output device to intercept corresponding audio data and then sending the audio data to the playing device.

2. The low-delay audio matrix configuration method according to claim 1, wherein S1 is specifically implemented by the steps of:

and sending audio configuration information to the input device and the output device, and driving the input device and the output device to establish connection through the audio configuration information so as to perform audio data interaction.

3. The low-delay audio matrix configuration method according to claim 1, wherein the steps between S1 and S2 further include:

a1: driving an input device to remove echoes from the audio frame, performing noise reduction treatment, and identifying and filtering background noise;

a2: judging whether the input equipment acquires the audio data currently, if not, encoding to generate a blank encoding frame, transmitting the blank encoding frame to the output equipment, and if so, executing A3;

a3: judging whether to carry out the audio mixing processing, if so, attenuating the voice by an attenuation factor, otherwise, analyzing the audio data into PCM audio stream data, and carrying out noise detection processing.

4. A low-latency audio matrix configuration server, characterized by: comprising a memory and a processor;

a memory for storing program instructions;

a processor for executing the program instructions to perform the low delay audio matrix configuration method of any of claims 1-3.