CN111212032B - Audio processing method, device, electronic equipment and storage medium based on Internet of Vision - Google Patents

Abstract

The embodiment of the present application provides an audio processing method, device, electronic equipment, and storage medium based on the Internet of Vision. In the Internet of Vision, an Internet-of-Vision terminal is deployed, and the Internet-of-Vision terminal communicates with a streaming media server. Communication connection, the mobile terminal is equipped with an audio playback component, the method is applied to the application object set in the mobile terminal, including: when it is detected that the preset audio and video call service is turned on, triggering the preset audio on the mobile terminal Collection mode; receive the first audio data sent by the streaming media server, and call the audio playback component to play the first audio data; obtain the second audio data collected by the mobile terminal in the audio collection mode; according to the first audio data and For the second audio data, echo cancellation processing is performed on the second audio data to obtain target audio data after echo cancellation processing. The application can improve the audio communication quality of the video network conference.

Description

Audio processing method, device, electronic equipment and storage medium based on Internet of Vision

technical field

The present application relates to the technical field of data processing, in particular to an audio processing method, device, electronic equipment and storage medium based on the Internet of Vision.

Background technique

At present, with the popularization and development of the Internet of Vision business across the country, the Internet of Vision high-definition Internet of Vision interactive technology plays a pivotal role in government departments and other industries. The Internet of Vision adopts the world's most advanced VisionVera real-time high-definition video switching technology, which realizes the real-time transmission of high-definition video on the entire network that cannot be realized by the Internet at present, and integrates high-definition video conferencing, video monitoring, remote training, intelligent monitoring and analysis, emergency command, video telephony, Dozens of video, voice, picture, text, communication, data and other services such as live broadcast, TV mail, and information release are all integrated into one system platform, and real-time interconnection of high-definition quality video communication is realized through a variety of terminal devices.

With the widespread use of video conferences on the Internet of Things, some video conferences on the Internet of Things are carried out under the environment of the Internet of Things and 4G networks. For example, when a UAV joins a meeting, the UAV is usually connected to a mobile phone, and the UAV is controlled on the mobile phone through the application in the Internet of Vision installed on the mobile phone. At the same time, the application in the Internet of Vision and the command hall The Internet of Things terminal can make audio and video calls. Generally, in the Internet-of-Vision video conference where drones participate in the meeting, the participants of the drone are usually in the wild environment. In this way, the user usually connects a microphone to the mobile phone. When the microphone is connected, the mobile phone plays the voice of the other party. While collecting with a microphone. However, after the mobile phone plays the voice of the other party, the voice will generate an echo, and then the echo will be transmitted to the Internet of Vision terminal together with the newly collected voice. In this way, when the Internet-of-Vision terminal in the command hall is playing the returned sound, the other party will hear the echo that they themselves sent in the previous call.

In the prior art, in order to suppress the echo in the Internet-based video conferencing, the general way is to set a time interval so that the human ear cannot distinguish the echo from the newly collected sound, but this method cannot completely eliminate the echo. Echo has a high requirement for setting the time interval, and because this loop continues continuously, the echo will accumulate more and more, and finally there will be a buzzing sound, which will affect the quality of the call.

Contents of the invention

In view of the above problems, the embodiments of the present application are proposed to provide an audio processing method, device, electronic device and storage medium based on the Internet of Vision that overcomes the above problems or at least partially solves the above problems.

In the first aspect, the embodiment of the present application provides an audio processing method based on the Internet of Vision, in which an Internet of Vision terminal is deployed, and the Internet of Vision terminal communicates with a streaming media server, and the streaming media server communicates with a mobile A terminal communication connection, the mobile terminal is configured with an audio playback component, and the method is applied to an application object set in the mobile terminal, including:

When it is detected that the preset audio and video call service is turned on, trigger the preset audio collection mode on the mobile terminal;

receiving the first audio data sent by the streaming media server in the audio and video call service, and calling the audio playback component to play the first audio data; wherein, the first audio data is provided by the The video network terminal sends to the streaming media server;

acquiring second audio data collected by the mobile terminal in the audio collection mode;

performing echo cancellation processing on the second audio data according to the first audio data and the second audio data, to obtain target audio data after echo cancellation processing;

sending the target audio data to the streaming media server, and the streaming media server is configured to send the target audio data to the Internet of Vision terminal.

Optionally, a first microphone and a second microphone are configured on the mobile terminal to trigger a preset audio collection mode on the mobile terminal, including:

invoke the first microphone and the second microphone;

Obtaining the second audio data collected by the mobile terminal in the audio collection mode includes:

acquiring first microphone audio data collected by the first microphone, and second microphone audio data collected by the second microphone;

Perform noise reduction processing on the second microphone audio data according to the first microphone audio data and the second microphone audio data to obtain second audio data.

Optionally, according to the first audio data and the second audio data, performing echo cancellation processing on the second audio data to obtain target audio data after echo cancellation processing, including:

In the second audio data, determining third audio data corresponding to the first audio data;

The third audio data is filtered out from the second audio data to obtain target audio data from which the third audio data is filtered out.

Optionally, while triggering the preset audio collection mode on the mobile terminal, the method further includes:

calling the adaptive filter set in the mobile terminal;

In the second audio data, determining third audio data having the same frequency as the first audio data includes:

inputting the first audio data into the adaptive filter to obtain output audio data output by the adaptive filter;

Among the second audio data, third audio data having the same frequency as the output audio data is determined.

Optionally, invoking the adaptive filter configured on the mobile terminal includes:

Determining at least one application program interface adapted to the application program object on the Internet of Vision terminal, and determining whether a target interface exists in the at least one application program interface;

When the target interface exists in the at least one application program interface, call the adaptive filter corresponding to the target interface through the target interface;

When the target interface does not exist in the at least one application program interface, call the adaptive filter corresponding to the preset application program interface through the preset application program interface

In the second aspect, the embodiment of the present application provides an audio processing device based on the Internet of Vision, in which an Internet of Vision terminal is deployed, and the Internet of Vision terminal communicates with a streaming media server, and the streaming media server communicates with a mobile Terminal communication connection, the mobile terminal is equipped with an audio playback component, the device is applied to the application program object set in the mobile terminal, the device can specifically be a virtual device, and can specifically include the following modules:

An audio mode trigger module, configured to trigger a preset audio data collection mode on the mobile terminal when it is detected that the preset audio call service is turned on;

An audio data receiving and playing module, configured to receive the first audio data sent by the streaming media server, call the audio playing component, and play the first audio data; the first audio data is provided by the video network The terminal sends to the streaming media server;

an audio data collection module, configured to obtain second audio data collected by the mobile terminal in the audio data collection mode;

An audio data processing module, configured to perform echo cancellation processing on the second audio data according to the first audio data and the second audio data, to obtain target audio data after echo cancellation processing;

An audio data sending module, configured to send the target audio data to the streaming media server, and the streaming media server is configured to send the target audio data to the Internet-of-Video terminal.

Optionally, a first microphone and a second microphone are configured on the mobile terminal, and the audio mode trigger module may specifically be used to invoke the first microphone and the second microphone;

The audio data acquisition module may specifically include the following units:

a microphone audio data acquisition unit, configured to acquire first microphone audio data collected by the first microphone, and second microphone audio data collected by the second microphone;

The noise reduction processing unit is configured to perform noise reduction processing on the second microphone audio data according to the first microphone audio data and the second microphone audio data to obtain second audio data.

Optionally, the audio data processing module may specifically include the following units:

an audio data search unit, configured to determine, among the second audio data, third audio data corresponding to the first audio data;

An audio data filtering unit, configured to filter out the third audio data from the second audio data to obtain target audio data from which the third audio data is filtered out.

Optionally, the device may specifically include the following modules:

A calling module, used to call the adaptive filter set in the mobile terminal;

The audio data search unit may specifically include the following units:

an audio data input unit, configured to input the first audio data into the adaptive filter to obtain output audio data output by the adaptive filter;

The audio data determining unit is configured to determine, among the second audio data, third audio data having the same frequency as the output audio data.

Optionally, the calling module may specifically include the following units:

A target interface determining unit, configured to determine at least one application program interface adapted to the application program object on the Internet-of-Vision terminal, and determine whether there is a target interface in the at least one application program interface;

A first calling unit, configured to call the adaptive filter corresponding to the target interface through the target interface when the target interface exists in the at least one application program interface;

The second calling unit is configured to call the adaptive filter corresponding to the preset application program interface through the preset application program interface when the target interface does not exist in the at least one application program interface.

In the third aspect, the embodiment of the present application also discloses an electronic device, including:

one or more processors; and

One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, causes the device to execute one or more Internet-of-Vision-based Audio processing methods.

In the fourth aspect, the embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program to enable the processor to execute the video-network-based audio processing method described in the embodiment of the present application.

Compared with the prior art, the embodiment of the present application includes the following advantages:

In the embodiment of the present application, when the application object set in the mobile terminal detects that the preset audio and video call service is enabled, the preset audio collection mode is triggered. Afterwards, when receiving the first audio data sent by the streaming media server, the audio playback component on the mobile terminal can be called to play the first audio data, and then the second audio data collected by the mobile terminal in the audio collection mode can be obtained , and perform echo cancellation processing on the second audio data according to the first audio data to obtain target audio data, and then send the target audio data after the echo cancellation processing to the Internet-of-Vision terminal via the streaming media server. Because the application program object can make the mobile terminal collect the second audio data in the preset audio collection mode, and then can improve the audio quality of the second audio data, and because the application is based on the first audio data, the collected second audio data The echo cancellation processing of the audio data can cancel the echo generated by playing the first audio data in the second audio data, thereby improving the echo processing effect of the present application, and further improving the call quality.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments of the present application. Obviously, the accompanying drawings in the following description are only some embodiments of the present application , for those skilled in the art, other drawings can also be obtained according to these drawings without paying creative labor.

Fig. 1 is a kind of networking schematic diagram of the present application;

Fig. 2 is a schematic diagram of the hardware structure of a node server of the present application;

FIG. 3 is a schematic diagram of a hardware structure of an access switch of the present application;

Fig. 4 is the hardware structural representation of a kind of Ethernet protocol conversion gateway of the present application;

FIG. 5 is an application scene diagram of an audio processing method based on the Internet of Vision according to an embodiment of the present application;

FIG. 6 is a flow chart of the steps of an audio processing method based on the Internet of Vision according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an audio processing device based on the Internet of Vision according to an embodiment of the present application.

detailed description

In order to make the above objects, features and advantages of the present application more obvious and comprehensible, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

The Internet of Vision is an important milestone in the development of the network. It is a real-time network that can realize real-time transmission of high-definition video, and push many Internet applications to high-definition video, high-definition face-to-face.

The Internet of View adopts real-time high-definition video exchange technology, which can provide required services on one network platform, such as high-definition video conferencing, video surveillance, intelligent monitoring and analysis, emergency command, digital broadcast TV, time-lapse TV, online teaching, live broadcast , VOD on demand, TV mail, personalized recording (PVR), intranet (self-managed) channel, intelligent video broadcast control, information release and other dozens of video, voice, picture, text, communication, data and other services are all integrated in one System platform, realize high-definition quality video playback through TV or computer.

In order to enable those skilled in the art to better understand the embodiment of the present application, the Internet of Things is introduced as follows:

Some of the technologies applied in the Internet of Things are as follows:

Network Technology (Network Technology)

The network technology innovation of the Internet of View has improved the traditional Ethernet (Ethernet) to face the potentially huge first video traffic on the network. Different from pure network packet switching (Packet Switching) or network circuit switching (Circuit Switching), video networking technology uses Packet Switching to meet Streaming requirements. The Internet of Vision technology has the flexibility, simplicity and low price of packet switching, and at the same time has the quality and security guarantee of circuit switching, realizing the seamless connection of switched virtual circuits and data formats throughout the network.

Switching Technology

Video networking adopts the two advantages of Ethernet asynchronous and packet switching, eliminates the defects of Ethernet under the premise of full compatibility, has end-to-end seamless connection of the whole network, directly connects to user terminals, and directly carries IP data packets. User data does not require any format conversion within the entire network. Video networking is a more advanced form of Ethernet. It is a real-time switching platform, which can realize the real-time transmission of large-scale high-definition video in the whole network that cannot be realized by the Internet at present, and push many network video applications to high-definition and unification.

Server Technology (Server Technology)

The server technology on the Internet of View and unified video platform is different from the server in the traditional sense. Its streaming media transmission is based on connection-oriented, and its data processing capability has nothing to do with traffic and communication time. A single network layer can contain information command and data transmission. For voice and video services, the complexity of video streaming and unified video platform streaming media processing is much simpler than data processing, and the efficiency is greatly improved by more than 100 times compared with traditional servers.

Storage Technology

The ultra-high-speed storage technology of the unified video platform adopts the most advanced real-time operating system in order to adapt to the super-large capacity and super-large-flow media content, and maps the program information in the server command to the specific hard disk space, and the media content no longer passes through the server. It is delivered directly to the user terminal in an instant, and the user generally waits for less than 0.2 seconds. The optimized sector distribution greatly reduces the mechanical movement of the hard disk head seeking. The resource consumption is only 20% of the IP Internet of the same level, but the concurrent traffic generated is 3 times larger than that of the traditional hard disk array, and the overall efficiency is increased by more than 10 times.

Network Security Technology

The structural design of the Internet of View completely eradicates the network security problems that plague the Internet through the individual licensing system for each service, complete isolation of equipment and user data, and generally does not require anti-virus programs and firewalls, preventing hackers and virus attacks , to provide users with a structured worry-free security network.

Service Innovation Technology

The unified video platform integrates business and transmission together, whether it is a single user, a private network user or the sum of a network, it is just an automatic connection. User terminals, set-top boxes or PCs are directly connected to the unified video platform to obtain rich and colorful multimedia video services in various forms. The unified video platform adopts the "recipe-style" table matching mode to replace the traditional complex application programming. It can realize complex applications with very little code and realize "unlimited" new business innovations.

The networking of the Internet of View is as follows:

Vision networking is a network structure with centralized control. The network can be a tree network, star network, ring network, etc., but on this basis, a centralized control node is required in the network to control the entire network.

As shown in Figure 1, the Internet of Things is divided into two parts: the access network and the metropolitan area network.

The equipment in the access network part can be mainly divided into three categories: node server, access switch, terminal (including various set-top boxes, encoding boards, storage, etc.). The node server is connected with the access switch, and the access switch can be connected with multiple terminals and can be connected with Ethernet.

Wherein, the node server is a node with centralized control function in the access network, which can control the access switches and terminals. The node server can be directly connected to the access switch, and can also be directly connected to the terminal.

Similarly, the devices in the MAN part can also be divided into three categories: MAN servers, node switches, and node servers. The metro server is connected to the node switch, and the node switch can be connected to multiple node servers.

Wherein, the node server is the node server of the access network part, that is, the node server belongs to both the access network part and the metropolitan area network part.

The metropolitan area server is a node with a centralized control function in the metropolitan area network, which can control node switches and node servers. The metro server can be directly connected to the node switch, or directly connected to the node server.

It can be seen that the entire Vision Network is a layered centralized control network structure, while the network controlled by the node server and the metro server can be in various structures such as tree, star, and ring.

Vividly speaking, the access network part can form a unified video platform (the part in the dotted circle), and multiple unified video platforms can form a video network; each unified video platform can be interconnected through the metropolitan area and the wide area video network.

Classification of Internet of Things devices

1.1 The devices in the Internet of View in the embodiment of the present application can be mainly divided into three categories: servers, switches (including Ethernet protocol conversion gateways), terminals (including various set-top boxes, encoding boards, memory, etc.). As a whole, the Internet of Things can be divided into a metropolitan area network (or a national network, a global network, etc.) and an access network.

1.2 The equipment in the access network can be mainly divided into three categories: node server, access switch (including Ethernet protocol conversion gateway), terminal (including various set-top boxes, encoding boards, storage, etc.).

The specific hardware structure of each access network device is:

Node server:

As shown in Figure 2, it mainly includes a network interface module 201, a switching engine module 202, a CPU module 203, and a disk array module 204;

Wherein, network interface module 201, the bag that CPU module 203, disk array module 204 come in all enters switching engine module 202; The guiding information of packet stores this packet into the queue of corresponding packet cache 206; If the queue of packet cache 206 is close to full, then discards; Switching engine module 202 polls all packet cache queues, if satisfying following conditions forwarding: 1) The sending buffer of the port is not full; 2) The queue packet counter is greater than zero. The disk array module 204 mainly realizes the control of the hard disk, including operations such as initialization and reading and writing of the hard disk; (including the configuration of the downlink protocol packet address table, the uplink protocol packet address table, and the data packet address table), and the configuration of the disk array module 204 .

Access switch:

As shown in Figure 3, mainly comprise network interface module (downlink network interface module 301, uplink network interface module 302), switching engine module 303 and CPU module 304;

Wherein, the packet (upstream data) that the downstream network interface module 301 comes in enters the packet detection module 305; Whether the destination address (DA), source address (SA), data packet type and packet length of the packet detection module 305 detection packet meet the requirements, if Meet, then distribute corresponding flow identifier (stream-id), and enter switching engine module 303, otherwise discard; The packet (downstream data) that upstream network interface module 302 comes in enters switching engine module 303; The data packet that CPU module 304 comes in Enter switching engine module 303; Switching engine module 303 carries out the operation of looking into address table 306 to the bag that comes in, thereby obtains the guiding information of packet; If the bag that enters switching engine module 303 is that downlink network interface goes to uplink network interface, then combines Flow identifier (stream-id) stores this packet into the queue of corresponding packet cache 307; If the queue of this packet cache 307 is close to full, then discards; If the packet that enters switching engine module 303 is not downlink network interface, goes up If the data packet is sent to the network interface, the data packet is stored in the queue of the corresponding packet buffer 307 according to the direction information of the packet; if the queue of the packet buffer 307 is nearly full, it is discarded.

The switching engine module 303 polls all packet buffer queues, which can include two situations:

If the queue goes from the downlink network interface to the uplink network interface, the following conditions are met for forwarding: 1) the port sending buffer is not full; 2) the queue packet counter is greater than zero; 3) the token generated by the code rate control module is obtained ;

If the queue does not go from the downlink network interface to the uplink network interface, the following conditions are met for forwarding: 1) the sending buffer of the port is not full; 2) the packet counter of the queue is greater than zero.

The code rate control module 308 is configured by the CPU module 304 to generate tokens for all packet buffer queues going from the downlink network interface to the uplink network interface within a programmable interval to control the uplink forwarding code rate.

The CPU module 304 is mainly responsible for protocol processing with the node server, configuration of the address table 306 , and configuration of the code rate control module 308 .

Ethernet protocol conversion gateway :

As shown in Figure 4, it mainly includes network interface modules (downlink network interface module 401, uplink network interface module 402), switching engine module 403, CPU module 404, packet detection module 405, code rate control module 408, address table 406, packet Buffer 407, MAC adding module 409, and MAC deleting module 410.

Wherein, the data packet coming in from the downlink network interface module 401 enters the packet detection module 405; the packet detection module 405 detects the Ethernet MAC DA, Ethernet MAC SA, Ethernet length or frame type, visual networking destination address DA, visual networking Source address SA, depending on whether the network data packet type and the packet length meet the requirements, if it is met, the corresponding flow identifier (stream-id) is allocated; then, the MAC DA, MAC SA, length or frametype ( 2byte), and enter the corresponding receive buffer, otherwise discard;

The downlink network interface module 401 detects the sending buffer of the port, if there is a packet, the Ethernet MAC DA of the corresponding terminal is known according to the visual network destination address DA of the packet, and the Ethernet MAC DA of the terminal, the MACSA of the Ethernet protocol conversion gateway, and the MACSA of the Ethernet protocol conversion gateway are added. Ethernet length or frame type, and send.

The functions of other modules in the Ethernet protocol conversion gateway are similar to those of the access switch.

terminal:

It mainly includes a network interface module, a business processing module and a CPU module; for example, a set-top box mainly includes a network interface module, an video and audio codec engine module, and a CPU module; an encoding board mainly includes a network interface module, an video and audio encoding engine module, and a CPU module; It mainly includes network interface module, CPU module and disk array module.

1.3 The equipment of the metropolitan area network can be mainly divided into two categories: node server, node switch, and metropolitan area server. Among them, the node switch mainly includes a network interface module, a switching engine module and a CPU module; the metro server mainly includes a network interface module, a switching engine module and a CPU module.

2. Definition of Internet of Things data package

2.1 Definition of access network data packet

The data packet of the access network mainly includes the following parts: destination address (DA), source address (SA), reserved bytes, payload (PDU), and CRC.

As shown in the table below, the data packets of the access network mainly include the following parts:

DA SA Reserved Payload CRC

in:

The destination address (DA) consists of 8 bytes (byte), the first byte indicates the type of data packet (such as various protocol packets, multicast data packets, unicast data packets, etc.), there are up to 256 possibilities, The second byte to the sixth byte is the address of the metropolitan area network, and the seventh and eighth bytes are the address of the access network;

The source address (SA) is also composed of 8 bytes (byte), and the definition is the same as that of the destination address (DA);

Reserved bytes consist of 2 bytes;

The payload part has different lengths according to different datagram types. If it is a variety of protocol packets, it is 64 bytes. If it is a single multicast data packet, it is 32+1024=1056 bytes. Of course, it is not limited to Above 2 types;

CRC consists of 4 bytes, and its calculation method follows the standard Ethernet CRC algorithm.

2.2 MAN packet definition

The topology of the metropolitan area network is a graph. There may be two or even more than two types of connections between two devices, that is, there may be more than 2 connections between node switches and node servers, node switches and node switches, and node switches and node servers. kind of connection. However, the MAN address of the MAN device is unique. In order to accurately describe the connection relationship between MAN devices, a parameter: label is introduced in the embodiment of this application to uniquely describe a MAN device.

The definition of labels in this manual is similar to the definition of labels in MPLS (Multi-Protocol Label Switch, Multi-Protocol Label Switching). Assuming that there are two connections between device A and device B, then the data packets from device A to device B have 2 labels, the packet from device B to device A also has 2 labels. The label is divided into an incoming label and an outgoing label. Assuming that the label (incoming label) of the data packet entering device A is 0x0000, the label (outgoing label) of the data packet when it leaves device A may become 0x0001. The network access process of the metropolitan area network is a network access process under centralized control, which means that the address allocation and label allocation of the metropolitan area network are all dominated by the metropolitan area server, and the node switches and node servers are all passively executed. Different from MPLS label allocation, MPLS label allocation is the result of mutual negotiation between switches and servers.

As shown in the table below, the data packet of the MAN mainly includes the following parts:

DA SA Reserved Label Payload CRC

That is, destination address (DA), source address (SA), reserved byte (Reserved), label, payload (PDU), and CRC. Among them, the format of the label can refer to the following definition: the label is 32bit, of which the high 16bit is reserved, and only the low 16bit is used, and its position is between the reserved byte of the data packet and the payload.

Based on the above-mentioned characteristics of the Internet of Vision, there are more and more video conferencing based on the Internet of Vision, and the application scenarios of the video conference are also becoming more and more diverse. Therefore, in order to ensure the conference quality of the video conference conducted in different application scenarios, it is necessary to ensure the stability of the audio call quality in the video conference.

For example, one application scenario is a video conferencing scenario in which drones are commanded. In this video conference, the Internet of Vision terminals in the Internet of Vision will perform audio and video communications with mobile terminals in the Internet, wherein the mobile terminals in the Internet An application service for this application scenario is installed on the platform, which can provide users with local services for drone control and audio and video communication with Internet-of-Vision terminals.

In such an application scenario, in order to ensure the audio call quality of a mobile terminal that is often in the wild, it is necessary to avoid echoes in the audio sent by the mobile terminal. The current method is to connect the application service to a third-party echo suppression tool, so that the generation of echo can be avoided by setting a time interval. However, when connecting to a third-party echo suppression tool, the third-party library must be packaged in C language as a so library (dynamic link library) that can be called by the mobile terminal, and the so library is called through the jni (JavaNative Interface) interface. The process is very cumbersome. It not only increases the development workload of developers, but also the effect of echo suppression is not good. After a period of time in the call, there will be a buzzing sound, which will affect the call quality.

Based on this, the applicant conceived one of the technical ideas of the present application to at least solve the above-mentioned technical concepts in order to improve the audio call quality on the mobile terminal side in the above-mentioned application scenarios, on the basis of comprehensively considering the characteristics of the Internet of Things and the above-mentioned application objects. The humming problem occurs after the call in question has been going on for a while. Specifically, when the application object detects that the audio and video call service is started, it triggers the preset audio collection mode on the mobile terminal, so that the mobile terminal collects the second audio data in this collection mode, and according to the first audio data pair The echo cancellation is performed on the second audio data, so that the echo generated by playing the first audio data can be canceled in the second audio data. It avoids the problem of calling the third-party echo suppression library and setting the time interval so that the human ear cannot distinguish the echo, which is cumbersome to connect to the third party, and the echo is superimposed over time to reduce the call quality.

Referring to FIG. 5 , it shows an application scenario of an audio processing method based on the Internet of Vision according to an embodiment of the present application. In this application scenario, an Internet-of-Vision terminal and an Internet terminal conduct an Internet-of-Vision video conference, wherein the Internet terminal remotely The handle establishes a communication connection with the UAV. Specifically, the remote control handle sends control instructions to the UAV to control the flight altitude and heading of the UAV. Among them, the pictures and flight data taken by the UAV will be sent back to the UAV. The Internet terminal, and furthermore, the Internet terminal can send pictures and flight data back to the Internet of Vision terminal through the Internet of Vision.

Wherein, a video networking terminal is deployed in the video networking, the video networking terminal is communicatively connected to a streaming media server, and the streaming media server is communicatively connected to a mobile terminal. Mobile terminals, unmanned aerial vehicles and remote controllers can all be deployed in the Internet, and the streaming media server can communicate with the Internet of Vision terminals in the Internet of Vision and communicate with mobile terminals in the Internet of Vision respectively.

Wherein, an audio playback component is configured on the mobile terminal, wherein the audio playback component may be but not limited to the following audio players: mp3, mp4, etc. An application object is also set in the mobile terminal, and the application object can correspond to the streaming media server, and the application object can provide users with local services such as data analysis, storage, and transmission of drones, as well as video-network audio and video conferences. , the streaming media server can provide background services for the above services performed by the application object, such as providing data forwarding services.

In the embodiment of the present application, the mobile terminal may be a terminal installed with an Android system, such as an Android mobile phone, an Android tablet computer, and the like.

Referring to FIG. 6 , it shows a flow chart of steps of an audio processing method based on the Internet of Things according to an embodiment of the present application. The method can be applied to the application object set in the mobile terminal, as shown in FIG. 6, specifically the following steps may be included:

Step S601, triggering a preset audio collection mode on the mobile terminal when it is detected that a preset audio and video call service is enabled.

In the embodiment of the present application, the preset audio and video call service may refer to that the audio and video call service is preconfigured in the application program object as one of multiple services provided by the application program object. For example, application objects can provide services such as drone data analysis, audio and video calls, and surveillance playback.

During specific implementation, the application program object can detect whether the audio and video call service is enabled according to the user's operation on the audio and video call service. operation, and then according to the start operation, to trigger the preset audio collection mode on the mobile terminal.

Wherein, the preset audio collection mode may refer to: an audio collection mode preset on the mobile terminal corresponding to the audio and video call service on the application program object. The audio collection mode can control the manner in which the mobile terminal collects audio data and preprocesses the audio data. In practice, when the audio and video call service is started, the application program object can trigger the audio collection mode, so that the mobile terminal can collect audio data in the audio collection mode. Triggering in this embodiment of the present application may refer to starting, that is, starting the preset audio collection mode.

In practice, multiple original audio collection modes can be configured in the mobile terminal. In this embodiment, the audio with the least environmental noise can be selected according to the audio data collected by the mobile terminal in each original audio collection mode. The audio collection mode corresponding to the data is determined to be the above preset audio collection mode.

Exemplarily, taking the mobile terminal as an example of an Android phone device, there are 9 original audio collection modes on the Android phone device, which are:

AudioSource (audio source).DEFAULT default mode;

AudioSource.MIC microphone mode;

AudioSource.VOICE_UPLINK telephone uplink mode;

AudioSource.VOICE_DOWNLINK phone downlink mode;

AudioSource.VOICE_CALL phone uplink + downlink mode;

AudioSource.CAMCORDER camera mode;

AudioSource.VOICE_RECOGNITION voice recognition mode;

AudioSource.VOICE_COMMUNICATION voice communication mode, such as VoIP (Voice over Internet Protocol, IP-based voice transmission) mode;

AudioSource.REMOTE_SUBMIX remote sound mode, such as wifi display (wireless display) mode.

Then, through actual testing, it is determined that the audio data collected by the mobile terminal in the voice communication mode contains the least noise, then the voice communication mode can be determined as the preset audio collection mode, so that the application set in the Android phone When the program object detects that the audio and video call service is enabled, the voice communication mode may be triggered.

Step S602, receiving the first audio data sent by the streaming media server in the audio and video call service, and calling the audio playing component to play the first audio data.

Wherein, the first audio data is sent by the Internet-of-TV terminal to the streaming media server.

In this embodiment, the first audio data is data collected by the Internet-of-Vision terminal, and is sent by the Internet-of-Vision terminal to the streaming media server through the Internet of Vision, and then sent to the application program object on the mobile terminal by the streaming media server.

During specific implementation, the mobile terminal can decode and play the first audio data by invoking the audio playback component, and can cache the first audio data at the same time, so that the mobile terminal can be updated later based on the cached first audio data. The collected audio data is echo-cancelled.

Step S603, acquiring second audio data collected by the mobile terminal in the audio collection mode.

In practice, after playing the first audio data, the application program object starts to acquire the second audio data collected by the mobile terminal in the audio collection mode. Because in the process of playing the first audio data, the sound emitted by the playback component is reflected by the surrounding environment to form an echo. Since the echo travels a longer distance than the directly transmitted sound, after the first audio data is played, The echo generated by the first audio data will be collected by the mobile terminal together with the user's speech. Therefore, the second audio data includes echo audio data in which the environment reflects the played first audio data.

In this embodiment, because the environmental noise included in the audio data collected in the preset audio collection mode can be the least, therefore, the environmental noise included in the second audio data collected in the preset audio collection mode is also less , thereby improving the audio quality of the second audio data. Specifically, the environmental noise refers to the noise generated in the surrounding environment where the mobile terminal is located, and the echo can be one of the environmental noises. When the environmental noise included in the second audio data collected in the preset audio collection mode When there is less noise, there can also be fewer echoes that characterize it. Therefore, the quality of the collected second audio data can be improved to obtain a better echo cancellation effect.

Step S604: Perform echo cancellation processing on the second audio data according to the first audio data and the second audio data, to obtain target audio data after echo cancellation processing.

In this embodiment, when the second audio data is obtained, the echo audio data included in the second audio data may be eliminated according to the buffered first audio data, so as to obtain the target audio data after the echo audio data is eliminated. Wherein, the echo audio data is the audio data collected by the mobile terminal and generated by the environment reflecting the first audio data being played.

In this way, the echo audio data is not included in the obtained target audio data, thereby realizing the echo cancellation processing on the mobile terminal side in the audio and video call service.

Step S605, sending the target audio data to the streaming media server, and the streaming media server is used to send the target audio data to the Internet-of-Video terminal.

After obtaining the target audio data, the application program object can send the target audio data to the streaming media server, so that the streaming media server can send the target audio data to the video networking terminal through the video network. Since echo audio data is not included in the target audio data, when the video network terminal plays the target audio data, what the user of the video network terminal hears is the clear voice of the user of the mobile terminal, and will not hear his last voice, thereby improving call quality.

In the embodiment of the present application, since the second audio data collected in the preset audio collection mode contains less environmental noise, less echoes are represented therein. Therefore, the present application can improve the quality of the collected second audio data. Furthermore, since the echo cancellation process is performed on the second audio data according to the first audio data, the obtained target audio data does not include echo audio data, thereby improving the audio quality of the transmitted target audio data. Compared with the way of setting the time interval, the target audio data sent by this application does not carry the echo audio data itself, so the call quality can be guaranteed and a clear call can be realized. It avoids the problem that the human ear can still hear the echo caused by the unreasonable time interval setting, and as time goes by, the echo accumulates more and more in the transmitted audio data to generate a buzzing sound.

In combination with the above-mentioned embodiments, in an optional example, the first microphone and the second microphone are configured on the mobile terminal, then in step S601, the preset audio collection mode on the mobile terminal is triggered, specifically Include the following steps:

Step S6011, call the first microphone and the second microphone.

In practice, the first microphone and the second microphone can be arranged at different positions of the mobile terminal. Optionally, the first microphone can be arranged at the bottom of the mobile terminal, and the second microphone can be arranged at the top of the mobile terminal. According to the description of the audio collection mode in the above step S601, it can be seen that multiple original audio collection modes can be set in the mobile terminal. In practice, the microphones used by each audio collection mode are different. The audio data collected by the microphone The way of preprocessing can also be different.

In this embodiment of the present application, the preset audio collection mode may correspond to calling the first microphone and the second microphone simultaneously. That is, when the preset audio collection mode is triggered, the application program object can call the first microphone and the second microphone at the same time, so as to use the first microphone and the second microphone to collect audio data.

Correspondingly, step S603 may specifically include the following steps:

Step S6031, acquiring the first microphone audio data collected by the first microphone and the second microphone audio data collected by the second microphone.

In practice, after calling the first microphone and the second microphone, the first microphone and the second microphone can simultaneously collect audio data.

Since the first microphone and the second microphone are located at different positions on the mobile terminal, the audio data collected by the two are different. Specifically, since the second microphone at the top and the first microphone at the bottom are at different distances from the user during a call, the volume of the user's voice included in the first microphone audio data and the user's voice included in the second microphone audio data The volume of the microphone is different, and the volume of the background noise picked up by the two microphones is basically the same, so the above-mentioned difference can be used to filter out the noise and preserve the human voice.

Step S6032: Perform noise reduction processing on the second microphone audio data according to the first microphone audio data and the second microphone audio data to obtain second audio data.

In the embodiment of the present application, in the preset audio collection mode, noise reduction processing may be performed on the second microphone audio data collected by the second microphone. Specifically, the first microphone audio data and the second microphone audio data can be decoded to generate a compensation signal, and then the second microphone audio data can be denoised according to the compensation signal, so that the ambient noise in the second microphone audio data can be removed. noise, so as to obtain the second audio data after noise reduction processing.

With reference to the above embodiments, in an optional example, step S604 may specifically include the following steps:

Step S6041, in the second audio data, determine third audio data corresponding to the first audio data.

In this optional example, when performing echo cancellation processing on the second audio data, since the echo audio data included in the second audio data is the echo reflected by playing the first audio data, the echo audio data is consistent with the first audio data. audio data related audio data. For example, the echo audio data and the first audio data both come from the same user's voice, and although they are reflected, their acoustic features are the same. Therefore, based on speech recognition technology, the third audio data whose matching degree with the acoustic features of the first audio data is greater than the preset matching degree can be found out from the second audio data, and the third audio data determined in this way is to play the first audio data. Echo audio data reflected by an audio data. Wherein, the acoustic feature may be a frequency feature or a spectral feature of the audio data.

Step S6042, filtering out the third audio data from the second audio data to obtain target audio data in which the third audio data is filtered out.

In practice, after the third audio data is determined, the third audio data may be removed from the second audio data to obtain target audio data.

Correspondingly, in an optional example, while triggering the preset audio collection mode on the mobile terminal, the method may specifically include the following steps:

Step S6012, calling the adaptive filter set in the mobile terminal.

In this optional example, in order to improve the efficiency of echo cancellation processing on the second audio data and reduce the underlying development amount of the application program object, the application program object can call the adaptive filter configured in the mobile terminal, so that the adaptive filter The device performs echo cancellation processing on the second audio data.

Optionally, step S6012 may specifically include the following steps:

Step S6012-1: Determine at least one application program interface on the Internet of Vision terminal that is compatible with the application program object, and determine whether there is a target interface in the at least one application program interface.

In practice, different adaptive filters can be obtained based on different underlying development software, and the echo cancellation efficiency and quality of different adaptive filters can also be different, and different adaptive filters correspond to different APIs.

In the embodiment of the present application, the application program interface corresponding to the above-mentioned different adaptive filters may establish a call relationship with the application program object, and determine the application program interface adapted to the application program object on the Internet of Vision terminal, that is It is at least one application program interface that can be determined to have a call relationship with the application program object on the Internet of Vision terminal. In this way, when the application program object performs echo cancellation processing on the second audio data, it can successfully call the adaptive filter configured by the mobile terminal by calling the application program interface that establishes a calling relationship with itself. In the case of the echo suppression tool, the calling process is simplified and the efficiency is improved.

Wherein, the target interface may be preset. Specifically, each application program interface has its own identifier. In practice, determining whether the target interface exists in the at least one application program interface may be in the at least one application program interface. , determine whether there is an application program interface whose identifier of the application program interface is consistent with the identifier of the target interface.

In an optional example, since different adaptive filters have different efficiency and quality of echo cancellation processing, in practice, the at least one application program interface may have its own priority, and the higher the priority, the application The better the efficiency and quality of echo cancellation processing performed by the adaptive filter corresponding to the interface. Then in this embodiment of the application, the target interface may refer to an application program interface with a priority higher than the preset priority, that is, it may be determined in at least one application program interface whether there is a target interface with a priority higher than the preset priority

During specific implementation, taking the mobile terminal as an Android phone as an example, the target interface may be an AEC (Acoustic Echo Canceller, acoustic echo canceller) interface. Because AEC can develop the echo cancellation program very quickly, therefore, the adaptive filter developed based on this AEC can quickly carry out echo cancellation to audio data, and the quality of echo cancellation is better, thereby can improve the audio communication of the present application quality.

Wherein, when it is determined that the target interface exists, go to step S6012-2, and when it is determined that the target interface does not exist, go to step S6012-3.

Step S6012-2, call the adaptive filter corresponding to the target interface through the target interface.

In practice, when the target interface exists in the at least one application program interface, the adaptive filter corresponding to the interface may be invoked through the target interface.

For example, if there is an AEC interface, the adaptive filter corresponding to the AEC interface is invoked through the AEC interface.

Step S6012-3, call the adaptive filter corresponding to the preset application program interface through the preset application program interface.

In this embodiment, the preset application program interface may refer to a spare application program interface among the at least one application program interface. In the process, one of the application program interfaces is designated as a standby application program interface, and when it is determined that the target interface does not exist in the at least one application program, then the application program object can call the standby application program interface.

For example, take the mobile terminal as an example of an Android phone and the target interface as an AEC interface. In practice, the AEC interface may not be applicable to the model of the mobile terminal. In this case, you can use the speex (echo cancellation algorithm) interface As a preset application program interface, the adaptive filter can be called through the speex interface. Since speex is applicable to various models of mobile terminals and has a wide range of adaptability, the speex interface can be used as a backup application program interface.

Correspondingly, in an optional example, since each application program interface may have its own priority, the priority of the preset application program interface is the same as the level of the next priority adjacent to the preset priority .

After the adaptive filter is successfully invoked, the adaptive filter can be used to perform echo cancellation processing on the second audio data. Specifically, the adaptive filter can be used to perform the above steps S6041 and S6042 respectively, wherein step S6041 can specifically Include the following steps:

S60411. Input the first audio data into the adaptive filter to obtain output audio data output by the adaptive filter.

Since the first audio data is cached when the first audio data is received, the first audio data can be extracted from the cache, the first audio data can be input to the adaptive filter, and the adaptive filter After processing, output audio data is obtained.

Specifically, taking the adaptive filter corresponding to the AEC interface as an example, assuming that the first audio data is x(n), and inputting x(n) into the adaptive filter, the adaptive filter performs an input signal sequence x(n) For each sample value of , update and adjust the weighting coefficient according to a specific algorithm, so that the mean square error of the output signal sequence y(n) compared with the expected output signal sequence d(n) is the smallest, that is, the output signal sequence y(n ) approximates the expected signal sequence d(n), and the closer y(n) is to d(n), the more consistent y(n) and x(n) are. Among them, the output signal sequence y(n) is the output audio data, and the coefficients of the adaptive filter designed based on the minimum mean square error can be obtained by solving the Wiener-Hopf equation.

S60412. In the second audio data, determine third audio data having the same frequency as the output audio data.

Wherein, the application program object may use an adaptive filter to determine third audio data having the same frequency as the output audio data in the second audio data. Since the output audio data is consistent with x(n), when the frequency of the third audio data is the same as that of the output audio data, it can be indicated that the third audio data is audio data related to the first audio data.

Correspondingly, step S6042 may specifically be the following steps:

Step S6043: Filter out the third audio data from the second audio data by using the adaptive filter to obtain target audio data in which the third audio data is filtered out.

In practice, when the third audio data is determined, an adaptive filter may also be used to filter the third audio data from the second audio data.

It should be noted that, for the method embodiment, for the sake of simple description, it is expressed as a series of action combinations, but those skilled in the art should know that the embodiment of the present application is not limited by the described action sequence, because According to the embodiment of the present application, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification belong to preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

Referring to FIG. 7 , it shows a structural block diagram of an audio processing device based on the Internet of Vision according to an embodiment of the present application. In the Internet of Vision, an Internet-of-Vision terminal is deployed, and the Internet-of-Vision terminal communicates with a streaming media server. The streaming media server is communicatively connected to the mobile terminal, the mobile terminal is equipped with an audio playback component, and the device is applied to an application program object set in the mobile terminal, the device may specifically be a virtual device, and specifically may include The following modules:

An audio mode trigger module 701, configured to trigger a preset audio data collection mode on the mobile terminal when it is detected that a preset audio call service is turned on;

Audio data receiving and playing module 702, configured to receive the first audio data sent by the streaming media server, call the audio playing component, and play the first audio data; the first audio data is played by the video The networked terminal sends to the streaming media server;

An audio data collection module 703, configured to obtain second audio data collected by the mobile terminal in the audio data collection mode;

An audio data processing module 704, configured to perform echo cancellation processing on the second audio data according to the first audio data and the second audio data, to obtain target audio data after echo cancellation processing;

An audio data sending module 705, configured to send the target audio data to the streaming media server, and the streaming media server is configured to send the target audio data to the Internet-of-Video terminal.

Optionally, a first microphone and a second microphone are configured on the mobile terminal, and the audio mode trigger module may specifically be used to invoke the first microphone and the second microphone;

The audio data acquisition module may specifically include the following units:

a microphone audio data acquisition unit, configured to acquire first microphone audio data collected by the first microphone, and second microphone audio data collected by the second microphone;

The noise reduction processing unit is configured to perform noise reduction processing on the second microphone audio data according to the first microphone audio data and the second microphone audio data to obtain second audio data.

Optionally, the audio data processing module may specifically include the following units:

an audio data search unit, configured to determine, among the second audio data, third audio data corresponding to the first audio data;

An audio data filtering unit, configured to filter out the third audio data from the second audio data to obtain target audio data from which the third audio data is filtered out.

Optionally, the device may specifically include the following modules:

A calling module, used to call the adaptive filter set in the mobile terminal;

The audio data search unit may specifically include the following units:

an audio data input unit, configured to input the first audio data into the adaptive filter to obtain output audio data output by the adaptive filter;

The audio data determining unit is configured to determine, among the second audio data, third audio data having the same frequency as the output audio data.

Optionally, the calling module may specifically include the following units:

A target interface determining unit, configured to determine at least one application program interface adapted to the application program object on the Internet-of-Vision terminal, and determine whether there is a target interface in the at least one application program interface;

A first calling unit, configured to call the adaptive filter through the target interface when the target interface exists in the at least one application program interface;

The second calling unit is configured to call the adaptive filter through a preset application program interface when the target interface does not exist in the at least one application program interface.

For the embodiment of the audio processing device based on the Internet of Vision, since it is basically similar to the embodiment of the audio processing method based on the Internet of Vision, the description is relatively simple. That's it.

The embodiment of the present application also provides an electronic device, including:

One or more processors; and one or more machine-readable media with instructions stored thereon, when executed by the one or more processors, make the device execute one of the methods described in the embodiments of the present application or multiple video-network-based audio processing methods.

The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program to enable a processor to execute the audio processing method based on the Internet of Vision as described in the embodiment of the present application.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other.

Those skilled in the art should understand that the embodiments of the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiment of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Embodiments of the present application are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing terminal equipment to produce a machine such that instructions executed by the computer or processor of other programmable data processing terminal equipment Produce means for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the The instruction means implements the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded into a computer or other programmable data processing terminal equipment, so that a series of operational steps are performed on the computer or other programmable terminal equipment to produce computer-implemented processing, thereby The instructions executed above provide steps for implementing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

While the preferred embodiments of the embodiments of the present application have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once the basic inventive concept is understood. Therefore, the appended claims are intended to be interpreted to cover the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the application.

Finally, it should also be noted that in this text, relational terms such as first and second etc. are only used to distinguish one entity or operation from another, and do not necessarily require or imply that these entities or operations, any such actual relationship or order exists. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or terminal equipment comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements identified, or also include elements inherent in such a process, method, article, or end-equipment. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising said element.

The above is a detailed introduction to an audio processing method, device, electronic equipment and storage medium based on the Internet of Vision provided by this application. In this paper, specific examples are used to illustrate the principle and implementation of this application. The above embodiments The description is only used to help understand the method of the present application and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope, in summary , the contents of this specification should not be construed as limiting the application.

Claims (10)

1. An audio processing method based on a video network is characterized in that a video network terminal is deployed in the video network, the video network terminal is in communication connection with a streaming media server, the streaming media server is in communication connection with a mobile terminal, and an audio playing component is configured on the mobile terminal, and the method is applied to an application program object arranged in the mobile terminal, and comprises the following steps:

when detecting that a preset audio and video call service is started, triggering a preset audio acquisition mode on the mobile terminal, wherein the preset audio acquisition mode is an audio acquisition mode corresponding to audio data with the least environmental noise;

receiving first audio data sent by the streaming media server in the audio and video call service, and calling the audio playing component to play the first audio data; the first audio data is sent to the streaming media server by the video networking terminal;

acquiring second audio data acquired by the mobile terminal in the audio acquisition mode;

according to the first audio data and the second audio data, performing echo cancellation processing on the second audio data to obtain target audio data after echo cancellation processing;

and sending the target audio data to the streaming media server, wherein the streaming media server is used for sending the target audio data to the video networking terminal.

2. The method according to claim 1, wherein a first microphone and a second microphone are configured on the mobile terminal, and triggering a preset audio capture mode on the mobile terminal comprises:

invoking the first microphone and the second microphone;

acquiring second audio data acquired by the mobile terminal in the audio acquisition mode, including:

obtaining first microphone audio data captured by the first microphone and second microphone audio data captured by the second microphone;

and according to the first microphone audio data and the second microphone audio data, carrying out noise reduction processing on the second microphone audio data to obtain second audio data.

3. The method of claim 1, wherein performing echo cancellation processing on the second audio data according to the first audio data and the second audio data to obtain target audio data after echo cancellation processing, comprises:

determining third audio data corresponding to the first audio data in the second audio data;

and filtering the third audio data from the second audio data to obtain target audio data with the third audio data filtered.

4. The method according to claim 3, wherein while triggering a preset audio capture mode on the mobile terminal, the method further comprises:

calling a self-adaptive filter arranged in the mobile terminal;

determining, among the second audio data, third audio data having the same frequency as the first audio data, including:

inputting the first audio data into the adaptive filter to obtain output audio data output by the adaptive filter;

in the second audio data, third audio data having the same frequency as the output audio data is determined.

5. The method of claim 4, wherein invoking the adaptive filter configured on the mobile terminal comprises:

determining at least one application program interface which is matched with the application program object on the video network terminal, and determining whether a target interface exists in the at least one application program interface;

when the target interface exists in the at least one application program interface, calling an adaptive filter corresponding to the target interface through the target interface;

and when the target interface does not exist in the at least one application program interface, calling the adaptive filter corresponding to the preset application program interface through a preset application program interface.

6. An audio processing device based on video networking, wherein a video networking terminal is deployed in the video networking, the video networking terminal is in communication connection with a streaming media server, the streaming media server is in communication connection with a mobile terminal, an audio playing component is configured on the mobile terminal, and the device is applied to an application program object arranged in the mobile terminal, and comprises:

the mobile terminal comprises an audio mode triggering module, a voice acquisition module and a voice processing module, wherein the audio mode triggering module is used for triggering a preset audio acquisition mode on the mobile terminal when detecting that a preset audio communication service is started, and the preset audio acquisition mode is an audio acquisition mode corresponding to audio data with the least environmental noise;

the audio data receiving and playing module is used for receiving first audio data sent by the streaming media server, calling the audio playing component and playing the first audio data; the first audio data is sent to the streaming media server by the video networking terminal;

the audio data acquisition module is used for acquiring second audio data acquired by the mobile terminal in the audio acquisition mode;

the audio data processing module is used for performing echo cancellation processing on the second audio data according to the first audio data and the second audio data to obtain target audio data after echo cancellation processing;

and the audio data sending module is used for sending the target audio data to the streaming media server, and the streaming media server is used for sending the target audio data to the video network terminal.

7. The apparatus according to claim 6, wherein a first microphone and a second microphone are configured on the mobile terminal, and the audio mode triggering module is specifically configured to invoke the first microphone and the second microphone;

the audio data acquisition module comprises:

a microphone audio data acquisition unit configured to acquire first microphone audio data acquired by the first microphone and second microphone audio data acquired by the second microphone;

and the noise reduction processing unit is used for carrying out noise reduction processing on the second microphone audio data according to the first microphone audio data and the second microphone audio data to obtain second audio data.

8. The apparatus of claim 6, wherein the audio data processing module comprises:

the audio data searching unit is used for determining third audio data corresponding to the first audio data in the second audio data;

and the audio data filtering unit is used for filtering the third audio data from the second audio data to obtain target audio data with the third audio data filtered.

9. An electronic device, comprising:

one or more processors; and

one or more machine readable media having instructions stored thereon that, when executed by the one or more processors, cause the electronic device to perform the video networking-based audio processing method of any of claims 1-5.

10. A computer-readable storage medium storing a computer program for causing a processor to execute the video network-based audio processing method according to any one of claims 1 to 5.

Images