CN110062268A - A kind of audio-video sends and receives processing method and processing device with what screen played - Google Patents
A kind of audio-video sends and receives processing method and processing device with what screen played Download PDFInfo
- Publication number
- CN110062268A CN110062268A CN201910316786.0A CN201910316786A CN110062268A CN 110062268 A CN110062268 A CN 110062268A CN 201910316786 A CN201910316786 A CN 201910316786A CN 110062268 A CN110062268 A CN 110062268A
- Authority
- CN
- China
- Prior art keywords
- audio
- video
- stream
- output
- sending
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 29
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 230000005540 biological transmission Effects 0.000 claims abstract description 9
- 238000000034 method Methods 0.000 claims description 34
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012163 sequencing technique Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008571 general function Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000017525 heat dissipation Effects 0.000 description 1
- 238000010438 heat treatment Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The embodiment of the invention discloses a kind of audio-videos to send and receive processing method and processing device with what screen played, and audio-video includes: that transmitting terminal captures video flowing and audio stream in the transmitting terminal with the transmission processing method that screen plays；The video flowing and audio stream of capture are encoded, generate the outputting video streams of the first preset format and the output audio stream of the second preset format respectively；It will be played out with the outputting video streams of stamp at the first time and output audio streams to the receiving end with the second timestamp, and play video flowing and audio stream in the transmitting terminal, to realize that the same screen of the transmitting terminal and the receiving end plays.It is encoded by video flowing and audio stream of the transmitting terminal to capture, and the outputting video streams with timestamp are played out with output audio streams to receiving end, so that transmitting terminal and receiving end be not under the premise of by other hardware, it realizes and is played with screen, and realize that there is universality based on common terminal.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a device for transmitting and receiving audio and video on-screen playing.
Background
With the development of mobile internet technology and the emergence of various mobile devices, the requirement of wireless one-screen is more and more extensive. Currently, the mainstream wireless one-screen technology mainly includes three types: AirPlay, DLNA and Miracast: the AirPlay is a wireless technology developed by apple Inc., pictures, audios, videos and the like on IOS equipment such as iPhone, iPad, iPod and the like can be wirelessly transmitted to equipment supporting the AirPlay for playing and viewing through WiFi; DLNA (Digital living network Alliance) is a protocol for interconnection and intercommunication among PCs, mobile devices and electric appliances, initiated by sony, intel, microsoft, etc., and its purpose is to enjoy music, photos and videos in a time-bearing and place-bearing manner; miracast is a wireless display standard based on Wi-Fi direct, which was introduced by the Wi-Fi alliance in 2012, and devices supporting the standard can share video pictures in a wireless manner, for example, a mobile phone can play a movie or a photo directly on a television or other devices through Miracast without being affected by the length of a connection cable.
However, each of the three mainstream technologies has its limitations: the AirPlay has the advantages that the AirPlay supports the same Screen (Screen smoothing), namely, the whole Screen picture of the sending end is delivered to the receiving end in real time, and meanwhile, the AirPlay also supports sound transmission, is convenient to use, but has the defect of no universality, is only suitable for running iOS and macOS apple equipment, and also does not support reverse control, namely, control signals cannot be transmitted from the receiving end to the sending end; DLNA has the defects that media files such as photos, audio, video and the like can be shared between devices, and the devices do not support the same screen and reverse control; miracast is the most applicable protocol among the three, the function of a sending end is integrated in an android system, and the method is simple and easy to use, but the method has the defects that a receiving end only supports a win10 operating system, does not support a win7 operating system which is still used in a large range at present, does not support reverse control, only supports a small amount of equipment which realizes a UIBC (user Input Back channel) protocol, needs hardware support for the receiving end, is usually a simple Miracast receiver or a same-screen box which integrates the Miracast receiver, generally has the heat dissipation problem of the hardware, has serious screen splash phenomenon due to heating and the like after long-time use, has the same-screen and control functions at the level of hardware and system, belongs to the general function, and cannot customize the application level aiming at a specific application program.
Disclosure of Invention
Because the existing method has the problems, the embodiment of the invention provides a method and a device for processing the sending and receiving of audio and video on-screen playing.
In a first aspect, an embodiment of the present invention provides a method for sending and processing audio and video on-screen play, including:
a sending end captures video stream and audio stream in the sending end;
encoding the captured video stream and audio stream to respectively generate an output video stream in a first preset format and an output audio stream in a second preset format;
and sending the output video stream with the first time stamp and the output audio stream with the second time stamp to a receiving end for playing, and playing the video stream and the audio stream in the sending end so as to realize the same-screen playing of the sending end and the receiving end.
In a second aspect, an embodiment of the present invention provides a method for receiving and processing audio and video played on the same screen, including:
a receiving end receives an output video stream with a first time stamp and an output audio stream with a second time stamp which are sent by a sending end;
assembling the output video stream and the output audio stream according to the first time stamp and the second time stamp to obtain an assembled audio and video;
and decoding and playing the assembled audio and video to realize the on-screen playing of the sending end and the receiving end.
In a third aspect, an embodiment of the present invention further provides a device for sending and processing audio and video on-screen play, where the device includes:
the audio/video capture module is used for capturing the video stream and the audio stream in the sending end;
the audio and video coding module is used for coding the captured video stream and audio stream and respectively generating an output video stream in a first preset format and an output audio stream in a second preset format;
and the audio and video transmitting module is used for transmitting the output video stream with the first time stamp and the output audio stream with the second time stamp to a receiving end for playing, and playing the video stream and the audio stream in the transmitting end so as to realize the same-screen playing of the transmitting end and the receiving end.
In a fourth aspect, an embodiment of the present invention further provides a receiving and processing device for audio and video on-screen playing, including:
the audio and video receiving module is used for receiving an output video stream with a first time stamp and an output audio stream with a second time stamp which are sent by the sending end;
the audio and video assembly module is used for assembling the output video stream and the output audio stream according to the first time stamp and the second time stamp to obtain an assembled audio and video;
and the audio and video decoding module is used for decoding and playing the assembled audio and video so as to realize the same-screen playing of the transmitting end and the receiving end.
In a fifth aspect, an embodiment of the present invention further provides an electronic device, including:
at least one processor; and
at least one memory communicatively coupled to the processor, wherein:
the memory stores program instructions which can be executed by the processor, and the processor calls the program instructions to execute the sending processing method of audio and video on-screen playing and the receiving processing method of audio and video on-screen playing.
In a sixth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores a computer program, and the computer program enables the computer to execute the above-mentioned sending processing method for audio and video on-screen playing and receiving processing method for audio and video on-screen playing.
According to the technical scheme, the captured video stream and audio stream are coded by the sending end, the output video stream and the output audio stream with the time stamps are sent to the receiving end to be played, the sending end and the receiving end achieve the same-screen playing on the premise of not using other hardware, and the method and the device are achieved based on a common terminal and have universality.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a sending processing method for audio and video on-screen playing according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a receiving processing method for audio and video on-screen playing according to an embodiment of the present invention;
fig. 3 is an interaction diagram of a transmitting and receiving processing system for audio and video on-screen playing according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a sending processing device for audio and video on-screen playing according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a receiving and processing device for audio and video on-screen playing according to an embodiment of the present invention;
fig. 6 is a logic block diagram of a first electronic device according to an embodiment of the present invention;
fig. 7 is a logic block diagram of a second electronic device according to an embodiment of the present invention.
Detailed Description
The following further describes embodiments of the present invention with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.
Fig. 1 shows a schematic flow chart of a sending processing method for audio and video on-screen playing provided in this embodiment, including:
s101, a sending end captures video stream and audio stream in the sending end.
The sending end is a terminal used for sending video streams and audio streams, and can be a mobile phone, a PAD or a computer.
The video stream is a video resource being played in a sending end.
The audio stream is an audio resource being played in a sending end.
S102, the captured video stream and the captured audio stream are encoded, and an output video stream in a first preset format and an output audio stream in a second preset format are generated respectively.
The first preset format is a preset output format of the video stream.
The second preset format is an output format of a preset audio stream.
The output video stream is the video stream which is output after being coded by the coder.
The output audio stream is an audio stream which is output after being encoded by an encoder.
S103, sending the output video stream with the first time stamp and the output audio stream with the second time stamp to a receiving end for playing, and playing the video stream and the audio stream in the sending end to realize the same-screen playing of the sending end and the receiving end.
And the first time stamp is a time stamp carried by each output video stream.
And the second time stamp is the time stamp carried by each output audio stream.
The receiving end is a terminal for receiving video streams and audio streams, and can be a mobile phone, a PAD or a computer.
Specifically, a sending end sends an output video stream and an output audio stream to a receiving end, so that the receiving end decodes the output video stream and the output audio stream and plays the decoded output video stream and the decoded output audio stream; meanwhile, the sending end plays the video stream and the audio stream in the sending end, so that the sending end and the receiving end play the video and the audio simultaneously, and the effect of playing on the same screen is achieved.
In this embodiment, the captured video stream and audio stream are encoded by the sending end, and the output video stream and the output audio stream with the timestamps are sent to the receiving end for playing, so that the sending end and the receiving end realize the same-screen playing without using other hardware, and the sending end and the receiving end are realized based on a common terminal, and have universality.
Further, on the basis of the above method embodiment, S101 specifically includes:
a transmitting end captures a video stream in the transmitting end through a mediaproject interface, and captures an audio stream in the transmitting end through an AudioRecord interface.
Correspondingly, S102 specifically includes:
the captured video stream and audio stream are encoded through a MediaCodec interface to generate an output video stream of a first preset format and an output audio stream of a second preset format, respectively.
The mediaproject interface, the AudioRecord interface and the MediaCodec interface are Android self-contained program interfaces, a video stream is captured through the mediaproject interface, an audio stream is captured through the AudioRecord interface, and the captured video stream and the captured audio stream are respectively encoded through the MediaCodec interface, so that the processing efficiency can be greatly improved.
Further, on the basis of the above method embodiment, S103 specifically includes:
and sending the output video stream with the first time stamp and the output audio stream with the second time stamp to a receiving end through a User Datagram Protocol (UDP) channel for playing.
Among them, UDP is a connectionless protocol used for processing packets in a network like TCP, and is located at a layer above IP protocol in the fourth layer, the transport layer, in the OSI model. UDP has the disadvantage of not providing packet packetization, assembly and the inability to sequence packets, i.e. after a message is sent, it is not known whether it arrives safely and completely. UDP is used to support network applications that require data to be transferred between computers, and many client/server mode network applications, including network video conferencing systems, require the use of the UDP protocol. The main role of the UDP protocol is to compress network data traffic into packets.
Further, on the basis of the above method embodiment, the method for sending and processing audio and video on-screen playing further includes:
and S104, if the trigger operation of the sending end on the video stream and/or the audio stream is detected, sending a Control signal corresponding to the trigger operation to a receiving end through a Transmission Control Protocol (TCP) channel for corresponding Control.
Wherein the triggering operation is an operation of starting, pausing or ending video or audio.
TCP is a connection-oriented, reliable, byte-stream-based transport-layer communication protocol, defined by IETF's RFC 793. In the simplified OSI model of computer networks, which performs the functions specified by the fourth transport layer, the TCP layer is an intermediate layer located above the IP layer and below the application layer in the internet protocol suite. The application layer sends the data stream for internetwork transmission in 8-bit bytes to the TCP layer, which then partitions the data stream into segments of appropriate length, and then the TCP passes the resulting packets to the IP layer, which then transmits the packets over the network to the TCP layer of the receiving end entity. TCP gives each packet a sequence number in order to ensure that no packet is lost, and the sequence number also ensures in-sequence reception of packets transmitted to the receiving end entity. Then the receiving end entity sends back a corresponding confirmation to the successfully received packet; if the sending entity does not receive an acknowledgement within a reasonable round trip delay, the corresponding data packet is assumed to have been lost and will be retransmitted. TCP uses a checksum function to check whether the data has errors; the checksum needs to be calculated during sending and receiving, and the correctness of control signal transmission is ensured.
Fig. 2 is a schematic flow chart of a receiving processing method for audio and video on-screen playing provided in this embodiment, and includes:
s201, a receiving end receives an output video stream with a first time stamp and an output audio stream with a second time stamp which are sent by a sending end.
S202, assembling the output video stream and the output audio stream according to the first time stamp and the second time stamp to obtain the assembled audio and video.
S203, decoding and playing the assembled audio and video to realize the same-screen playing of the sending end and the receiving end.
The receiving end is a terminal for receiving video streams and audio streams, and may be a mobile phone, a PAD, or a computer.
The sending end is a terminal used for sending video streams and audio streams, and can be a mobile phone, a PAD or a computer.
The first time stamp is a time stamp carried by each output video stream when the sending end sends the output video stream to the receiving end.
And the second time stamp is the time stamp carried by each output audio stream when the sending end sends the audio stream to the receiving end.
Specifically, a sending end sends an output video stream and an output audio stream to a receiving end, so that the receiving end decodes the output video stream and the output audio stream and plays the decoded output video stream and the decoded output audio stream; meanwhile, the sending end plays the video stream and the audio stream in the sending end, so that the sending end and the receiving end play the video and the audio simultaneously, and the effect of playing on the same screen is achieved.
In this embodiment, the captured video stream and audio stream are encoded by the sending end, and the output video stream and the output audio stream with the timestamps are sent to the receiving end for playing, so that the sending end and the receiving end realize the same-screen playing without using other hardware, and the sending end and the receiving end are realized based on a common terminal, and have universality.
Further, on the basis of the above method embodiment, S202 specifically includes:
and sequencing the output video streams according to the first time stamp, sequencing the output audio streams according to the second time stamp, and synchronizing the sequenced output video streams and output audio streams according to the first time stamp and the second time stamp to obtain the assembled audio and video.
Specifically, the output video stream sent by the sending end to the receiving end includes a plurality of streams, and each output video stream carries a timestamp for recording the sending time of the current output video stream; similarly, the output audio stream sent by the sending end to the receiving end comprises a plurality of streams, and each output audio stream carries a time stamp for recording the sending time of the current output audio stream. Sequencing the output video streams according to the first time stamps carried by the output video streams to obtain the output video streams in the correct sequence; similarly, sequencing the output audio streams according to the second time stamps carried by the output audio streams to obtain the output audio streams in the correct sequence; and finally, when the video stream and the audio stream are integrally output, the video and the audio are synchronized according to the time stamps carried by the video stream and the audio stream.
In the practical application process, the embodiment provides a pure software-based on-screen scheme which is highly flexible, can support multiple platform receiving ends such as windows/mac/linux, and can be deeply integrated with an application scene, and the system structure of the scheme is shown in fig. 3 and comprises two parts: an Android sending module (sending end) and a windows/mac/linux receiving end (desktop receiving end).
The Android sending end is used for obtaining video and audio streams on Android equipment in real time and sending the video and audio streams to the receiving end through a UDP channel. Wherein the acquisition and processing of the video stream is mediaproject through the android system. The method comprises the following specific steps: creating a virtual display VirtualDisplay through MediaProjection, and creating a Surface through MediaCodec; the VirtualDisplay renders the screen video stream to the Surface in real time, and obtains the encoded H.264 video stream through MediaCodec. The audio stream is obtained and processed in a similar way as the video stream: system audio is acquired through AudioRecord, an audio stream is input into MediaCodec, and an encoded audio stream is acquired through MediaCodec. Meanwhile, the sending end is also connected with the receiving end through a TCP channel to transmit control signals, such as start, pause, end and the like.
The desktop receiving end is developed based on cross-platform framework Electron, can run in mainstream desktop operating systems such as win7/win10/osx/linux, and reassembles received video and audio packets, and decodes and plays the video and audio packets.
For example, in a 100M classroom with wireless bandwidth, an android flat plate of C5 model is used, and a picture on the flat plate is projected to a win7 computer desktop in the classroom in real time, so that the effect is good, the same-screen quality achieves the Miracast hard screen projection effect, and the screen can be run for a long time without a flower screen and dropped; in addition, the 720P film is played on the flat plate, and the played pictures are projected to a receiving end of a desktop in real time, so that the same-screen effect is ideal, and basically no blockage exists.
The method and the device have the advantages that the pure software co-screen is realized, no extra hardware is needed, the method and the device are realized based on a common terminal, the applicability is wide, the customization is flexible, and the device and the method can be integrated in a software product as a module, so that seamless co-screen experience is provided.
Fig. 4 is a schematic structural diagram illustrating a sending processing device for audio and video on-screen playing provided in this embodiment, where the device includes: an audio and video capture module 401, an audio and video encoding module 402, and an audio and video sending module 403, wherein:
the audio/video capturing module 401 is configured to capture a video stream and an audio stream at the sending end;
the audio/video encoding module 402 is configured to encode the captured video stream and audio stream, and generate an output video stream in a first preset format and an output audio stream in a second preset format, respectively;
the audio/video transmitting module 403 is configured to transmit an output video stream with a first timestamp and an output audio stream with a second timestamp to a receiving end for playing, and play the video stream and the audio stream at the transmitting end, so as to implement on-screen playing of the transmitting end and the receiving end.
Specifically, the audio/video capture module 401 captures a video stream and an audio stream in the sending end; the audio/video encoding module 402 encodes the captured video stream and audio stream to generate an output video stream in a first preset format and an output audio stream in a second preset format, respectively; the audio/video transmitting module 403 transmits the output video stream with the first timestamp and the output audio stream with the second timestamp to the receiving end for playing, and plays the video stream and the audio stream in the transmitting end, so as to realize the on-screen playing of the transmitting end and the receiving end.
In this embodiment, the captured video stream and audio stream are encoded by the sending end, and the output video stream and the output audio stream with the timestamps are sent to the receiving end for playing, so that the sending end and the receiving end realize the same-screen playing without using other hardware, and the sending end and the receiving end are realized based on a common terminal, and have universality.
Further, on the basis of the above apparatus embodiment, the audio/video capturing module 401 is specifically configured to capture a video stream in the transmitting end through a media projection interface, and capture an audio stream in the transmitting end through an audio recording interface;
correspondingly, the audio/video encoding module 402 is specifically configured to encode the captured video stream and audio stream through a MediaCodec interface, and generate an output video stream in a first preset format and an output audio stream in a second preset format, respectively.
Further, on the basis of the above apparatus embodiment, the audio/video sending module 403 is specifically configured to send the output video stream with the first timestamp and the output audio stream with the second timestamp to the receiving end through a user datagram protocol UDP channel for playing.
Further, on the basis of the above device embodiment, the sending and processing device for audio and video on-screen playing further includes:
and the operation control module is used for sending a control signal corresponding to the trigger operation to a receiving end through a Transmission Control Protocol (TCP) channel for corresponding control if the trigger operation of the sending end on the video stream and/or the audio stream is detected.
The sending processing device for audio and video on-screen playing described in this embodiment may be configured to execute the method embodiments, and the principle and technical effect are similar, which are not described herein again.
Fig. 5 is a schematic structural diagram of a receiving processing device for audio and video on-screen playing provided in this embodiment, where the device includes: audio video receiving module 501, audio video assembly module 502 and audio video decoding module 503, wherein:
the audio/video receiving module 501 is configured to receive an output video stream with a first timestamp and an output audio stream with a second timestamp sent by a sending end;
the audio and video assembling module 502 is configured to perform assembling processing on the output video stream and the output audio stream according to the first timestamp and the second timestamp to obtain an assembled audio and video;
the audio/video decoding module 503 is configured to decode and play the assembled audio/video to achieve the same-screen playing of the transmitting end and the receiving end.
Specifically, the audio/video receiving module 501 receives an output video stream with a first timestamp and an output audio stream with a second timestamp sent by a sending end; the audio and video assembling module 502 assembles the output video stream and the output audio stream according to the first timestamp and the second timestamp to obtain an assembled audio and video; the audio/video decoding module 503 decodes the assembled audio/video and plays the audio/video to realize the on-screen playing of the transmitting end and the receiving end.
In this embodiment, the captured video stream and audio stream are encoded by the sending end, and the output video stream and the output audio stream with the timestamps are sent to the receiving end for playing, so that the sending end and the receiving end realize the same-screen playing without using other hardware, and the sending end and the receiving end are realized based on a common terminal, and have universality.
Further, on the basis of the above device embodiment, the audio/video assembly module 502 is specifically configured to sequence the output video streams according to the first time stamp, sequence the output audio streams according to the second time stamp, and synchronize the sequenced output video streams and output audio streams according to the first time stamp and the second time stamp, so as to obtain an assembled audio/video.
The receiving and processing device for audio and video on-screen playing described in this embodiment may be used to implement the above method embodiments, and the principle and technical effect are similar, which are not described herein again.
Referring to fig. 6, the first electronic device includes: a first processor (processor)601, a first memory (memory)602, and a first bus 603;
wherein,
the first processor 601 and the first memory 602 communicate with each other via the first bus 603;
the first processor 601 is configured to call the program instruction in the first memory 602 to execute the sending processing method for audio and video on-screen playing provided in the foregoing method embodiment.
The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium, where the computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the transmission processing method for audio and video on-screen playing provided by the above method embodiments.
The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions, where the computer instructions enable the computer to execute the sending processing method for audio and video on-screen playing provided in the foregoing method embodiments.
Referring to fig. 7, the second electronic device includes: a second processor (processor)701, a second memory (memory)702, and a second bus 703;
wherein,
the second processor 701 and the second memory 702 complete communication with each other through the second bus 703;
the second processor 701 is configured to call the program instruction in the second memory 702 to execute the receiving and processing method for audio and video on-screen playing provided in the foregoing method embodiment.
The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium, where the computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the method for receiving and processing audio and video played on the same screen provided in the above method embodiment.
The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions, where the computer instructions cause the computer to execute the receiving and processing method for audio and video on-screen playing provided in the foregoing method embodiments.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
It should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A sending processing method for audio and video on-screen playing is characterized by comprising the following steps:
a sending end captures video stream and audio stream in the sending end;
encoding the captured video stream and audio stream to respectively generate an output video stream in a first preset format and an output audio stream in a second preset format;
and sending the output video stream with the first time stamp and the output audio stream with the second time stamp to a receiving end for playing, and playing the video stream and the audio stream in the sending end so as to realize the same-screen playing of the sending end and the receiving end.
2. The method for sending and processing audio and video on-screen play according to claim 1, wherein the sending end captures the video stream and the audio stream in the sending end, and specifically comprises:
a sending end captures a video stream in the sending end through a media projection interface and captures an audio stream in the sending end through an audio recording interface;
correspondingly, the encoding the captured video stream and audio stream to generate an output video stream in a first preset format and an output audio stream in a second preset format respectively includes:
and coding the captured video stream and audio stream through a media codec interface to respectively generate an output video stream in a first preset format and an output audio stream in a second preset format.
3. The method for sending and processing audio and video on-screen playing according to claim 1, wherein the sending of the output video stream with the first time stamp and the output audio stream with the second time stamp to a receiving end for playing specifically comprises:
and sending the output video stream with the first time stamp and the output audio stream with the second time stamp to a receiving end through a User Datagram Protocol (UDP) channel for playing.
4. The method for sending and processing audio and video on-screen playing according to any one of claims 1 to 3, wherein the method for sending and processing audio and video on-screen playing further comprises:
and if the triggering operation of the sending end on the video stream and/or the audio stream is detected, sending a control signal corresponding to the triggering operation to a receiving end through a Transmission Control Protocol (TCP) channel for corresponding control.
5. A receiving and processing method for audio and video on-screen playing is characterized by comprising the following steps:
a receiving end receives an output video stream with a first time stamp and an output audio stream with a second time stamp which are sent by a sending end;
assembling the output video stream and the output audio stream according to the first time stamp and the second time stamp to obtain an assembled audio and video;
and decoding and playing the assembled audio and video to realize the on-screen playing of the sending end and the receiving end.
6. The method for receiving and processing audio and video on-screen playing according to claim 5, wherein the assembling processing of the output video stream and the output audio stream according to the first timestamp and the second timestamp to obtain an assembled audio and video specifically comprises:
and sequencing the output video streams according to the first time stamp, sequencing the output audio streams according to the second time stamp, and synchronizing the sequenced output video streams and output audio streams according to the first time stamp and the second time stamp to obtain the assembled audio and video.
7. A sending processing device for audio and video on-screen playing is characterized by comprising:
the audio/video capture module is used for capturing the video stream and the audio stream in the sending end;
the audio and video coding module is used for coding the captured video stream and audio stream and respectively generating an output video stream in a first preset format and an output audio stream in a second preset format;
and the audio and video transmitting module is used for transmitting the output video stream with the first time stamp and the output audio stream with the second time stamp to a receiving end for playing, and playing the video stream and the audio stream in the transmitting end so as to realize the same-screen playing of the transmitting end and the receiving end.
8. A receiving and processing device for audio and video on-screen playing is characterized by comprising:
the audio and video receiving module is used for receiving an output video stream with a first time stamp and an output audio stream with a second time stamp which are sent by the sending end;
the audio and video assembly module is used for assembling the output video stream and the output audio stream according to the first time stamp and the second time stamp to obtain an assembled audio and video;
and the audio and video decoding module is used for decoding and playing the assembled audio and video so as to realize the same-screen playing of the transmitting end and the receiving end.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements a transmission processing method of audio-video on-screen playback according to any one of claims 1 to 4 and a reception processing method of audio-video on-screen playback according to claim 5 or 6 when executing the program.
10. A non-transitory computer-readable storage medium on which a computer program is stored, the computer program, when executed by a processor, implementing the transmission processing method of the audio-video on-screen playback according to any one of claims 1 to 4 and the reception processing method of the audio-video on-screen playback according to claim 5 or 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910316786.0A CN110062268A (en) | 2019-04-19 | 2019-04-19 | A kind of audio-video sends and receives processing method and processing device with what screen played |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910316786.0A CN110062268A (en) | 2019-04-19 | 2019-04-19 | A kind of audio-video sends and receives processing method and processing device with what screen played |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110062268A true CN110062268A (en) | 2019-07-26 |
Family
ID=67319676
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910316786.0A Pending CN110062268A (en) | 2019-04-19 | 2019-04-19 | A kind of audio-video sends and receives processing method and processing device with what screen played |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110062268A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110460904A (en) * | 2019-08-22 | 2019-11-15 | 聚好看科技股份有限公司 | Audio/video pushing method, mobile terminal and display terminal |
| CN110545454A (en) * | 2019-08-27 | 2019-12-06 | 北京奇艺世纪科技有限公司 | Data synchronous playing method and device |
| CN112860208A (en) * | 2021-04-25 | 2021-05-28 | 深圳乐播科技有限公司 | Screen-projection-based software sharing method, device, equipment and storage medium |
| CN113364646A (en) * | 2021-06-03 | 2021-09-07 | 杭州朗和科技有限公司 | Method, device and system for determining round-trip delay, storage medium and electronic equipment |
| CN113438224A (en) * | 2021-06-23 | 2021-09-24 | 广州智会云科技发展有限公司 | Method and system for multiple persons to jointly participate in online exhibition |
| WO2022032458A1 (en) * | 2020-08-10 | 2022-02-17 | 北京小米移动软件有限公司 | Multi-modal data transmission method and apparatus, and multi-modal data processing method and apparatus |
| CN114697733A (en) * | 2020-12-31 | 2022-07-01 | 华为技术有限公司 | Transmission method of screen projection audio and video data and related equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060176396A1 (en) * | 2005-01-27 | 2006-08-10 | Samsung Electronics Co.; Ltd | Apparatus for receiving transport stream to provide multi-screen and control method therefor |
| CN103150132A (en) * | 2013-02-21 | 2013-06-12 | 海信集团有限公司 | Screen transmission method and electronic equipment |
| CN103517153A (en) * | 2012-11-30 | 2014-01-15 | Tcl集团股份有限公司 | Method and system for achieving one-screen video playing in multi-screen mode |
| CN109257641A (en) * | 2018-09-05 | 2019-01-22 | 福建星网智慧科技股份有限公司 | Wireless screen transmission sound intermediate frequency and video synchronization method and system |
| CN109361945A (en) * | 2018-10-18 | 2019-02-19 | 广州市保伦电子有限公司 | A fast transmission and synchronization conference audio-visual system and its control method |
-
2019
- 2019-04-19 CN CN201910316786.0A patent/CN110062268A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060176396A1 (en) * | 2005-01-27 | 2006-08-10 | Samsung Electronics Co.; Ltd | Apparatus for receiving transport stream to provide multi-screen and control method therefor |
| CN103517153A (en) * | 2012-11-30 | 2014-01-15 | Tcl集团股份有限公司 | Method and system for achieving one-screen video playing in multi-screen mode |
| CN103150132A (en) * | 2013-02-21 | 2013-06-12 | 海信集团有限公司 | Screen transmission method and electronic equipment |
| CN109257641A (en) * | 2018-09-05 | 2019-01-22 | 福建星网智慧科技股份有限公司 | Wireless screen transmission sound intermediate frequency and video synchronization method and system |
| CN109361945A (en) * | 2018-10-18 | 2019-02-19 | 广州市保伦电子有限公司 | A fast transmission and synchronization conference audio-visual system and its control method |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110460904A (en) * | 2019-08-22 | 2019-11-15 | 聚好看科技股份有限公司 | Audio/video pushing method, mobile terminal and display terminal |
| CN110545454A (en) * | 2019-08-27 | 2019-12-06 | 北京奇艺世纪科技有限公司 | Data synchronous playing method and device |
| WO2022032458A1 (en) * | 2020-08-10 | 2022-02-17 | 北京小米移动软件有限公司 | Multi-modal data transmission method and apparatus, and multi-modal data processing method and apparatus |
| US12003604B2 (en) | 2020-08-10 | 2024-06-04 | Beijing Xiaomi Mobile Software Co., Ltd. | Multimodal data transmission method and apparatus, and multimodal data processing method and apparatus |
| CN114697733A (en) * | 2020-12-31 | 2022-07-01 | 华为技术有限公司 | Transmission method of screen projection audio and video data and related equipment |
| CN112860208A (en) * | 2021-04-25 | 2021-05-28 | 深圳乐播科技有限公司 | Screen-projection-based software sharing method, device, equipment and storage medium |
| CN113364646A (en) * | 2021-06-03 | 2021-09-07 | 杭州朗和科技有限公司 | Method, device and system for determining round-trip delay, storage medium and electronic equipment |
| CN113438224A (en) * | 2021-06-23 | 2021-09-24 | 广州智会云科技发展有限公司 | Method and system for multiple persons to jointly participate in online exhibition |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107846633B (en) | Live broadcast method and system | |
| CN110062268A (en) | A kind of audio-video sends and receives processing method and processing device with what screen played | |
| CN107135417B (en) | A kind of throwing screen method and system of HLS protocol | |
| US8988486B2 (en) | Adaptive video communication channel | |
| US8805171B2 (en) | Constructing video frames and synchronizing audio data in a media player from data received via a plurality of diverse protocol stack paths | |
| US20210377330A1 (en) | Low-latency video internet streaming for management and transmission of multiple data streams | |
| US9282448B2 (en) | Method, system and apparatus for providing streaming media service | |
| CN107819809B (en) | Method and device for synchronizing content | |
| CN112616065B (en) | Screen image initiating method, device and system and readable storage medium | |
| US20110285863A1 (en) | Live television broadcasting system for the internet | |
| CN112019877A (en) | Screen projection method, device and equipment based on VR equipment and storage medium | |
| CN202759552U (en) | Multi-terminal video synchronous playing system based on IP network | |
| CN103856809A (en) | Method, system and terminal equipment for multipoint at the same screen | |
| CN114554277B (en) | Multimedia processing method, device, server and computer readable storage medium | |
| CN108366044B (en) | VoIP remote audio/video sharing method | |
| CN108282685A (en) | A kind of method and monitoring system of audio-visual synchronization | |
| CN115904281A (en) | Cloud desktop conference sharing method, server and computer readable storage medium | |
| JP2007150916A (en) | Communication system, terminal device and computer program | |
| WO2023231478A1 (en) | Audio and video sharing method and device, and computer-readable storage medium | |
| CN105959786A (en) | Audio and video playing method and device | |
| CN111263217A (en) | Wireless screen projection method, wireless screen projection system and computer readable storage medium thereof | |
| CN113014950A (en) | Live broadcast synchronization method and system and electronic equipment | |
| EP2479984A1 (en) | Device and method for synchronizing content received from different sources | |
| CN110602065A (en) | Live broadcast stream pushing method and device | |
| CN108353035B (en) | Method and apparatus for multiplexing data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190726 |
|
| RJ01 | Rejection of invention patent application after publication |