CN115209230A - Method for realizing real-time video transmission based on RTMP protocol - Google Patents

Abstract

The invention relates to a method for realizing real-time video transmission based on an RTMP protocol, which transmits video data to a streaming media server in a local area network environment for scheduling and using by a client. The method for realizing real-time video transmission comprises three stages of video frame data uploading, streaming media server storage and client streaming. In the video frame data uploading stage, the video data acquired by the network camera is uploaded to the streaming media server in the form of RTMP packets, so that the reliability of the transmission process is ensured; the streaming media server ensures the real-time property of transmission through an RTMP real-time module of Nginx; the client pull stream adopts the video stream transmitted by the FFmpeg decoding server to ensure the video accuracy. The method uses the local area network to maintain the overall operation Of the system, and improves the real-time performance Of the video transmission system by adjusting the interval Of GOPs (Group Of Pictures) in the video uploading and receiving.

Description

A realization method of real-time video transmission based on RTMP protocol

technical field

The invention relates to a realization method of real-time video transmission based on RTMP protocol. Real-time video transmission is the core component of video surveillance system and belongs to the field of communication. It is widely used in various fields such as live broadcast, transportation, medical treatment and military.

Background technique

In the field of transmission protocols, the industry has always used RTP as the network protocol for transmitting video streams. This protocol relies on the UDP network protocol for transmission. Since UDP is connectionless and unreliable, it is easy to cause packet loss and time loss during network transmission. Delay and jitter, when the network condition is poor, it will seriously affect the user experience. The RTMP protocol uses TCP as the transport layer protocol, which can effectively ensure the transmission quality of the video stream, and supports the dynamic transmission of sound and video from the server to the client. In addition, the FLV encapsulation format supported by the RTMP protocol can display a clear picture under low bit rate conditions, which is very suitable for watching videos when the bandwidth is insufficient. The use of RTMP protocol to transmit video data has the effect of reliability and stability.

The realization method of real-time video transmission based on RTMP protocol proposed by the present invention collects video data through a network camera, transmits the video data to Nginx streaming media server based on RTMP protocol, and provides about 3000 request connections through Nginx's epoll concurrency technology It improves the reliability and real-time performance of video transmission.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a realization method of real-time video transmission based on RTMP protocol, which can effectively solve the problem of packet loss in the process of video data transmission and the problem of picture jamming in the process of video playback, and has the advantages of high efficiency and low cost of video data transmission. delay, etc.

The present invention adopts the following technical means to realize:

1. A realization method of real-time video transmission based on RTMP protocol, its video transmission process is divided into three stages of video frame data upload, streaming media server storage and client pull stream:

1.1 Collect high-definition video data through a remote network camera, then convert the video data format to YUV420 format, encode it according to the H264 standard, and finally convert the context into an FLV format file that meets the RTMP protocol transmission requirements through FFmpeg software;

1.2 The streaming media server adopts the Nginx lightweight server, which can provide the corresponding URL for the uploaded video data for storage or broadcasting. During the process of receiving or broadcasting, an RTMP connection channel is established for transmission. After the video data reaches the streaming media server, it is Its own epoll technology realizes concurrent connections to multiple clients;

1.3 The video surveillance system client is built through the Qt framework, and the video data broadcast by the streaming media server is received by the method of cyclically opening the thread. Every time the thread is opened, a channel is opened for transmission, and each thread is monitored. Release resources immediately when decoding video;

1.4 To solve the problem of picture distortion by creating a FIFO buffer queue, first the client receives the video data transmitted by the streaming media server, stores the data in the FIFO buffer queue, the decoder reads the video data from the FIFO buffer queue for decoding, and the free space continues Cache video data.

2. the realization method of a kind of real-time video transmission based on RTMP protocol according to claim 1, is characterized in that: video input source transmits data to FFmpeg, is uploaded to Nginx server with RTMP data format after encoding, encapsulation, then The server forwards the data to the client. Before uploading data to establish a network connection, the server internally calls the ngx_rtmp_handshake function and the ngx_rtmp_handshake_recv function to make the video pusher and the server perform handshake operations, select the nginx-rtmp module of Nginx to build the streaming media server, and then modify the rtmp module in the configuration file. , to realize the forwarding of the video data stream.

3. the realization method of a kind of real-time video transmission based on RTMP protocol according to claim 1 is characterized in that: among the aforementioned 1.2, Nginx server establishes multiple RTMP connections to carry out data transmission, and this comprises multiple channels in a client connections and connections to multiple channels across multiple clients.

4. the realization method of a kind of real-time video transmission based on RTMP protocol according to claim 1, is characterized in that: in the aforementioned 1.4, FIFO buffer queue buffers video data at queue entry, and decodes video at queue exit.

5. a kind of realization method based on the real-time video transmission of RTMP protocol according to claim 1, is characterized in that: in the aforementioned 1.4, FIFO buffer queue buffer double key frame, and in pushing video data and obtaining video data both sides are set reasonable. gop_size size to adjust the volume of keyframes and the gap between GOP frames.

A method for realizing real-time video transmission based on RTMP protocol of the present invention has the following advantages:

1. Even when multiple thread channels are enabled, the system can achieve zero packet loss during video transmission, and can display continuously and stably in high-definition without distortion on the client side.

2. Through the local area network and the setting of the GOP key frame, the video surveillance system can always maintain a low video delay from the beginning of uploading to the process of receiving.

3. The server uses Nginx lightweight server has the characteristics of good scheduling resources, and can provide a good and stable connection for the client.

Description of drawings

Fig. 1 is the real-time video transmission flow chart of the present invention

Fig. 2 is the real-time video transmission architecture diagram of the present invention

Fig. 3 is the flow media forwarding flow chart of the present invention

Fig. 4 is the FIFO structure diagram of the present invention

Fig. 5 is the GOP frame structure diagram of the present invention

Fig. 6 is the flow chart of the video stream push module of the present invention

Fig. 7 is the flow chart of the video data acquisition module of the present invention

Detailed ways

The implementation of the present invention will be further described below in conjunction with the accompanying drawings:

Figure 1 is a flow chart of real-time video transmission, which includes three parts: the device side, the streaming media server side, and the client side. First set up the webcam to start video data collection, encode it according to the H.264 standard format, and encapsulate it according to the FLV encapsulation strategy provided by FFmpeg; then transmit the data packet to the streaming media server configured by Nginx through the LAN for broadcasting, and enter the waiting request Status; the client obtains the video by sending a request to the streaming media server. When the video reaches the client, it first enters the buffer area to be decoded and the thread is started, and finally plays the obtained real-time video through the playback module. RTMP/TCP protocol is adopted in the process of video data upload and video data acquisition.

Figure 2 is an architecture diagram of real-time video transmission, which is mainly divided into four parts: camera video capture, streaming media server, network transmission, and client. The four parts are connected by a switch router to form a local area network. The video data transmission is first collected by the camera, and then the RTMP connection is established through the local area network and transmitted to the streaming media server; the server mainly includes two servers: one is the streaming media server, responsible for video data. The forwarding function, the second is that the database server is responsible for the storage and query functions of user information and device information. If the number of users and devices is not large, the database server and the streaming media server can be built in the same server; the video surveillance client is responsible for managing user device information, and multiple clients are connected to the local area network and send messages to the streaming media server. Request to obtain real-time video data, and display the video data to the client for users to watch. Because Nginx is used as the streaming media server, it can support 3000 RTMP connections for different clients.

Fig. 3 is a flow chart of streaming media forwarding. The streaming media needs to go through three processes during the uploading process: establishing a network connection, establishing a network stream, and transmitting video data. Before uploading data to establish a network connection, the server internally calls the ngx_rtmp_handshake function and the ngx_rtmp_handshake_recv function to make the video pusher and the server perform a handshake operation. After the handshake is completed, the rtmp network connection is established. Nginx calls the internal function ngx_rtmp_recv to parse the message block, and uses the ngx_event_t structure Get the video data sent by the video pusher, and store the data in the in linked list structure of ngx_rtmp_stream_t. During the continuous connection process, the server uses the ngx_rtmp_cmd_connect_init function to initialize the network connection, and obtains the amf information through ngx_rtmp_receive_amf, where the amf information is the format information of the data exchanged between the remote server and the flash. Finally, set the necessary parameters in the connection process through ngx_rtmp_cmd_args, and use ngx_rtmp_notify_connect to notify the video data sender of the configured parameters, so as to complete the information establishment in the connection process.

A network stream represents a channel for sending audio and video data, and a network connection can contain multiple channels. After the ngx_rtmp_recv function parses the received data, the parsed data is passed to the ngx_rtmp_receive_message function to establish a network flow. After receiving the information, Nginx calls the processing functions ngx_rtmp_protocol_message_handler and ngx_rtmp_amf_message_handler. The former can process the message twice according to the type of the message, and the latter will process the corresponding amf control message. Finally, the network stream is initialized by the ngx_rtmp_cmd_create_stream_init function, and the ngx_rtmp_cmd_create_stream function is called to create the network stream. After the creation, the client and the server can transmit information to realize the stream forwarding function.

Figure 4 is the structure of the FIFO buffer queue. It has the characteristics of first-in, first-out. The problem of picture distortion is solved by creating a FIFO buffer queue. When the video data starts to transmit, the video data continues to enter the FIFO buffer queue until it is full of one frame of data size, and then The decoding thread performs decoding, and the video data is still buffered in the FIFO, forming a state of buffering and decoding at the same time; the decoder reads the video data from the FIFO, and the FIFO continuously downloads data from the streaming media server, so that when the network is unreliable, the client The FIFO buffer queue created by the terminal can make the user invisible visually, and the playback process is distortion-free and smooth.

FIG. 5 is a structural diagram of a GOP frame, which indicates the number of B frames between I frames and P frames. In the video stream, GOP (Group of Picture) frame is a group of continuous pictures, which is composed of one I frame and several B/P frames. It is the basic unit of video image codec access and has a certain volume. The GOP frame is the key frame of the video. At least one key frame is required for each second of video. When the video is uploaded, the data of each frame will be marked with a timing label before being transmitted on the network. Due to the large data volume of the GOP frame, when the network is uploaded When the environment is not good, the GOP frame may not be able to be downloaded within seconds, thus affecting the latency. The present invention adjusts the volume of the key frame by reducing the spacing between the I frame and the B frame in the GOP in both pushing the video data and acquiring the video data, and by caching the double key frames, the loading process can be completed in a short time, In order to reduce the delay generated in the video transmission process.

Fig. 6 is the flow chart of the video stream push module, the video data collection and push are equivalent to the data input source of the system, which affects the video quality and overall performance of the monitoring system. FFmpeg encodes, encapsulates and packs data. After the above two processes are completed, the pusher will initiate a connection request according to the server address specified in the encapsulation header. After handshake, a network connection will be created. After the connection between the pusher and the server is established, the pusher can transmit video data to the server through the network stream. .

FIG. 7 is a flowchart of a video data acquisition module. The video frame data receiving module is the most important module in the video surveillance client, and it is also the bridge connecting the server and the client during the video transmission process. The present invention encapsulates the video data acquisition process in the run() function of the thread in a multi-threaded manner, so that when n windows are played at the same time, n threads are opened to acquire the video stream from the streaming media server and display it to the client in the corresponding interface. In the process of receiving video frame data, the device list module of the client requests a URL (Uniform ResourceLocator, Uniform Resource Locator) of the device from the database server, and then requests video data from the streaming media server through the URL of the device. In order to prevent the phenomenon of monitoring video mosaic, the video data needs to enter the cache when it reaches the client. The client reads the video data from the cache, decapsulates, decodes, and transcodes the data to obtain image data in RGB format.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand: The specific embodiment of the invention is modified or some technical features are equivalently replaced; without departing from the spirit of the technical solution of the present invention, all of them should be included in the scope of the technical solution claimed in the present invention.

Claims (5)

1. A method for realizing real-time video transmission based on RTMP protocol, the video transmission process is divided into three stages of video frame data uploading, streaming media server storage and client terminal stream pulling:

1.1, acquiring high-definition video data through a remote network camera, converting the video data format into a YUV420 format, coding according to an H.264 standard, and finally converting the video data format into an FLV format file meeting the transmission requirement of an RTMP protocol through FFmpeg software encapsulation context;

1.2 the streaming media server adopts a Nginx lightweight server, can provide a corresponding website for the uploaded video data to be stored or broadcasted, establishes an RTMP connection channel for transmission in the process of receiving or broadcasting, and realizes concurrent connection of a plurality of clients through the epoll technology of the streaming media server after the video data reaches the streaming media server;

1.3 the client of the video monitoring system is built through a Qt framework, the video data broadcasted by the streaming media server is received by adopting a mode of circularly starting threads, one channel is opened for transmission each time the threads are started, each thread is monitored, and resources are released immediately when the threads stop decoding videos;

1.4, the problem of picture distortion is solved by creating a FIFO buffer queue, firstly, a client receives video data transmitted by a streaming media server, stores the data into the FIFO buffer queue, a decoder reads the video data from the FIFO buffer queue for decoding, and the video data is continuously buffered in an idle space.

2. The method of claim 1, wherein the RTMP protocol is applied to a real-time video transmission system, and the method comprises: the video input source transmits data to the FFmpeg, the data is uploaded to the Nginx server in an RTMP data format after being encoded and packaged, and then the server forwards the data to the client. Before network connection is established for uplink data, a video pushing end and a server are subjected to handshake operation by calling ngx _ rtm _ handshake function and ngx _ rtm _ handshake _ recv function in the server, a Nginx-rtmp module of Nginx is selected to build a streaming media server, and then forwarding of video data streams is achieved by modifying the rtmp module in a configuration file.

3. The method according to claim 1, wherein the RTMP protocol is used for real-time video transmission, and the method comprises: the aforementioned Nginx server in 1.2 establishes a plurality of RTMP connections for data transmission, which include connections of a plurality of channels in one client and connections of a plurality of channels in a plurality of clients.

4. The method according to claim 1, wherein the RTMP protocol is used for real-time video transmission, and the method comprises: the FIFO buffer queue in 1.4 above buffers video data at the entry of the queue and decodes video at the exit of the queue.

5. The method according to claim 1, wherein the RTMP protocol is used for real-time video transmission, and the method comprises: the FIFO buffer queue in 1.4 above buffers the double key frames and sets reasonable GOP _ size for both pushing video data and fetching video data to adjust the key frame volume and the gap between GOP frames.

Images