CN118972695A - Camera interaction method, system and server - Google Patents

Abstract

The present invention provides a camera interaction method, system and server, and relates to the field of camera interaction. The scheme reduces the pressure of network bandwidth by encoding and compressing the original data collected by the physical camera through an encoder arranged on the client; and realizes the decoupling between the physical camera and the virtual machine through a virtual camera additionally arranged in the server, further reducing the bandwidth pressure of the virtual bus of the virtual machine, thereby effectively alleviating the problems of application display screen faults, screen flashing or freeze caused by the virtual machine in the prior art when the network status is not good.

Description

Camera interaction method, system and server

Technical Field

The present invention relates to the field of camera interaction, and in particular to a camera interaction method, system and server.

Background Art

At present, most of the network-based camera interaction solutions adopt the redirection forwarding method. Specifically, the client host connects to the physical camera, transmits the relevant information of the camera to the corresponding server through the network, and then forwards it to the virtual machine through the virtual bus set by the server. When the relevant application in the virtual machine needs to use the camera, the video proxy driver can use the video data collected by the physical camera to display the recorded video screen.

This solution uses the original data packets collected by the physical camera and completes the camera interaction process between the virtual end and the physical end through operations such as encapsulation and unpacking of various protocols. However, this solution requires a higher network bandwidth, which will cause greater pressure on the virtual bus when there are a large number of physical cameras, especially when the network status is poor, which will cause the application display in the virtual machine to be disconnected, flicker or freeze.

Summary of the invention

In view of this, the purpose of the present invention is to provide a camera interaction method, system and server, which reduces the pressure on network bandwidth by encoding and compressing the original data collected by the physical camera through an encoder set on the client; and through the virtual camera additionally set in the server, the decoupling between the physical camera and the virtual machine is achieved, and the bandwidth pressure of the virtual bus of the virtual machine is further reduced, thereby effectively alleviating the problems of application display screen faults, screen flashing or freezes caused by the virtual machine in the prior art when the network status is not good.

In a first aspect, an embodiment of the present invention provides a camera interaction method, which is applied to a server including a virtual end and a physical end, wherein the virtual end is provided with a virtual machine and a virtual camera; the physical end includes: a client host, a physical camera and an encoder; the input end of the encoder is connected to the physical camera through the client host, and the output end of the encoder is connected to the input end of the virtual camera through the server end; the output end of the virtual camera is connected to the virtual machine;

The method includes:

Generate enumerated device data corresponding to the virtual camera based on the device connection parameters between the physical camera and the client where the encoder is located, and use the virtual connection instruction generated by the enumerated device data to control the virtual camera to complete the connection with the virtual machine;

When a camera request instruction issued by the virtual machine is detected, a video request instruction corresponding to the virtual camera is generated according to the camera request instruction, and after the server is controlled to send the video request instruction to the physical camera, callback data of the physical camera is received in real time;

The callback data is encoded by an encoder to obtain the video stream data corresponding to the physical camera, and the virtual camera is controlled to generate the video response data corresponding to the camera request instruction based on the video stream data;

The control server sends the video response data to the virtual machine.

In one implementation, generating enumerated device data corresponding to a virtual camera based on device connection parameters between a physical camera and a client where an encoder is located includes:

Based on the connection status parameters between the client host providing the physical camera working environment and the client where the encoder is located, the device connection parameters between the physical camera and the client where the encoder is located are obtained;

When it is detected that the physical camera is in a state of being inserted into the client host, the virtual camera is controlled to send a device descriptor request instruction to the physical camera based on the device connection parameters, and the device descriptor of the physical camera response is received in real time;

Control the virtual camera to send a configuration descriptor request instruction to the physical camera, and receive the configuration descriptor responded by the physical camera in real time;

Generates enumerated device data using device descriptors and configuration descriptors.

In one embodiment, the virtual connection instruction generated by enumerating device data is used to control the virtual camera to complete the connection with the virtual machine, including:

Cache one or more descriptor data of a device descriptor, a configuration descriptor, an interface descriptor, an endpoint descriptor, a video head descriptor, a terminal descriptor, a processing unit descriptor, and a video frame descriptor contained in the enumerated device data to a storage unit of the virtual camera;

The control server obtains the descriptor data from the storage unit, and generates a device loading instruction corresponding to the virtual camera using the descriptor data;

The device loading instruction is used to control the virtual camera to be loaded into the virtual machine and perform connection processing.

In one implementation, after the control server sends the video request instruction to the physical camera, the callback data of the physical camera is received in real time, including:

Obtain the corresponding physical camera based on the video request instruction;

The control server sends the video request instruction to the corresponding physical camera, and controls the physical camera to collect and obtain video stream data in real time according to the video request instruction;

After converting the video stream data according to the format requirements of the virtual camera, the physical camera is controlled to send the video stream data that has completed the format conversion as callback data to the server;

Control the server to receive callback data sent by the physical camera in real time.

In one implementation, the callback data is encoded by an encoder to obtain video stream data corresponding to the physical camera, including:

Based on the callback data, the initial data corresponding to the physical camera is obtained; wherein the initial data is YUV encoded data;

After controlling the encoder to perform encoding conversion on the initial data, converted encoded data is obtained; wherein the converted encoded data is one or more encoded data of H265, H264, MJPEG, etc.;

Generate video stream data corresponding to the physical camera based on the converted encoded data.

In one implementation, controlling the virtual camera to generate video response data corresponding to the camera request instruction based on the video stream data includes:

Determine whether the preset storage area of the virtual camera contains historical data corresponding to the video stream data;

If yes, then the video response data is generated based on the historical data; if no, then the video response data is generated based on the video stream data, and the video response data is saved in the storage area as historical data.

In one embodiment, after detecting a camera closing instruction issued by the virtual machine, the method further includes:

Control the virtual machine to send a first closing instruction to the virtual camera, and use the first closing instruction to control the virtual camera to be in a closed state;

The control server sends a second shutdown instruction to the physical camera, and uses the second shutdown instruction to control the physical camera to be in a closed state.

In one embodiment, when it is detected that the physical camera is in an offline state, the method further includes:

The control server generates a close instruction corresponding to the virtual camera, and controls the server to send the close instruction to the virtual machine.

In a second aspect, an embodiment of the present invention provides a camera interaction system, which is applied to a server including a virtual end and a physical end, wherein the virtual end is provided with a virtual machine and a virtual camera; the physical end includes: a client host, a physical camera and an encoder; the input end of the encoder is connected to the physical camera through the client host, and the output end of the encoder is connected to the input end of the virtual camera through the server end; the output end of the virtual camera is connected to the virtual machine;

The system includes:

A connection interaction module is used to generate enumerated device data corresponding to the virtual camera based on the device connection parameters between the physical camera and the client where the encoder is located, and use the virtual connection instructions generated by the enumerated device data to control the virtual camera to complete the connection with the virtual machine;

The instruction interaction module is used to generate a video request instruction corresponding to the virtual camera according to the camera request instruction after monitoring the camera request instruction issued by the virtual machine, and control the server to send the video request instruction to the physical camera, and receive the callback data of the physical camera in real time;

The data interaction module is used to encode the callback data using an encoder to obtain the video stream data corresponding to the physical camera, and control the virtual camera to generate video response data corresponding to the camera request instruction based on the video stream data;

The video interaction module is used to control the server to send video response data to the virtual machine.

In a third aspect, an embodiment of the present invention further provides a server, the server comprising a virtual end and a physical end, the virtual end being provided with a virtual machine and a virtual camera; the physical end comprising: a client host, a physical camera and an encoder; the input end of the encoder is connected to the physical camera via the client host, the output end of the encoder is connected to the input end of the virtual camera via the server end; the output end of the virtual camera is connected to the virtual machine;

During the process of video data interaction between the virtual camera and the physical camera, the server adopts the steps of the camera interaction method mentioned in the first aspect above.

In a fourth aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the camera interaction method provided in the first aspect.

In a fifth aspect, an embodiment of the present invention further provides a storage medium, which stores computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions prompt the processor to implement the steps of the camera interaction method provided in the first aspect.

A camera interaction method, system and server provided by the embodiment of the present invention are applied to a server including a virtual end and a physical end, wherein the virtual end is provided with a virtual machine and a virtual camera; the physical end includes: a client host, a physical camera and an encoder; the input end of the encoder is connected to the physical camera through the client host, and the output end of the encoder is connected to the input end of the virtual camera through the server end of the server; the output end of the virtual camera is connected to the virtual machine. In the process of video data interaction between the virtual camera and the physical camera, the method first generates enumerated device data corresponding to the virtual camera based on the device connection parameters between the physical camera and the client where the encoder is located, and uses the virtual connection instruction generated by the enumerated device data to control the virtual camera and the virtual machine to complete the connection; when the camera request instruction issued by the virtual machine is monitored, a video request instruction corresponding to the virtual camera is generated according to the camera request instruction, and after controlling the server end to send the video request instruction to the physical camera, the callback data of the physical camera is received in real time; then the callback data is encoded by the encoder to obtain the video stream data corresponding to the physical camera, and the virtual camera is controlled to generate the video response data corresponding to the camera request instruction based on the video stream data; finally, the server end is controlled to send the video response data to the virtual machine. This solution reduces the pressure on network bandwidth by encoding and compressing the raw data collected by the physical camera through an encoder set up on the client; and through the additional virtual camera set up on the server, it achieves decoupling between the physical camera and the virtual machine, further reducing the bandwidth pressure of the virtual machine's virtual bus, thereby effectively alleviating the problems of application display interruptions, screen flashing or freezing caused by the virtual machine in the existing technology when the network status is poor.

Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a camera interaction method provided by an embodiment of the present invention;

FIG2 is a flow chart of generating enumerated device data corresponding to a virtual camera based on device connection parameters between a physical camera and an encoder in a camera interaction method provided by an embodiment of the present invention;

3 is a flow chart of controlling a virtual camera to complete connection with a virtual machine by using a virtual connection instruction generated by enumerating device data in a camera interaction method provided by an embodiment of the present invention;

FIG4 is a flow chart of a camera interaction method provided by an embodiment of the present invention, in which a control server sends a video request instruction to a physical camera and then receives callback data of the physical camera in real time;

FIG5 is a flow chart of obtaining video stream data corresponding to a physical camera after encoding callback data using an encoder in a camera interaction method provided by an embodiment of the present invention;

6 is a flow chart of controlling a virtual camera to generate video response data corresponding to a camera request instruction based on video stream data in a camera interaction method provided by an embodiment of the present invention;

FIG7 is a flow chart of a camera interaction method provided by an embodiment of the present invention, when a camera closing instruction issued by a virtual machine is detected;

FIG8 is an interaction flow chart of a camera interaction method provided by an embodiment of the present invention;

FIG9 is a schematic structural diagram of a camera interaction system provided by an embodiment of the present invention;

FIG10 is a schematic diagram of the structure of a server provided in an embodiment of the present invention;

FIG. 11 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

icon:

910-connection interaction module; 920-command interaction module; 930-data interaction module; 940-video interaction module;

10-virtual terminal; 10a-virtual machine; 10b-virtual camera;

20-physical end; 20a-client host; 20b-physical camera; 20c-encoder;

30-Server;

101 - processor; 102 - memory; 103 - bus; 104 - communication interface.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

At present, most of the network-based camera interaction solutions adopt the redirection forwarding method. Specifically, the client host connects to a physical camera, such as a USB interface camera, and transmits the relevant information of the camera to the corresponding server through the network, and then forwards it to the virtual machine through the USB virtual bus set by the server. When the relevant application in the virtual machine needs to use the camera, the video data collected by the physical camera can be used through the video proxy driver to display the recorded video screen.

Taking USB cameras as an example, this type of solution uses the original URB data packets collected by the USB camera, and completes the camera interaction process between the virtual end and the physical end through operations such as encapsulation and unpacking of various protocols. Due to the large amount of data in the URB packet, a higher network bandwidth is required, especially when there are a large number of USB cameras, which will cause greater pressure on the USB virtual bus. In actual use, when the network status is poor, the application in the virtual machine will display screen faults, flashing or freezing, and the user experience will be poor.

Based on this, the present invention provides a camera interaction method, system and server, which reduces the pressure on network bandwidth by encoding and compressing the original data collected by the physical camera through an encoder set on the client; and through the virtual camera additionally set in the server, the decoupling between the physical camera and the virtual machine is achieved, further reducing the bandwidth pressure of the virtual machine's virtual bus, and effectively alleviating the problems of application display screen faults, screen flashing or freezes caused by the virtual machine in the prior art when the network status is not good, thereby solving the above-mentioned problems existing in the prior art.

To facilitate understanding of this embodiment, first, a camera interaction method disclosed in an embodiment of the present invention is applied to a server including a virtual end and a physical end, wherein the virtual end is provided with a virtual machine and a virtual camera; the physical end includes: a client host, a physical camera and an encoder; the input end of the encoder is connected to the physical camera through the client host, and the output end of the encoder is connected to the input end of the virtual camera through the server end; the output end of the virtual camera is connected to the virtual machine;

As shown in FIG1 , the method comprises the following steps:

Step S101, generating enumerated device data corresponding to the virtual camera based on the device connection parameters between the physical camera and the client where the encoder is located, and controlling the virtual camera to complete the connection with the virtual machine using the virtual connection instruction generated by the enumerated device data;

Step S102, when a camera request instruction issued by the virtual machine is monitored, a video request instruction corresponding to the virtual camera is generated according to the camera request instruction, and after the server is controlled to send the video request instruction to the physical camera, callback data of the physical camera is received in real time;

Step S103, using an encoder to encode the callback data to obtain video stream data corresponding to the physical camera, and controlling the virtual camera to generate video response data corresponding to the camera request instruction based on the video stream data;

Step S104, controlling the server to send the video response data to the virtual machine.

The camera interaction method in this embodiment additionally sets up an encoder and a virtual camera at the structural level, connects the encoder to the physical camera through the client host, and connects the virtual camera to the virtual machine, and the encoder and the virtual camera are connected to the server end of the server through a network transmission line, thereby realizing a video data interaction channel to implement a specific interaction process.

Specifically, the method includes the interaction of two types of data, one is the enumerated device data generated based on the device connection parameters between the physical camera and the client where the encoder is located, and the other is the video stream data processed by the encoder. Specifically, the enumerated device data contains various device descriptors, which can be used to set the interaction instructions; and the video stream data is simplified after being encoded by the encoder, and then the transmission interaction process is performed.

After obtaining the enumerated device data, the method first executes the device connection process, uses the virtual connection instruction generated by the enumerated device data to control the virtual camera to complete the connection with the virtual machine, and can simultaneously execute the connection process of the physical camera, so that after completing the connection with the encoder, it can interact with the virtual camera through the server.

When the camera request command issued by the virtual machine is detected, the video request command corresponding to the virtual camera is first generated according to the command, and then the server is controlled to send the command to the physical camera for calling, and the callback data when the physical camera responds to the command is received in real time. In layman's terms, when the virtual machine requests to interact with the physical camera, the request command is first converted into the relevant command of the virtual camera, and the virtual camera on the virtual machine side is triggered at this time; then the virtual camera is associated with the physical camera, and the video request command is sent to the physical camera through the server, thereby calling the physical camera.

After the physical camera responds to the command, its video stream is sent to the server as callback data. At this time, the callback data is encoded by the encoder to obtain the compressed video stream data corresponding to the physical camera, and the video response data corresponding to the camera request command is generated based on the video stream data, and finally sent to the virtual machine to realize the camera interaction process.

In one implementation, enumerated device data corresponding to a virtual camera is generated based on device connection parameters between a physical camera and an encoder, as shown in FIG2 , including:

Step S201, based on the connection status parameters between the client host providing the physical camera working environment and the client where the encoder is located, obtaining the device connection parameters between the physical camera and the client where the encoder is located;

Step S202, when it is detected that the physical camera is in a state of being inserted into the client host, the virtual camera is controlled based on the device connection parameters to send a device descriptor request instruction to the physical camera, and the device descriptor of the physical camera response is received in real time;

Step S203, controlling the virtual camera to send a configuration descriptor request instruction to the physical camera, and receiving the configuration descriptor responded by the physical camera in real time;

Step S204: Generate enumerated device data using the device descriptor and the configuration descriptor.

The enumerated device data is essentially generated based on the connection status parameters between the client host and the client where the encoder is located. This stage can be understood as the connection stage. At this time, the physical camera needs to be connected to the client host. The client host is an actual host with a built-in USB interface. The physical camera is inserted into the client host through the USB interface to achieve connection. In the process of obtaining the enumerated device data, firstly, the device connection parameters between the physical camera and the encoder are obtained, and then it is determined whether the physical camera is inserted into the client host. From the perspective of the client, when the client host decides to connect the device, the step of obtaining the enumerated device data is executed. At this time, a device descriptor request instruction is sent to the physical camera based on the device connection parameters, and the device descriptor responded by the physical camera is received in real time; in addition, a configuration descriptor request instruction is sent to the physical camera, and the configuration descriptor responded by the physical camera is received in real time, and then the required enumerated device data is generated using the obtained device descriptor and configuration descriptor.

In one embodiment, the virtual connection instruction generated by enumerating device data is used to control the virtual camera to complete the connection with the virtual machine, as shown in FIG3, including:

Step S301, caching one or more descriptor data of a device descriptor, a configuration descriptor, an interface descriptor, an endpoint descriptor, a video head descriptor, a terminal descriptor, a processing unit descriptor, and a video frame descriptor contained in the enumerated device data into a storage unit of the virtual camera;

Step S302, controlling the server to obtain descriptor data from the storage unit, and using the descriptor data to generate a device loading instruction corresponding to the virtual camera;

Step S303, using the device loading instruction to control the virtual camera to be loaded into the virtual machine and perform connection processing.

Specifically, the connection phase is the virtual camera data preparation phase, which starts when the client decides to connect the device and sends the query command CMD_VCAM_CHECK to check whether the virtual camera is supported. If there is no response within the timeout (such as 3 seconds), it is not supported, otherwise the virtual camera requests the device descriptor, which describes the basic information of the device, such as pid, vid and other important identifiers. After the client responds, continue to request the device's configuration descriptor information, and the client responds with the configuration descriptor and all its sub-descriptors, such as interface descriptors, endpoint descriptors, VC, VS and other descriptor information. All information will be cached in the virtual camera. When this information is prepared, the virtual camera is inserted into the virtual machine in the attach mode. At this time, the camera information can be viewed in the device manager in the virtual machine, and this connection phase ends.

In one implementation, after the control server sends the video request instruction to the physical camera, the callback data of the physical camera is received in real time, as shown in FIG4 , including:

Step S401, obtaining a corresponding physical camera based on a video request instruction;

Step S402, controlling the server to send the video request instruction to the corresponding physical camera, and controlling the physical camera to collect and obtain video stream data in real time according to the video request instruction;

Step S403, after converting the video stream data according to the format requirements of the virtual camera, the physical camera is controlled to send the video stream data that has completed the format conversion as callback data to the server;

Step S404, controlling the server to receive the callback data sent by the physical camera in real time.

This can be simply understood as the data transmission stage, which is the process of the virtual camera transmitting data to the virtual machine in a mapping manner. First, the corresponding physical camera is determined based on the video request instruction, and then the video request instruction is sent to the corresponding physical camera under the control of the server, so that the physical camera performs the work according to the video request instruction, thereby collecting and obtaining video stream data.

It is worth mentioning that the format of the acquired video stream data has certain requirements and needs to be converted according to the format requirements of the virtual camera, so that the formatted video stream data is used as callback data for reception by the server. Specifically, in one embodiment, the callback data is encoded by an encoder to obtain the video stream data corresponding to the physical camera, as shown in FIG5, including:

Step S501, obtaining initial data corresponding to the physical camera based on the callback data; wherein the initial data is YUV encoded data;

Step S502, controlling the encoder to perform encoding conversion on the initial data to obtain converted encoded data; wherein the converted encoded data is one or more encoded data of H265, H264, MJPEG, etc.;

Step S503, generating video stream data corresponding to the physical camera according to the converted coded data.

In specific scenarios, the device video stream data is the video data. At this time, it is no longer necessary to take the traditional single-packet URB data from the physical camera. Instead, the complete YUV data is taken from the USB camera frame by frame. After passing through the newly added encoder, it is encoded into the corresponding encoded data (such as H265/H264/MJPEG) according to the user's selection in the virtual machine. When the user controls the relevant application in the virtual machine to open the camera, the virtual camera will continuously request data frames from the client. When data from the USB camera is detected, it is immediately reported to the virtual machine, and the software in the virtual machine can see the video screen.

In one implementation, controlling the virtual camera to generate video response data corresponding to the camera request instruction based on the video stream data, as shown in FIG6 , includes:

Step S601, determining whether the preset storage area of the virtual camera contains historical data corresponding to the video stream data;

Step S602: if yes, generate video response data based on the historical data; if no, generate video response data based on the video stream data, and save the video response data as historical data in the storage area.

In actual scenarios, the data correctness, frame integrity and expiration of each frame of data in the preset storage area of the virtual camera can be determined. If it is expired data, it is discarded. If not, video response data is generated based on the video stream data and saved to the storage area. Once the virtual machine issues a data request instruction, the virtual camera submits the oldest frame of video data in the storage area to the virtual bus address.

The process of obtaining the video response data can be combined with whether the virtual camera contains the corresponding historical video stream data. Specifically, after the virtual camera obtains the data, it waits for the virtual machine to issue an instruction to obtain the data. When the virtual machine sends a request, it finds the corresponding historical data in the storage area and maps it to the virtual machine, so that the application in the virtual machine displays the picture. When there is no data in the relevant storage area of the virtual camera, it immediately requests and obtains the next video response data, and repeats this process to achieve the effect of playing the video.

In one embodiment, after detecting a camera closing instruction issued by the virtual machine, as shown in FIG7 , the method further includes:

Step S701, controlling the virtual machine to send a first closing instruction to the virtual camera, and using the first closing instruction to control the virtual camera to be in a closed state;

Step S702, control the server to send a second closing instruction to the physical camera, and use the second closing instruction to control the physical camera to be in a closed state.

When the user turns off the camera in the virtual machine, the virtual machine is first controlled to send a first closing instruction to the virtual camera, thereby turning off the virtual camera; then the server is controlled to send a second closing instruction to the physical camera, thereby turning off the physical camera. During the entire interaction process, the virtual camera and the physical camera complete direct interaction.

In one embodiment, when it is detected that the physical camera is in an offline state, the method further includes: controlling the server to generate a shutdown instruction corresponding to the virtual camera, and controlling the server to send the shutdown instruction to the virtual machine.

The disconnection phase is also a phase that requires user operation to trigger. It is the process of whether the virtual camera is a resource. When the user disconnects the camera or unplugs the camera, the client will send a disconnect camera command to the virtual camera. After the virtual camera receives the information, it will be unplugged from the virtual camera. There will be no camera-related information in the virtual machine. At the same time, the virtual camera will reclaim all memory and other resources.

In summary, the camera interaction method in this embodiment adds an encoder on the client execution side and a virtual camera on the server execution side. At the same time, the video proxy driver is no longer required inside the virtual machine. It mainly optimizes the processing of data streams.

The data stream is divided into two parts, namely, enumerated device data and video stream data. The enumerated device-related data includes data such as device descriptors, configuration descriptors, interface descriptors, endpoint descriptors, VC header descriptors, VC input/output terminal descriptors, VC processing unit descriptors, VS input header descriptors, and video frame descriptors of various encoding types. After getting it from the USB camera, it is directly transferred to the newly added virtual uvc camera for storage. The format description information supported by the uvc camera is fixed to the encoding format, and is no longer the data format in the USB device. After the virtual uvc camera obtains all the required information from the USB camera through the network, the virtual uvc camera is connected to the virtual machine system. In the device manager in the virtual machine, you can view the relevant information of the virtual camera device.

The device video stream data is the video data. Instead of taking the traditional single-packet URB data from the device, it takes complete frames of YUV data from the USB camera. After passing through the newly added encoder, it is encoded into the corresponding encoded data (such as H265/H264/MJPEG) according to the user's selection in the virtual machine. When the user opens the camera in the virtual machine, the virtual UVC camera will continuously request data frames from the client. When data from the USB camera is detected, it is immediately reported to the virtual machine, and the software in the virtual machine can see the video screen.

When the user turns off the camera in the virtual machine, the virtual uvc camera will notify the usb camera to turn off the video stream. In the whole process of interaction, the virtual uvc camera interacts directly with the USB camera. The above data are all packaged and unpacked using the custom VCAM protocol as the carrier. The following describes the VCAM protocol with the virtual camera as the focus. The virtual camera uses qemu to virtualize an empty device of uvc. The device data filling process is shown in Figure 8, which is an interaction flow chart of a camera interaction method. It can be seen from the figure that the physical camera client host interacts with the virtual camera in a network manner, and the virtual camera interacts with the virtual machine in a uvc manner. The entire interaction process is roughly divided into three stages: connection stage, data transmission stage, and disconnection stage.

The connection phase is the virtual camera data preparation phase, which starts when the client decides to connect the device and sends a query command CMD_VCAM_CHECK to check whether the virtual camera is supported. If there is no response within the timeout (such as 3 seconds), it is not supported. Otherwise, the virtual camera requests the device descriptor (command CMD_VCAM_DEVICE_DESC), which describes the basic information of the device, such as pid, vid and other important identifiers. After the client responds, continue to request the device's configuration descriptor information (command CMD_VCAM_CFG_DESC), and the client responds with the configuration descriptor and all its sub-descriptors, such as interface descriptors, endpoint descriptors, VC, VS and other descriptor information. All information will be cached in the virtual camera. When this information is prepared, the virtual camera is inserted into the virtual machine in the attach mode. At this time, the camera information can be viewed in the device manager in the virtual machine, and this connection phase ends.

The data transmission stage is a stage that needs to be triggered by user operation. It is the process in which the virtual camera transmits data to the virtual machine through the UVC protocol in a mapped manner. When the user clicks to open the camera player and plays it, the virtual camera will receive the relevant instructions to open the camera, and transmit the instruction CMD_VCAM_STREAM_ON to the client through the network. After receiving the instruction, the client gets the video data n of the physical camera and transmits it to the virtual camera through the instruction CMD_VCAM_STREAM_TRANS. After the virtual camera gets the data, it waits for the virtual machine instruction to send the data. When the virtual machine sends the request, it finds that there is data and can map it to the virtual machine, so that the software displays the picture. When there is no data in the virtual camera, it immediately requests the next data n+1, and repeats this process to achieve the effect of playing the video. When the user closes the player or turns off the video playback, the virtual camera will receive the shutdown instruction from the virtual machine, and can send the CMD_VCAM_STREAM_OFF instruction to the client, and the client will turn off the data transmission.

The disconnection phase is also a phase that requires user operation to trigger. It is the process of whether the virtual camera is a resource. When the user disconnects the camera or unplugs the camera, the client will send the disconnect camera command CMD_VCAM_DETACH to the virtual camera. After the virtual camera gets the information, it will be unplugged from the virtual camera (ie, detach), and there will be no more camera-related information in the virtual machine. At the same time, the virtual camera will reclaim all memory and other resources.

It can be seen from the camera interaction method in the above embodiment that this solution abandons the large data volume transmission of urb and uses coded data transmission with small data volume. In the process of executing remote camera communication, it has the characteristics of low data volume and low network bandwidth requirements. It can support more devices under the same bandwidth and reduce the pressure of the virtual bus.

In actual scenarios, the video images in applications running in virtual machines are more stable, and are less likely to experience screen faults, screen flashes, or freezes. This method better decouples the physical camera and is more controllable, because compared to the traditional solution where all data depends on the USB physical camera, the solution of the present invention only needs to take its basic data encoding for use by the virtual UVC camera, and the data can be self-policy controlled in the virtual UVC camera, including other device description information that can be controlled by itself, and no longer entirely dependent on the actual USB camera.

For the camera interaction method provided in the above embodiment, an embodiment of the present invention provides a camera interaction system, as shown in FIG9 , the system is applied to a server including a virtual end and a physical end, the virtual end is provided with a virtual machine and a virtual camera; the physical end includes: a client host, a physical camera and an encoder; the input end of the encoder is connected to the physical camera through the client host, and the output end of the encoder is connected to the input end of the virtual camera through the server end; the output end of the virtual camera is connected to the virtual machine;

The system includes:

A connection interaction module 910 is used to generate enumerated device data corresponding to the virtual camera based on the device connection parameters between the physical camera and the client where the encoder is located, and use the virtual connection instruction generated by the enumerated device data to control the virtual camera to complete the connection with the virtual machine;

The instruction interaction module 920 is used to generate a video request instruction corresponding to the virtual camera according to the camera request instruction after monitoring the camera request instruction issued by the virtual machine, and control the server to send the video request instruction to the physical camera, and receive the callback data of the physical camera in real time;

The data interaction module 930 is used to encode the callback data using an encoder to obtain video stream data corresponding to the physical camera, and control the virtual camera to generate video response data corresponding to the camera request instruction based on the video stream data;

The video interaction module 940 is used to control the server to send the video response data to the virtual machine.

From the camera interaction system mentioned in the above embodiment, it can be seen that the system reduces the pressure on network bandwidth by encoding and compressing the original data collected by the physical camera through the encoder set on the client; and through the virtual camera additionally set in the server, the decoupling between the physical camera and the virtual machine is realized, and the bandwidth pressure of the virtual bus of the virtual machine is further reduced, thereby effectively alleviating the problems of application display screen faults, screen flashing or freezes caused by the virtual machine in the prior art when the network status is poor.

The camera interaction system in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned camera interaction method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned camera interaction method embodiment.

The present embodiment also provides a server, as shown in FIG10 , the server comprises a virtual terminal 10 and a physical terminal 20, the virtual terminal 10 is provided with a virtual machine 10a and a virtual camera 10b; the physical terminal 20 comprises: a client host 20a, a physical camera 20b and an encoder 20c; the input end of the encoder 20c is connected to the physical camera 20b through the client host 20a, the output end of the encoder 20c is connected to the input end of the virtual camera 10b through the server end 30; the output end of the virtual camera 10b is connected to the virtual machine 10a;

During the process of video data interaction between the virtual camera 10b and the physical camera 20b, the server adopts the steps of the camera interaction method mentioned in the above embodiment.

In the embodiment of the present invention, when the server adopts the camera interaction method, its implementation principle and the technical effects produced are the same as those in the aforementioned camera interaction method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding contents in the aforementioned camera interaction method embodiment.

This embodiment also provides an electronic device, the structural diagram of the electronic device is shown in Figure 11, and the device includes a processor 101 and a memory 102; wherein the memory 102 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the steps of the above-mentioned camera interaction method.

The electronic device shown in FIG. 11 further includes a bus 103 and a communication interface 104 , and the processor 101 , the communication interface 104 and the memory 102 are connected via the bus 103 .

The memory 102 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The bus 103 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, FIG. 11 only uses one bidirectional arrow to represent, but it does not mean that there is only one bus or one type of bus.

The communication interface 104 is used to connect to at least one user terminal and other network units through a network interface, and send the encapsulated IPv4 message or IPv4 message to the user terminal through the network interface.

The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 102, and the processor 101 reads the information in the memory 102 and completes the steps of the method of the above embodiment in combination with its hardware.

An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the camera interaction method in the aforementioned embodiment are executed.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that can be executed by a processor. Based on this understanding, the technical solution of the present invention can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims (10)

1. A camera interaction method, characterized in that the method is applied to a server including a virtual end and a physical end, wherein the virtual end is provided with a virtual machine and a virtual camera; the physical end includes: a client host, a physical camera and an encoder; the input end of the encoder is connected to the physical camera through the client host, the output end of the encoder is connected to the input end of the virtual camera through the service end of the server; the output end of the virtual camera is connected to the virtual machine; The method comprises: Generate enumerated device data corresponding to the virtual camera based on device connection parameters between the physical camera and the client where the encoder is located, and use the virtual connection instruction generated by the enumerated device data to control the virtual camera to complete the connection with the virtual machine; When a camera request instruction issued by the virtual machine is detected, a video request instruction corresponding to the virtual camera is generated according to the camera request instruction, and after the server is controlled to send the video request instruction to the physical camera, callback data of the physical camera is received in real time; The encoder is used to encode the callback data to obtain video stream data corresponding to the physical camera, and the virtual camera is controlled to generate video response data corresponding to the camera request instruction based on the video stream data; Control the server to send the video response data to the virtual machine. 2. The camera interaction method according to claim 1, characterized in that the enumerated device data corresponding to the virtual camera is generated based on the device connection parameters between the physical camera and the client where the encoder is located, comprising: Based on the connection status parameters between the client host providing the working environment of the physical camera and the client where the encoder is located, the device connection parameters between the physical camera and the client where the encoder is located are obtained; When it is detected that the physical camera is in a state of being inserted into the client host, the virtual camera is controlled to send a device descriptor request instruction to the physical camera based on the device connection parameters, and the device descriptor of the physical camera response is received in real time; Control the virtual camera to send a configuration descriptor request instruction to the physical camera, and receive the configuration descriptor responded by the physical camera in real time; The enumerated device data is generated using the device descriptor and the configuration descriptor. 3. The camera interaction method according to claim 1, characterized in that the virtual connection instruction generated by the enumerated device data is used to control the virtual camera to complete the connection with the virtual machine, comprising: Cache one or more descriptor data of a device descriptor, a configuration descriptor, an interface descriptor, an endpoint descriptor, a video head descriptor, a terminal descriptor, a processing unit descriptor, and a video frame descriptor contained in the enumerated device data into a storage unit of the virtual camera; Controlling the server to obtain the descriptor data from the storage unit, and using the descriptor data to generate a device loading instruction corresponding to the virtual camera; The device loading instruction is used to control the virtual camera to be loaded into the virtual machine and perform connection processing. 4. The camera interaction method according to claim 1, characterized in that after controlling the server to send the video request instruction to the physical camera, receiving the callback data of the physical camera in real time, comprises: Acquire the corresponding physical camera based on the video request instruction; Control the server to send the video request instruction to the corresponding physical camera, and control the physical camera to collect and obtain video stream data in real time according to the video request instruction; After converting the video stream data according to the format requirement of the virtual camera, controlling the physical camera to send the video stream data that has completed the format conversion as callback data to the server; Control the server to receive the callback data sent by the physical camera in real time. 5. The camera interaction method according to claim 4, characterized in that the process of encoding the callback data using the encoder to obtain the video stream data corresponding to the physical camera comprises: Acquire initial data corresponding to the physical camera based on the callback data; wherein the initial data is YUV encoded data; After controlling the encoder to perform encoding conversion on the initial data, converted encoded data is obtained; wherein the converted encoded data is one or more encoded data of H265, H264, MJPEG, etc.; The video stream data corresponding to the physical camera is generated according to the converted encoded data. 6. The camera interaction method according to claim 1, characterized in that controlling the virtual camera to generate video response data corresponding to the camera request instruction based on the video stream data comprises: Determining whether the preset storage area of the virtual camera contains historical data corresponding to the video stream data; If yes, the video response data is generated based on the historical data; if no, the video response data is generated based on the video stream data, and the video response data is saved in the storage area as the historical data. 7. The camera interaction method according to claim 1, characterized in that, after detecting a camera closing instruction issued by the virtual machine, the method further comprises: Controlling the virtual machine to send a first closing instruction to the virtual camera, and using the first closing instruction to control the virtual camera to be in a closed state; The server is controlled to send a second shutdown instruction to the physical camera, and the physical camera is controlled to be in a shutdown state using the second shutdown instruction. 8. The camera interaction method according to claim 1, characterized in that when it is detected that the physical camera is in an offline state, the method further comprises: The server is controlled to generate a closing instruction corresponding to the virtual camera, and the server is controlled to send the closing instruction to the virtual machine. 9. A camera interaction system, characterized in that the system is applied to a server including a virtual end and a physical end, the virtual end is provided with a virtual machine and a virtual camera; the physical end includes: a client host, a physical camera and an encoder; the input end of the encoder is connected to the physical camera through the client host, the output end of the encoder is connected to the input end of the virtual camera through the service end of the server; the output end of the virtual camera is connected to the virtual machine; The system comprises: A connection interaction module, configured to generate enumerated device data corresponding to the virtual camera based on device connection parameters between the physical camera and the client where the encoder is located, and control the virtual camera to complete the connection with the virtual machine using a virtual connection instruction generated by the enumerated device data; An instruction interaction module is used to generate a video request instruction corresponding to the virtual camera according to the camera request instruction after monitoring the camera request instruction issued by the virtual machine, and control the server to send the video request instruction to the physical camera, and then receive callback data of the physical camera in real time; A data interaction module, configured to obtain video stream data corresponding to the physical camera after encoding the callback data using the encoder, and control the virtual camera to generate video response data corresponding to the camera request instruction based on the video stream data; The video interaction module is used to control the server to send the video response data to the virtual machine. 10. A server, characterized in that the server comprises a virtual end and a physical end, the virtual end is provided with a virtual machine and a virtual camera; the physical end comprises: a client host, a physical camera and an encoder; the input end of the encoder is connected to the physical camera through the client host, the output end of the encoder is connected to the input end of the virtual camera through the service end of the server; the output end of the virtual camera is connected to the virtual machine; During the process of interacting video data between the virtual camera and the physical camera, the server adopts the steps of the camera interaction method described in any one of claims 1 to 8.