CN116582637A - Screen splitting method of video conference picture and related equipment - Google Patents

Abstract

The disclosure provides a split screen method of a video conference picture and related equipment. The method comprises the following steps: acquiring a target image acquired by an acquisition unit; detecting a target object in the target image; dividing a video conference picture into at least two sub-pictures according to the target object; and correspondingly displaying the target object in the target image in the at least two sub-pictures.

Description

Screen splitting method and related equipment for video conferencing images

technical field

The present disclosure relates to the field of computer technology, and in particular to a method for splitting screens of a video conference screen and related equipment.

Background technique

In video conferencing, face-to-face interaction is crucial. However, when many people are sitting in a conference room for a video conference, since there is only one camera in most conference rooms, and only one video conference screen is collected, it is difficult to distinguish the participants and the number of participants from the video conference screen, especially the Identify the current speaker.

Contents of the invention

The present disclosure proposes a method for splitting screens of a video conference screen and related equipment to solve or partially solve the above-mentioned problems.

In a first aspect of the present disclosure, a method for splitting a video conference screen is provided, including:

Acquiring the target image collected by the acquisition unit;

detecting a target object in the target image;

dividing the video conference screen into at least two sub-pictures according to the target object;

Correspondingly displaying the target object in the target image in the at least two sub-pictures.

The second aspect of the present disclosure provides a split-screen device for a video conference screen, including:

The acquisition module is configured to: acquire the target image collected by the acquisition unit;

a detection module configured to: detect a target object in the target image;

A division module configured to: divide the video conference screen into at least two sub-pictures according to the target object;

The display module is configured to: correspondingly display the target object in the target image in the at least two sub-pictures.

In a third aspect of the present disclosure, a computer device is provided, including one or more processors, memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the Executed by one or more processors, the program includes instructions for performing the method according to the first aspect.

In a fourth aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium containing a computer program. When the computer program is executed by one or more processors, the processors execute the program described in the first aspect. Methods.

A fifth aspect of the present disclosure provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer is made to execute the method described in the first aspect.

The video conference screen splitting method and related equipment provided by the embodiments of the present disclosure, by detecting the target object in the target image collected by the acquisition unit, and then dividing the video conference screen into at least two sub-pictures according to the target object, and dividing the corresponding The target object is correspondingly displayed in the sub-picture, so that when the captured target image contains multiple participants, the video conference screen can be automatically divided into screens, which helps to improve conference room participation during the video conference The sense of interaction between people and other online participants, thereby improving the user experience.

Description of drawings

In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following will briefly introduce the accompanying drawings that need to be used in the descriptions of the embodiments or related technologies. Obviously, the accompanying drawings in the following description are only for the present disclosure Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1A shows a schematic diagram of an exemplary system provided by an embodiment of the present disclosure.

FIG. 1B shows a schematic diagram of video conference images collected in the scene shown in FIG. 1A .

Fig. 2 shows a schematic diagram of an exemplary process according to an embodiment of the present disclosure.

FIG. 3A shows a schematic diagram of an exemplary target image according to an embodiment of the present disclosure.

Fig. 3B shows a schematic diagram of displaying a detection frame in a target image according to an embodiment of the present disclosure.

FIG. 3C shows a schematic diagram of an exemplary video conference screen according to an embodiment of the present disclosure.

Fig. 3D shows a schematic diagram of another exemplary video conference screen according to an embodiment of the present disclosure.

FIG. 3E shows a schematic diagram of a split-screen mode according to an embodiment of the present disclosure.

FIG. 3F shows a schematic diagram of another split-screen mode according to an embodiment of the present disclosure.

Fig. 3G shows a schematic diagram of another exemplary video conference screen according to an embodiment of the present disclosure.

FIG. 3H shows a schematic diagram of face key point detection.

Fig. 3I shows a schematic diagram of a turned human face.

Fig. 4 shows a schematic diagram of an exemplary method provided by an embodiment of the present disclosure.

Fig. 5 shows a schematic diagram of a hardware structure of an exemplary computer device provided by an embodiment of the present disclosure.

Fig. 6 shows a schematic diagram of an exemplary device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure shall have ordinary meanings understood by those skilled in the art to which the present disclosure belongs. "First", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Comprising" or "comprising" and similar words mean that the elements or items appearing before the word include the elements or items listed after the word and their equivalents, without excluding other elements or items. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "Down", "Left", "Right" and so on are only used to indicate the relative positional relationship. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

FIG. 1A shows a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.

As shown in FIG. 1A , the system 100 may include at least one terminal device (eg, terminal devices 102 and 104 ), a server 106 and a database server 108 . The media between the terminal devices 102 and 104 and the server 106 and the database server 108 may include a communication link, for example, a network, and the network may include various connection types, such as wired, wireless communication links or optical fiber cables.

Users 110A-110C can use terminal device 102 to interact with server 106 through the network to receive or send messages, and similarly, user 112 can use terminal device 104 to interact with server 106 through the network to receive or send messages. Various application programs (APPs) can be installed on the terminal devices 102 and 104, for example, video conferencing application programs, reading application programs, video application programs, social networking application programs, payment application programs, web browsers and instant communication tools, etc. In some embodiments, users 110A-110C and user 112 can respectively use video conference applications installed on terminal devices 102 and 104 to use video conference services provided by server 106, and terminal devices 102 and 104 can use The camera (for example, the camera set on the terminal equipment 102 and 104) collects the images 1022 and 1042 and can collect live audio through the microphone (for example, the microphone set on the terminal equipment 102 and 104) and uploads it to the server 106, and then The users 110A-110C and the user 112 can watch each other's picture and hear the other party's voice through the video conferencing application programs on the terminal devices 102 and 104 respectively.

The terminal devices 102 and 104 here may be hardware or software. When the terminal devices 102 and 104 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, e-book readers, MP3 players, laptop computers (Laptop) and desktop Computer (PC) and so on. When the terminal devices 102 and 104 are software, they can be installed in the electronic devices listed above. It can be implemented as a plurality of software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

The server 106 may be a server that provides various services, such as a background server that provides support for various applications displayed on the terminal devices 102 and 104 . The database server 108 may also be a database server that provides various services. It can be understood that, in the case that the server 106 can implement related functions of the database server 108 , the system 100 may not be provided with the database server 108 .

The server 106 and the database server 108 here can also be hardware or software. When they are hardware, they can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When they are software, they may be implemented as a plurality of software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

It should be noted that the method for splitting a video conference screen provided in the embodiment of the present application is generally performed by the terminal devices 102 and 104 .

It should be understood that the numbers of terminal devices, users, servers and database servers in FIG. 1A are only illustrative. According to the realization needs, there can be any number of terminal devices, users, servers and database servers.

FIG. 1B shows a schematic diagram of a video conference picture 120 collected in the scene shown in FIG. 1A .

As shown in FIG. 1B , in the scene shown in FIG. 1A , the video conference screen 120 can display the screens captured by the two cameras in two sub-screens 1202 and 1204 respectively, wherein the video captured by the camera on the side of the terminal device 102 It is the picture of the entire meeting room, therefore, the picture including the target objects (for example, facial images) corresponding to the users 110A-110C will be correspondingly displayed in the sub-picture 1202 .

It can be seen that since the positions of the cameras set in the conference room are fixed, when the relative positions of the users 110A-110C and the cameras are different, the size and orientation of the target objects of the users 110A-110C in the sprite 1202 will be different, and, Since the camera in the meeting room is usually far away from the seats, it is difficult to see each participant clearly in the sprite 1202 . Therefore, a solution for automatically splitting screens captured by a single lens is of great practical significance for improving the interaction between conference rooms and online participants in video conferencing.

In view of this, an embodiment of the present disclosure provides a method for splitting a video conference screen, by detecting a target object in a target image collected by an acquisition unit, and then dividing the video conference screen into at least two sub-pictures according to the target object , and correspondingly display the target object in the target image in the sub-picture, so that when the collected target image contains multiple participants, the video conference picture can be automatically divided into screens. During the video conference, the interaction between conference room participants and other online participants is improved, thereby improving user experience.

FIG. 2 shows a schematic flowchart of an exemplary method 200 provided by an embodiment of the present disclosure. The method 200 can be used to automatically split the screen of the video conference picture. Optionally, the method 200 may be implemented by the terminal devices 102 and 104 in FIG. 1A , and may also be implemented by the server 106 in FIG. 1A . The method 200 implemented by the server 106 will be described below.

As shown in FIG. 2 , the method 200 may further include the following steps.

In step 202, the server 106 may obtain the target image collected by the collection unit. Taking FIG. 1A as an example, the acquisition unit may be a camera installed in the terminal devices 102 and 104, and the target image may be the images 1022 and 1042 collected by the camera. After the camera captures the image, the terminal devices 102 and 104 can upload the captured image to the server 106 for processing.

FIG. 3A shows a schematic diagram of an exemplary target image 300 according to an embodiment of the present disclosure.

The target image 300 may be an image captured by any terminal device participating in the video conference in the system 100. As shown in FIG. 3A, the target image 300 may include target objects 302A- 302C of multiple conference participants.

Next, at step 204 , the server 106 may detect the target object in the target image 300 . As an optional embodiment, an object detection (Object Detection) technology may be used to detect the target object in the target image 300 . Optionally, a pre-trained target detection model can be used to detect the target objects 302A-302C in the target image 300, and detection frames 304A-304C corresponding to the target objects 302A-302C can be obtained, as shown in FIG. 3B.

Furthermore, considering that the position of the participants in the conference room is constantly changing, if the detection frame is fixed after the detection frame is obtained, the participant may not be able to follow the position of the face in time when the participant moves. Therefore, in In some embodiments, the target tracking technology can be used to track the position change of the target object in real time, and correspondingly, the detection frame will also change accordingly. In this way, even if the participant moves during the video conference, it can also be detected. The target object is tracked. Optionally, a pre-trained target tracking model can be used to track the target objects 302A-302C in the target image 300, and the target tracking model can be a real-time face frame and face key point detection and tracking model based on deep learning, Model structures include but are not limited to various forms of Convolutional Neural Networks and various forms of Transformer networks. In this way, through image face detection and tracking, each sub-picture of the split screen can follow the face on the screen in real time.

In some embodiments, after the target object in the target image 300 is detected, step 206 may be entered, and the server 106 may directly determine the split-screen layout based on the target object, and then divide the video conference screen based on the split-screen layout.

FIG. 3C shows a schematic diagram of an exemplary video conference screen 310 according to an embodiment of the present disclosure. As shown in FIG. 3C , the video conference screen 310 is divided into multiple sub-pictures. Specifically, when dividing sub-pictures, in some embodiments, considering that the video conference scene requires multi-party interaction, the total number of target objects in the target images collected by each terminal device currently participating in the video conference can be determined. Divide the video conference screen. For example, taking the scene shown in FIG. 1A as an example, the number of sub-pictures may be determined according to the total number of target objects in the images captured by the terminal devices 102 and 104 respectively, and the number is four in this example. After the number of sprites is determined, a split-screen layout can be determined. There are many types of split screen layouts. For example, in order to maintain the basic split-screen layout of the existing video conference, the video conference screen 310 may be divided into n sub-pictures according to the number n of terminal devices, and then each terminal device is corresponding to the n sub-pictures. Next, according to the number of target objects in the screens collected by each terminal device, the sub-pictures corresponding to the terminal device are further divided, so that a split-screen layout is obtained. As shown in FIG. 3C , there are two left and right sub-pictures respectively corresponding to the images collected by the two terminal devices 102 and 104 , wherein the left picture further includes three sub-pictures corresponding to the target objects 302A- 302C respectively. The sub-picture on the right displays the picture collected by the terminal device 104 and may include the target object 312 . In this way, a complete video conference picture 310 is formed. Moreover, since the sub-pictures corresponding to different terminal devices are divided (for example, divided into equal sizes) in the picture 310, the user can know the specific number of terminals participating in the video conference through the split-screen layout.

It can be understood that, in addition to adopting the split-screen layout of the foregoing embodiments, there may be other split-screen layout manners, for example, dividing sub-pictures directly according to the number of all target objects. FIG. 3D shows a schematic diagram of another exemplary video conference screen 320 according to an embodiment of the present disclosure. As shown in FIG. 3D , the video conference screen 320 is divided into four equal-sized sub-pictures according to the total number of target objects, corresponding to the target objects 302A-302C and the target object 312 respectively. In this way, all participants, whether they are multiple participants in the meeting room or participants alone, can occupy a sprite screen of the same size as others, so that each participant can quickly Clearly interact with each participant.

In some embodiments, when performing split-screen layout, according to the number of target objects, the layout method can also have different options, for example, the screen is divided into equal-sided sub-picture arrays according to the number of layouts (such as , N×N sub-pictures) or an array of unequal-sided sub-pictures (for example, N×M sub-pictures). For example, as shown in FIG. 3D , when the number is 4, 2×2 sub-pictures can be obtained by division. For another example, when the number is 3, the picture 320 can be divided into two rows, one sub-picture is placed in the first row, and two sub-pictures are placed in the second row, as shown in the left picture of FIG. 3C .

It can be understood that, when the number of target objects is larger, the foregoing method may also be used to perform split-screen layout. For example, when the number of target objects is 7, 3×3 sub-pictures may be used to correspond to the target objects, wherein two sub-pictures may be left blank, as shown in FIG. 3E .

In some embodiments, considering that the ratio of the usual display screen is not a square but a rectangle (for example, 16:9), it may be considered to increase the sub-picture in the length direction when performing a split-screen layout quantity. For example, the maximum number of people supported by a video conference screen is 12. When N<5, the screen can be divided into 1 row and N columns. When N≤8, the screen can be divided into 2 rows, and the first row can be divided into N/2 columns. (can be rounded up or down), the second row is N-N/2 columns (can be rounded down or up), when N≤12, the screen can be divided into 3 lines, the first two lines are divided into It is N/3 columns (can be rounded up or down), and the last row is N-N/3×2 columns (can be rounded down or up). The width of each sprite in the last row is the same as that of the previous row, and can be centered. Fig. 3F shows a schematic diagram of another exemplary video conference screen according to an embodiment of the present disclosure. As shown in FIG. 3F , when the number of target objects is 7, the screen can be divided into a 4+3 layout by using the above method.

It can be seen that by detecting the target object in the target image, the video conference screen is divided into at least two sub-pictures according to the number of target objects, and the corresponding target objects are correspondingly displayed in the sub-pictures, so that the captured target image can When there are multiple participants in the conference, the video conference screen is automatically divided into screens, which helps to improve the interaction between the conference room participants and other online participants during the video conference, thereby improving the user experience .

In some embodiments, when performing split-screen layout, in addition to considering the number of target objects, it is also possible to detect whether a participant is speaking, and then make a corresponding response based on the sub-picture corresponding to the detected participant who is speaking. processing. Therefore, as shown in FIG. 2 , the method 200 further includes step 208 of performing speaker detection on the target object. Optionally, this step can be processed in parallel with the target object detection, thereby increasing the processing speed.

Optionally, when it is determined that the target object in the target image is speaking (that is, the participant corresponding to the target object is speaking), an indication may be displayed in the sub-picture corresponding to the speaking participant, for example, for Icon to indicate that a participant is speaking. As shown in FIG. 3D , a microphone-like icon may be displayed on the sub-picture corresponding to the speaking participant, so as to remind other people that the participant is speaking.

As an optional embodiment, the split-screen layout may also be changed according to whether a speaking participant is detected.

Optionally, if it is determined according to the detection result that the target object in the target image is speaking, the video conference screen can be divided into at least two sub-pictures according to the first split-screen mode; if it is determined according to the detection result that all participants of the video conference If no participant speaks, divide the video conference picture into at least two sub-pictures according to the second split-screen mode. The second split-screen mode may be any one of the split-screen layouts in the foregoing embodiments.

Further, dividing the video conference picture into at least two sub-pictures according to the first split-screen mode includes: enlarging and displaying the first sub-picture of the at least two sub-pictures, and displaying the first sub-picture in at least the first sub-picture Other sub-pictures of the at least two sub-pictures are displayed in parallel on one side, and the first sub-picture can be used to display the speaking target object.

FIG. 3G shows a schematic diagram of yet another exemplary video conference screen 330 according to an embodiment of the present disclosure. As shown in FIG. 3G , the screen 330 includes four sub-pictures, corresponding to the target objects 302A-302C and the target object 312 respectively, wherein the first sub-picture 3302 is enlarged and corresponds to the target object 302A of the speaking participant 110A. Correspondingly, other sub-pictures are displayed in parallel on one side of the first sub-picture 3302 . In this way, according to the speaker detection results, the sub-screens of the speaking participants are arranged in the middle of the screen and occupy a larger screen, and the sub-pictures of non-speakers are arranged on the side and occupy a smaller screen, so that Better enhance interactivity.

Like this, when someone speaks, adopt speaker mode (first split-screen mode), in speaker mode, the person who is currently speaking is placed on the largest sub-screen, and the remaining participants are arranged side by side on at least one of the largest sub-screens. Side (when the quantity is large, it can be placed on both sides or more sides). When no one speaks, the normal split-screen mode (the second split-screen mode) is adopted, and each sub-picture has the same size. By selecting different split-screen modes according to the speaker detection results, the interactivity of the video conference can be increased and the user experience can be improved.

Generally, the video stream and audio stream of online participants are independent. In related technologies, video conferencing software can confirm whether someone is speaking in the corresponding video according to the audio stream, but the participants in the conference room are If the video stream and audio stream are shared, the audio stream collected from the terminal device on the side of the meeting room cannot determine whether the person in it is speaking, nor can it tell who is currently speaking, which reduces the interactivity of the meeting.

In view of this, in some embodiments, speaker detection is performed by means of image processing, which can avoid the problem of being unable to distinguish who is currently speaking through the audio stream.

As an optional embodiment, the server 106 may perform key point detection on each detected target object, and then determine whether the participant corresponding to the target object in the target image is speaking according to the key point detection result.

FIG. 3H shows a schematic diagram of face key point detection.

As shown in Figure 3H, the face key point detection can adopt the detection method of 68 key points, and these key points are distributed in various parts of the face, among which, 0-16 points correspond to the chin, and 16-21 points correspond to the right eyebrow (Here is the mirror image, which is the right eyebrow of the person in the picture), 22-26 points correspond to the left eyebrow, 27-35 points correspond to the nose, 36-41 points correspond to the right eye, 42-47 points correspond to the left eye, Points 48-67 correspond to the lips. The human face can be recognized by detecting the key points, and according to the changes of the key points in the target object in consecutive multiple frames, it can be determined whether the corresponding participant is speaking.

It should be noted that the face key point detection method with 68 key points is only an example, and it can be understood that the face key point detection can also have other key point numbers, for example, 21 key points, 29 key points ,etc.

As an optional embodiment, 106 key points may be used to implement key point detection, so that more accurate detection results can be obtained.

In some embodiments, after detecting the key points of the target object; the lip height can be determined based on the key points of the target object, and the lip width can be determined based on the key points of the target object; and then according to the The lip height and the lip width are obtained to obtain the lip aspect ratio of the target object, and then based on the change information of the lip aspect ratio, it is determined whether the target object is speaking. In this way, the speaker detection is realized by means of image processing, which can avoid the problem of being unable to distinguish who is currently speaking through the audio stream.

Further, considering that the rotation of the human face may cause changes in the positions of the key points of the lips, in some embodiments, when performing key point detection, the detected key points may be corrected based on the rotation angle of the human face.

Fig. 3I shows a schematic diagram of a turned human face.

As shown in FIG. 3I , for a human face, in three-dimensional space, there are three rotation angles, yaw angle (Yaw), roll angle (Roll), and pitch angle (Pitch).

As an optional embodiment, affine transformation can be used to cancel the influence of Roll rotation, and then Pitch and Yaw information of face detection can be used to cancel the influence of Pitch and Yaw rotation.

Specifically, multiple key points (for example, coordinates of 106 key points) may be obtained according to the target object detection. Then the key points of the target object are corresponding to the key points (that is, standard key points) of the standard (average) face, so that an affine transformation matrix (mapping relationship) can be obtained, based on applying the affine transformation matrix to Obtain a plurality of key points according to target object detection, just can carry out rolling angle (Roll) correction to described a plurality of key points, and then obtain a plurality of key points after correction, that is, the current detected target object when Roll=0 key point coordinates.

Further, a plurality of first key points and a plurality of second key points respectively corresponding to the lip height and lip width among the corrected key points may be selected, and the pitch angle of the plurality of first key points may be selected. Correction, obtain the corrected lip height, and perform yaw angle correction on the multiple second key points to obtain the corrected lip width, so that the corrected lip height and the corrected lip width cancel out Pitch and Yaw The effect of rotation. As an optional embodiment, taking 106 key points as an example, the length of the line segment between 98 points and 102 points can be calculated to represent the lip height and divided by cos(Pitch) to offset the influence of Pitch, thereby obtaining the corrected lip height. Similarly, the length of the line segment from 96 points to 100 points is calculated to represent the lip width and divided by cos(Yaw) to cancel the influence of Yaw, thereby obtaining the corrected lip width. Among them, the angle information of Pitch and Yaw can be provided by the face detection module.

Then, according to the corrected lip height and the corrected lip width, the key point detection results (detection results for lip height and lip width) can be obtained, so that the corrected lip aspect ratio Indicates how wide the mouth is opened.

Considering that speaking is a dynamic process, relying solely on the aspect ratio of the lips at the current moment may not be able to accurately determine whether the current participant is speaking. Therefore, in some embodiments, the change of the lip aspect ratio within a period of time can be maintained, and the variance of the aspect ratio within the period of time can be used to determine whether the subject is currently speaking.

As an optional embodiment, the lip aspect ratio of the target object may be calculated according to the corrected lip height and the corrected lip width, and the lip aspect ratio may be stored.

Then, when determining whether the participant corresponding to the target object in the target image is speaking according to the key point detection result, the lip aspect ratio of the target object in the target image can be determined in combination with the key point detection result Corresponding change information, determining whether the participant corresponding to the target object in the target image is speaking according to the change information. For example, it can be judged whether the lips are speaking according to whether the variance of the lip aspect ratio within a preset time period (for example, within 1s) is greater than the variance threshold, and if it is greater, it can be judged to be speaking. In this way, when judging whether a person is speaking, the result can be more accurate.

Considering that the speaker may have intermittent pauses when speaking, in order to make the effect of speaking detection more stable, in some embodiments, a counter can be maintained to record the number of times the speaker has been judged to be speaking in the most recent period of time. The number of times, and then judge whether it is speaking according to the size relationship between the number of times and the preset counting threshold. Optionally, determining whether the target object is speaking based on the change information of the lip aspect ratio includes: setting a preset time period (for example, within 2s); counting the lips within the preset time period The number of times the aspect ratio changes; when the number of changes reaches a preset number, it is determined that the target object is speaking. In this way, the timing information of key points is used to enhance the stability of speaker detection in timing and reduce the fluctuation of the detection state.

As an optional embodiment, in response to determining according to the change information that the participant corresponding to the target object in the target image is speaking (that is, the subject in the current frame is determined to be speaking), the count value +1; When it is determined according to the change information that the participant corresponding to the target object in the target image is not speaking (that is, the subject is determined not to be speaking in the current frame), the count value is -1.

Then, it may be determined whether the participant corresponding to the target object in the target image is speaking according to the count value within a preset time period (for example, within 2s). For example, when the value of the counter is greater than the preset count threshold (for example, 2 times), the effect that the subject is speaking (for example, zooming in on the sub-screen corresponding to the speaker and/or displaying the microphone icon) can be displayed; when the value of the counter is less than When the counting threshold is preset, the effect of displaying that the subject is speaking can be canceled (for example, restore the sprite corresponding to the speaker to the same size as other sprites and/or hide the microphone icon).

In this way, the position of the face and lips is determined according to the key point detection of the face after the split screen, and whether the current participant is speaking is judged according to the relative positional relationship of the key points of the lips; The speech detection misjudgment, before the speech judgment, the key points of the face detection will be mapped to the unrotated standard face to reduce the impact of face movement. In addition, the timing information of key points is used to enhance the stability of speaker detection in timing and reduce the fluctuation of detection state.

Considering that there may be errors in speaker judgment by simply using image processing technology, in some embodiments, after determining that the number of changes has reached a preset number, the audio data of the video conference can also be obtained, and then, according to the key point detection results , combined with the audio data of the video conference, determine whether the target object in the target image is speaking. In this way, the accuracy of speaker judgment can be further improved by combining the key point detection result and the audio data of the current video conference. As an optional embodiment, when the pickup for collecting audio is a binaural pickup, the speaker can be located according to two sets of audio data respectively collected by the binaural, thereby further improving the accuracy of speaker judgment.

In related technologies, some video conference automatic screen splitting software determines the position of participants in the current screen through human body detection, and performs cutting and layout according to the body position to realize the function of automatic screen splitting of the software. When a participant enters or leaves the meeting room, it is difficult to achieve real-time split-screen changes.

Therefore, in order to increase or decrease the number of split screens in real time, after the split screen layout is determined, as shown in FIG. 2 , step 210 may be entered to correspond the target object to the detection frame according to the position of the detection frame and the split screen layout.

According to the foregoing embodiments, the target object in the target image may be detected by using target detection or target tracking technology to obtain a detection frame corresponding to the target object, for example, detection frames 304A to 304C in FIG. 3B . In this way, each detected target object can have a corresponding detection frame, and then, the position of each target object in the split-screen layout can be determined according to the split-screen layout, and then the detection frame of the target object and the split-screen corresponding to the position in the layout. In this way, by detecting the target object 300, the relative positional relationship of different human faces is judged through the position of the detection frame of the target object, and the position of the split screen is corresponding to the position of the human face, and then the detection frame is compared with the position of the human face. Corresponding to the split-screen layout, the number of split-screens can be increased or decreased according to the detection results, and the layout of the split-screens can be changed, so that when someone joins or leaves the meeting room, the relative position of the original participant can be kept unchanged, and can realize Increase or decrease the number of split screens in real time.

Optionally, the ROI (Region of Interest) corresponding to each participant in the split-screen layout can be determined according to the coordinates of each participant's detection frame in the original image (target image), so that according to each person's The position of the detection frame and the split-screen layout determine the one-to-one correspondence between the person, the detection frame and the sub-picture.

Then, as shown in FIG. 2, in step 212, the content of the detection frame may be matched with the sub-picture. Optionally, first determine the coordinates of the detection frame corresponding to the target object and the coordinates of the sub-picture corresponding to the target object; then, according to the coordinates of the detection frame corresponding to the target object and the coordinates of the sub-picture corresponding to the target object, the detection frame corresponds to The image of the target object is panned and/or zoomed into the sprite corresponding to the target object. By zooming the target object, all participants can clearly see other people, improving the interactivity of the meeting.

As an optional embodiment, after each detection frame corresponds to the split-screen sub-picture, the detection frame can be expanded to both sides according to a certain ratio (for example, the height is expanded by 20%, and the width is expanded by 40%). The width and height of the detection frame are extended to both sides until its aspect ratio is the same as that of the corresponding sub-screen. If the expanded ROI exceeds the screen boundary, it will be translated to the screen range, so that the content of the detection frame and the sub-screen can be detected. screen matching.

In some embodiments, when the image corresponding to the detection frame is translated and zoomed, the ROI of the sub-picture corresponding to each detection frame at the current moment can be calculated by linear interpolation, and the smooth movement and zooming from the current picture to the target face Effect, realize the horizontal transformation-vertical transformation-zooming function similar to that of a surveillance camera when switching between split-screen effects.

Specifically, assuming that the original coordinates of a sub-picture are represented by the coordinates of the upper left and lower right vertices of the rectangle as (x1_0, y1_0, x2_0, y2_0), the duration of panning and zooming is T, and the ROI of the detection frame of the target face The coordinates of the rectangle are expressed as (x1_T, y1_T, x2_T, y2_T) by the coordinates of the upper left and lower right vertices of the rectangle, then:

First, a time interval Δt can be determined, and then the amount of linear interpolation can be determined according to the translation scaling duration T and the time interval Δt.

Next, the number of linear interpolations determines the update coordinates of the sub-picture corresponding to each interpolation according to the original coordinates of the sub-picture and the coordinates of the ROI of the detection frame. For each interpolation, the updated coordinates may be transformed at equal intervals relative to the coordinates corresponding to the previous interpolation.

Then, according to the update coordinates of the sub-picture corresponding to each interpolation, gradually perform horizontal transformation-vertical transformation-scaling processing on the sub-picture in the manner of equal time intervals until the duration reaches T.

In this way, from turning on the split-screen function to completing the panning and zooming of the sub-picture, there can be a transition time of time T, and then there can be a horizontal transformation-vertical transformation-zooming function similar to a surveillance camera in terms of visual effects, which improves user experience and improves The problem that the switching effect is relatively rigid due to the direct switching of related technologies or the simple scaling of the clipped frame.

In some embodiments, the sharpness of the sub-picture processed in the foregoing manner may be affected. Therefore, super-resolution technology may be used to increase the resolution of the sub-picture, thereby improving the sharpness.

In the related art, there is generally no virtual background function supporting sprites. Embodiments of the present disclosure provide a virtual background function to fill in the gaps.

Therefore, as shown in FIG. 2 , in step 214, it may first be determined whether to enable the virtual background function. If the virtual background function is turned on, then enter step 216, and carry out semantic segmentation based on the target object on the current image (can be processed in parallel with face detection to improve processing efficiency), so that the virtual background function of each split screen can be realized by using the semantic segmentation capability . Optionally, a pre-trained semantic segmentation model can be used to segment the target object from the background image. The semantic segmentation model may be a real-time portrait semantic segmentation model based on deep learning, and the model structure includes but is not limited to various forms of convolutional neural networks (Convolutional Neural Network) and various forms of Transformer networks.

In some embodiments, the portrait segmentation function can be used to perform portrait segmentation on the entire current input image. After the split screen function is enabled, the segmentation result of each sub-screen corresponds to the segmentation result of the whole image under the ROI corresponding to the sub-screen. Then, for each pixel of each sub-screen, according to the value of the segmentation result at the pixel (normalized to [0,1]), the value corresponding to the pixel in the input image, and the new background to be replaced at the The pixel value at the pixel point, calculate the pixel value of the virtual background result at the pixel point.

Specifically, for each pixel, the first value of the segmentation result at the pixel (normalized to [0,1]) can be multiplied by the value corresponding to the pixel in the input image, plus the segmentation result at The second value (1 minus the first value) at the pixel point is multiplied by the pixel value of the new background to be replaced at the pixel point to obtain the pixel value of the virtual background result at the pixel point.

In this way, the process of replacing the real background with the virtual background is completed.

Next, step 218 may be entered to render the video conference picture.

Optionally, each sub-screen can be rendered with the content of the ROI area of the original image corresponding to each sub-screen. If the virtual background function is enabled, the background replacement will be performed in combination with the results of the portrait segmentation and the background to be replaced. If necessary, the detected Any special processing done by the speaker is also done in this step. For example, the target object corresponding to the speaking participant is displayed on the first sub-screen 3302 . A virtual background is displayed in the second sprite 3304 .

Then the processing of the current frame ends, and the next frame process is entered, and the next frame can be processed.

It can be seen from the above embodiments that the embodiment of the present disclosure adopts an automatic screen splitting system for video conferences, and the automatic video splitting function can be used to realize that people sitting in the same conference room can also "face-to-face" with colleagues participating in the conference remotely. communicate. In some embodiments, the speaker detection can conveniently identify who is speaking in the current meeting room, and put the video stream of the speaker in an obvious position to improve the video conference experience. In some scenes, for example, during outdoor live broadcast, sometimes the proportion of the subject in the frame may be very small due to the relationship between the venue and the distance. The PTZ function implemented through the software can realize the automatic tracking and focusing function of the lens without any operation.

It should be noted that the foregoing embodiments are described with the server 106 as the execution subject. In fact, the foregoing processing steps may not be limited to the execution subject. For example, the terminal devices 102 and 104 may also implement these processing steps. Therefore, the terminal device 102 , 104 may also be used as the execution subject of the foregoing embodiments.

The embodiment of the present disclosure also provides a screen splitting method of a video conference picture. FIG. 4 shows a schematic flowchart of an exemplary method 400 provided by an embodiment of the present disclosure. The method 400 can be applied to the server 106 in FIG. 1A , and can also be applied to the terminal devices 102 and 104 in FIG. 1A . As shown in FIG. 4 , the method 400 may further include the following steps.

In step 402, the target image collected by the collection unit is acquired.

Taking FIG. 1A as an example, the acquisition unit may be a camera installed in the terminal devices 102 and 104, and the target image may be the images 1022 and 1042 collected by the camera.

In step 404, target objects (eg, target objects 302A-302C in FIG. 3A ) in the target image (eg, image 300 in FIG. 3A ) are detected.

As an optional embodiment, an object detection (Object Detection) technology may be used to detect the target object in the target image 300 . Optionally, a pre-trained target detection model can be used to detect the target objects 302A-302C in the target image 300, and detection frames 304A-304C corresponding to the target objects 302A-302C can be obtained, as shown in FIG. 3B.

In step 406, the video conference screen is divided into at least two sub-pictures according to the target object.

In step 408, the target object in the target image is correspondingly displayed in the at least two sub-pictures, as shown in FIG. 3C to FIG. 3G .

The video conference screen splitting method provided by the embodiment of the present disclosure, by detecting the target object in the target image collected by the acquisition unit, and then dividing the video conference screen into at least two sub-pictures according to the target object, and dividing the corresponding target object Correspondingly displayed in the sub-picture, so that when the collected target image contains multiple participants, the video conference screen can be automatically divided into screens, which helps to improve the interaction between the participants in the conference room during the video conference. The sense of interaction of other online participants will improve the user experience.

In some embodiments, dividing the video conference screen into at least two sub-pictures according to the number of the target objects includes: determining whether the target object in the target image is speaking; in response to determining whether the target object in the target image The target object is speaking, and the video conference screen is divided into at least two sub-pictures according to the first split-screen mode, as shown in FIG. 3G . Like this, when someone speaks, adopt speaker mode (first split-screen mode), in speaker mode, the person who is currently speaking is placed on the largest sub-screen, and the remaining participants are arranged side by side on at least one of the largest sub-screens. Side (when the quantity is large, it can be placed on both sides or more sides). When no one speaks, the normal split-screen mode (the second split-screen mode) is adopted, and each sub-picture has the same size. By selecting different split-screen modes according to the speaker detection results, the interactivity of the video conference can be increased and the user experience can be improved.

In some embodiments, dividing the video conference screen into at least two sub-pictures according to the first split-screen mode includes: dividing the first sub-picture (for example, sub-picture 3302 in FIG. 3G ) of the at least two sub-pictures Enlarge the display, and display other sub-pictures of the at least two sub-pictures side by side on at least one side of the first sub-picture, and the first sub-picture is used to display the speaking target object.

In this way, according to the speaker detection results, the sub-screens of the speaking participants are arranged in the middle of the screen and occupy a larger screen, and the sub-pictures of non-speakers are arranged on the side and occupy a smaller screen, so that Better enhance interactivity.

In some embodiments, determining whether the target object in the target image is speaking includes: detecting key points of the target object; determining lip height based on the key points of the target object; point, determine the lip width; according to the lip height and the lip width, the lip aspect ratio of the target object is obtained; based on the change information of the lip aspect ratio, it is determined whether the target object is speaking, so that the image can be The speaker detection is performed in a processing way, which can avoid the problem that it is impossible to distinguish who is currently speaking through the audio stream.

In some embodiments, determining whether the target object in the target image is speaking includes: performing key point detection on the target object; determining whether the target object in the target image is speaking according to the key point detection result, thereby Speaker detection can be performed by means of image processing, which can avoid the problem of being unable to distinguish who is currently speaking through the audio stream.

In some embodiments, performing key point detection on the target object includes:

Obtaining a plurality of key points according to the target object detection;

Correcting roll angles for the plurality of key points according to the corresponding relationship between the key points and the standard key points to obtain a plurality of corrected key points;

Selecting a plurality of first key points corresponding to the lip height among the corrected key points, and correcting the pitch angle of the plurality of first key points to obtain the corrected lip height;

selecting a plurality of second key points corresponding to the width of the lips among the plurality of key points after correction, and correcting the yaw angle of the plurality of second key points to obtain the corrected lip width;

The key point detection result is obtained according to the corrected lip height and the corrected lip width.

In order to improve the misjudgment of speech detection caused by face movement, rotation and other actions, the key points of face detection will be mapped to the non-rotated standard face before speech judgment to reduce the impact of face movement.

In some embodiments, the method further includes: calculating the lip aspect ratio of the target object according to the corrected lip height and the corrected lip width, and storing the lip aspect ratio;

Determining whether the target object in the target image is speaking according to the key point detection result includes: combining with the key point detection result, determining change information corresponding to the lip aspect ratio of the target object in the target image, according to the The change information determines whether the target object in the target image is speaking.

In this way, the timing information of key points is used to enhance the stability of speaker detection in timing and reduce the fluctuation of the detection state.

In some embodiments, determining whether the participant corresponding to the target object in the target image is speaking includes: setting a preset time period; counting the number of changes of the lip aspect ratio within the preset time period ; When the number of changes reaches a preset number, determine that the target object is speaking. In this way, the timing information of key points is used to enhance the stability of speaker detection in timing and reduce the fluctuation of the detection state.

In some embodiments, when the number of changes reaches a preset number, determining that the target object is speaking includes: determining that the number of changes reaches a preset number; acquiring audio data of the video conference; combining the audio data of the video conference , to determine that the target object is speaking. In this way, the accuracy of speaker judgment can be further improved by combining the key point detection result and the audio data of the current video conference. As an optional embodiment, when the pickup for collecting audio is a binaural pickup, the speaker can be located according to two sets of audio data respectively collected by the binaural, thereby further improving the accuracy of speaker judgment.

In some embodiments, detecting the target object in the target image includes: using target detection or target tracking technology to detect the target object in the target image, and obtaining a detection frame of the target object;

Correspondingly displaying the target object in the target image in the at least two sub-pictures includes: determining the coordinates of the detection frame corresponding to the target object; determining the coordinates of the sub-picture corresponding to the target object; according to the detection The coordinates of the frame and the coordinates of the sub-picture are used to translate and/or zoom the image corresponding to the detection frame into the sub-picture corresponding to the target object.

By zooming the target object, all participants can clearly see other people, improving the interactivity of the meeting.

In some embodiments, correspondingly displaying the target object in the target image in the at least two sub-pictures further includes: in response to determining that the virtual background function in the sub-picture is turned on, dividing the The target object and the background are segmented to obtain a segmentation result; according to the segmentation result, a virtual background is displayed in the sub-picture, which fills a technical gap in the related art that does not display a virtual background in the sub-picture.

In some embodiments, correspondingly displaying the target object in the target image in the at least two sub-pictures includes: in response to determining that the virtual background function of the second sub-picture of the at least two sub-pictures is turned on, Displaying a virtual background in the second sub-picture (for example, sub-picture 3304 in FIG. 3G ) fills a technical gap in the related art that does not display a virtual background in a sub-picture.

In some embodiments, the method further includes: using semantic segmentation technology to segment the target object and the background in the target image to obtain a segmentation result;

Displaying the virtual background in the second sub-picture includes: displaying the virtual background in the second sub-picture according to the segmentation result.

In this way, the segmentation of the target object and the actual background is realized by using the semantic segmentation technology, and then the replacement of the virtual background is well realized.

In some embodiments, correspondingly displaying the target object in the target image in the at least two sub-pictures further includes: in response to determining that the target object is speaking, displaying in the sub-picture corresponding to the target object The indicator is used to remind others that the participant in the sub-picture corresponding to the icon is speaking, which improves interactivity.

It should be noted that the methods in the embodiments of the present disclosure may be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied in a distributed scenario, and is completed by cooperation of multiple devices. In the case of such a distributed scenario, one of the multiple devices may only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete all described method.

It should be noted that some embodiments of the present disclosure are described above. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from those in the above-described embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

An embodiment of the present disclosure also provides a computer device, configured to implement the above-mentioned method 200 or 400 . FIG. 5 shows a schematic diagram of a hardware structure of an exemplary computer device 500 provided by an embodiment of the present disclosure. The computer device 500 can be used to realize the server 106 in FIG. 1A , and can also be used to realize the terminal devices 102 and 104 in FIG. 1A . In some scenarios, the computer device 500 may also be used to implement the database server 108 of FIG. 1A.

As shown in FIG. 5 , a computer device 500 may include: a processor 502 , a memory 504 , a network module 506 , a peripheral interface 508 and a bus 510 . Wherein, the processor 502 , the memory 504 , the network module 506 and the peripheral interface 508 are connected to each other within the computer device 500 through the bus 510 .

The processor 502 may be a central processing unit (Central Processing Unit, CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application-specific An integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits. Processor 502 may be configured to perform functions related to the techniques described in this disclosure. In some embodiments, processor 502 may also include multiple processors integrated into a single logical component. For example, as shown in FIG. 5, processor 502 may include multiple processors 502a, 502b, and 502c.

Memory 504 may be configured to store data (eg, instructions, computer code, etc.). As shown in FIG. 5 , the data stored in the memory 504 may include program instructions (for example, program instructions for implementing the method 200 or 400 of the embodiment of the present disclosure) and data to be processed (for example, the memory may store configuration files of other modules wait). Processor 502 can also access program instructions and data stored in memory 504 and execute the program instructions to operate on the data to be processed. Memory 504 may include volatile storage or non-volatile storage. In some embodiments, memory 504 may include random access memory (RAM), read only memory (ROM), optical disks, magnetic disks, hard disks, solid state drives (SSD), flash memory, memory sticks, and the like.

The network interface 506 may be configured to provide the computer device 500 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network may be a wired network, a local wireless network (eg, Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It can be understood that the type of the network is not limited to the above specific examples.

The peripheral interface 508 may be configured to connect the computer device 500 with one or more peripheral devices for information input and output. For example, peripheral devices may include input devices such as keyboards, mice, touchpads, touch screens, microphones, and various sensors, and output devices such as displays, speakers, vibrators, and indicator lights.

Bus 510 may be configured to transfer information between various components of computer device 500 (e.g., processor 502, memory 504, network interface 506, and peripheral interface 508), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

It should be noted that although the architecture of the computer device 500 above only shows the processor 502, the memory 504, the network interface 506, the peripheral interface 508 and the bus 510, in the specific implementation process, the architecture of the computer device 500 may also include Other components necessary for proper operation. In addition, those skilled in the art can understand that the architecture of the computer device 500 may only include components necessary to implement the solutions of the embodiments of the present disclosure, instead of all the components shown in the figure.

The embodiment of the present disclosure also provides an interaction device. FIG. 6 shows a schematic diagram of an exemplary device 600 provided by an embodiment of the present disclosure. As shown in FIG. 6, the apparatus 600 may be used to implement the method 200 or 400, and may further include the following modules.

The obtaining module 602 is configured to: obtain the target image collected by the collection unit.

Taking FIG. 1A as an example, the acquisition unit may be a camera installed in the terminal devices 102 and 104, and the target image may be the images 1022 and 1042 collected by the camera.

The detection module 604 is configured to: detect a target object (for example, the target objects 302A-302C in FIG. 3A ) in the target image (for example, the image 300 in FIG. 3A ).

As an optional embodiment, an object detection (Object Detection) technology may be used to detect the target object in the target image 300 . Optionally, a pre-trained target detection model can be used to detect the target objects 302A-302C in the target image 300, and detection frames 304A-304C corresponding to the target objects 302A-302C can be obtained, as shown in FIG. 3B.

The division module 606 is configured to: divide the video conference picture into at least two sub-pictures according to the target object.

The display module 608 is configured to: correspondingly display the target object in the target image in the at least two sub-pictures.

The video conference screen splitting method provided by the embodiment of the present disclosure, by detecting the target object in the target image collected by the acquisition unit, and then dividing the video conference screen into at least two sub-pictures according to the target object, and dividing the corresponding target object Correspondingly displayed in the sub-picture, so that when the collected target image contains multiple participants, the video conference screen can be automatically divided into screens, which helps to improve the interaction between the participants in the conference room during the video conference. The sense of interaction of other online participants will improve the user experience.

In some embodiments, the division module 606 is configured to: determine whether the target object in the target image is speaking; in response to determining that the target object in the target image is speaking, divide the The video conference picture is divided into at least two sub-pictures, as shown in FIG. 3G . Like this, when someone speaks, adopt speaker mode (first split-screen mode), in speaker mode, the person who is currently speaking is placed on the largest sub-screen, and the remaining participants are arranged side by side on at least one of the largest sub-screens. Side (when the quantity is large, it can be placed on both sides or more sides). When no one speaks, the normal split-screen mode (the second split-screen mode) is adopted, and each sub-picture has the same size. By selecting different split-screen modes according to the speaker detection results, the interactivity of the video conference can be increased and the user experience can be improved.

In some embodiments, the division module 606 is configured to: enlarge and display the first sub-picture (for example, the sub-picture 3302 in FIG. 3G ) among the at least two sub-pictures, and at least One side displays other sub-pictures of the at least two sub-pictures side by side, and the first sub-picture is used to display the speaking target object;

The display module 608 is configured to: display the target object corresponding to the speaking participant in the first sub-screen.

In this way, according to the speaker detection results, the sub-screens of the speaking participants are arranged in the middle of the screen and occupy a larger screen, and the sub-pictures of non-speakers are arranged on the side and occupy a smaller screen, so that Better enhance interactivity.

In some embodiments, the detection module 604 is configured to: detect the key points of the target object; determine the lip height based on the key points of the target object; determine the lip width based on the key points of the target object; The lip height and the lip width are used to obtain the lip aspect ratio of the target object; based on the change information of the lip aspect ratio, it is determined whether the target object is speaking, so that the speaker can be determined by image processing Detection can avoid the problem that it is impossible to distinguish who is currently speaking through the audio stream.

In some embodiments, the detection module 604 is configured to: perform key point detection on the target object; determine whether the target object in the target image is speaking according to the key point detection result, so that the Performing speaker detection can avoid the problem of being unable to distinguish who is currently speaking through the audio stream.

In some embodiments, the detection module 604 is configured to:

Obtaining a plurality of key points according to the target object detection;

Correcting roll angles for the plurality of key points according to the corresponding relationship between the key points and the standard key points to obtain a plurality of corrected key points;

Selecting a plurality of first key points corresponding to the lip height among the corrected key points, and correcting the pitch angle of the plurality of first key points to obtain the corrected lip height;

selecting a plurality of second key points corresponding to the width of the lips among the plurality of key points after correction, and correcting the yaw angle of the plurality of second key points to obtain the corrected lip width;

The key point detection result is obtained according to the corrected lip height and the corrected lip width.

In order to improve the misjudgment of speech detection caused by face movement, rotation and other actions, the key points of face detection will be mapped to the non-rotated standard face before speech judgment to reduce the impact of face movement.

In some embodiments, the detection module 604 is configured to: calculate the lip aspect ratio of the target object according to the corrected lip height and the corrected lip width, and store the lip aspect ratio;

Combining with the key point detection result, determine the change information corresponding to the lip aspect ratio of the target object in the target image, and determine whether the participant corresponding to the target object in the target image is speaking according to the change information.

In this way, the timing information of key points is used to enhance the stability of speaker detection in timing and reduce the fluctuation of the detection state.

In some embodiments, the detection module 604 is configured to: set a preset time period; count the number of changes in the lip aspect ratio within the preset time period; when the number of changes reaches a preset number, It is determined that the target object is speaking.

In this way, the timing information of key points is used to enhance the stability of speaker detection in timing and reduce the fluctuation of the detection state.

In some embodiments, the detection module 604 is configured to: determine that the target object is speaking when the number of changes reaches a preset number, including: determining that the number of changes reaches a preset number; acquiring audio of the video conference data; in combination with the audio data of the video conference, it is determined that the target object is speaking. In this way, the accuracy of speaker judgment can be further improved by combining the key point detection result and the audio data of the current video conference. As an optional embodiment, when the pickup for collecting audio is a binaural pickup, the speaker can be located according to two sets of audio data respectively collected by the binaural, thereby further improving the accuracy of speaker judgment.

In some embodiments, the detection module 604 is configured to: use target detection or target tracking technology to detect the target object in the target image, and obtain a detection frame corresponding to the target object;

The display module 608 is configured to: determine the coordinates of the detection frame corresponding to the target object; determine the coordinates of the sub-picture corresponding to the target object; according to the coordinates of the detection frame and the coordinates of the sub-picture, display the The image corresponding to the detection frame is translated and/or zoomed into the sub-picture corresponding to the target object.

By zooming the target object, all participants can clearly see other people, improving the interactivity of the meeting.

In some embodiments, the display module 608 is configured to: in response to determining that the virtual background function in the sub-picture is turned on, segment the target object from the background using a segmentation technique to obtain a segmentation result; according to the segmentation As a result, the virtual background is displayed in the sub-picture, which fills the technical gap in the related art that the virtual background is not displayed in the sub-picture.

In some embodiments, the display module 608 is configured to: in response to determining that the virtual background function of the second sub-picture in the at least two sub-pictures is turned on, on the second sub-picture (for example, the sub-picture in FIG. 3G The virtual background is displayed in the sub-picture 3304), which fills the technical gap in the related art that the virtual background is not displayed in the sub-picture.

In some embodiments, the display module 608 is configured to: use semantic segmentation technology to segment the target object and the background in the target image to obtain a segmentation result; according to the segmentation result, in the second sub-picture Displays a virtual background. In this way, the segmentation of the target object and the actual background is realized by using the semantic segmentation technology, and then the replacement of the virtual background is well realized.

In some embodiments, the display module 608 is configured to: in response to determining that the target object is speaking, display an indicator in the sub-picture corresponding to the target object, so as to remind other people that the target object in the sub-picture corresponding to the icon Participants are speaking, enhancing interactivity.

For the convenience of description, when describing the above devices, functions are divided into various modules and described separately. Of course, when implementing the present disclosure, the functions of each module can be implemented in one or more pieces of software and/or hardware.

The apparatus in the foregoing embodiments is used to implement the corresponding method 400 in any of the preceding embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

Based on the same inventive concept, the present disclosure also provides a non-transitory computer-readable storage medium corresponding to the method in any of the above-mentioned embodiments, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions use To make the computer execute the method 400 described in any one of the above embodiments.

The computer-readable medium in this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the method 200 or 400 described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

Based on the same inventive concept, corresponding to the method 200 or 400 in any of the foregoing embodiments, the present disclosure further provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to execute the method 200 or 400 . Corresponding to the execution subject corresponding to each step in each embodiment of the method 200 or 400, the processor executing the corresponding step may belong to the corresponding execution subject.

The computer program product of the foregoing embodiment is used to cause a processor to execute the method 400 described in any one of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is exemplary only, and is not intended to imply that the scope of the present disclosure (including claims) is limited to these examples; under the idea of the present disclosure, the above embodiments or Combinations between technical features in different embodiments are also possible, steps may be implemented in any order, and there are many other variations of the different aspects of the disclosed embodiments as described above, which are not provided in detail for the sake of brevity.

In addition, for simplicity of illustration and discussion, and so as not to obscure the embodiments of the present disclosure, well-known power/supply circuits associated with integrated circuit (IC) chips and other components may or may not be shown in the provided figures. ground connection. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the embodiments of the disclosure, and this also takes into account the fact that details regarding the implementation of these block diagram devices are highly dependent on the implementation of the embodiments of the disclosure to be implemented. platform (ie, the details should be well within the purview of those skilled in the art). Where specific details (eg, circuits) have been set forth to describe example embodiments of the present disclosure, it will be apparent to those skilled in the art that other applications may be made without or with variations from these specific details. Embodiments of the present disclosure are implemented below. Accordingly, these descriptions should be regarded as illustrative rather than restrictive.

Although the disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations of those embodiments will be apparent to those of ordinary skill in the art from the foregoing description. For example, other memory architectures such as dynamic RAM (DRAM) may use the discussed embodiments.

The disclosed embodiments are intended to embrace all such alterations, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure shall fall within the protection scope of the present disclosure.

Claims (13)

1. A split screen method for video conference pictures, comprising:

acquiring a target image acquired by an acquisition unit;

detecting a target object in the target image;

dividing a video conference picture into at least two sub-pictures according to the target object;

and correspondingly displaying the target object in the target image in the at least two sub-pictures.

2. The method of claim 1, wherein dividing the video conference picture into at least two sub-pictures according to the target object, further comprises:

determining whether a target object in the target image is speaking;

in response to determining that the target object is speaking, the video conference screen is divided into at least two sub-screens in a first split screen mode.

3. The method of claim 2, wherein dividing the video conference picture into at least two sub-pictures in a first split-screen mode comprises:

Magnifying and displaying a first sub-picture in the at least two sub-pictures, wherein the first sub-picture is used for displaying a speaking target object;

and displaying other sub-pictures in the at least two sub-pictures in parallel on at least one side of the first sub-picture.

4. The method of claim 2, wherein determining whether a target object in the target image is speaking comprises:

detecting key points of the target object;

determining a lip height based on the keypoints of the target object;

determining a lip width based on the keypoints of the target object;

obtaining the lip height-width ratio of the target object according to the lip height and the lip width;

based on the information of the change in the lip aspect ratio, it is determined whether the target object is speaking.

5. The method of claim 4, wherein determining whether the target object is speaking based on the change information of the lip aspect ratio comprises:

setting a preset time period;

counting the change times of the lip height-width ratio in the preset time period;

and when the number of changes reaches a preset number, determining that the target object is speaking.

6. The method of claim 5, wherein determining that the target object is speaking when the number of changes reaches a preset number comprises:

Determining that the number of the changes reaches a preset number;

acquiring audio data of a video conference;

in connection with audio data of a video conference, it is determined that the target object is speaking.

7. The method of claim 1, wherein detecting a target object in the target image comprises:

detecting a target object in the target image by utilizing a target detection or target tracking technology to obtain a detection frame of the target object;

correspondingly displaying the target object in the target image in the at least two sub-pictures, wherein the method comprises the following steps:

determining coordinates of a detection frame corresponding to the target object;

determining the coordinates of the sub-picture corresponding to the target object;

and translating and/or scaling the image corresponding to the detection frame into the sub-picture corresponding to the target object according to the coordinates of the detection frame and the coordinates of the sub-picture.

8. The method of claim 1, wherein displaying the target object correspondence in the target image in the at least two sprites further comprises:

responding to the fact that the virtual background function in the sub-picture is opened, and dividing the target object and the background by utilizing a dividing technology to obtain a dividing result;

And displaying a virtual background in the sub-picture according to the segmentation result.

9. The method of claim 1, wherein displaying the target object correspondence in the target image in the at least two sprites further comprises:

and in response to determining that the target object is speaking, displaying an indication identifier in a sub-picture corresponding to the target object.

10. A split screen device for video conference pictures, comprising:

an acquisition module configured to: acquiring a target image acquired by an acquisition unit;

a detection module configured to: detecting a target object in the target image;

a partitioning module configured to: dividing a video conference picture into at least two sub-pictures according to the target object;

a display module configured to: and correspondingly displaying the target object in the target image in the at least two sub-pictures.

11. A computer device comprising one or more processors, memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the programs comprising instructions for performing the method of any of claims 1-9.

12. A non-transitory computer readable storage medium containing a computer program which, when executed by one or more processors, causes the processors to perform the method of any of claims 1-9.

13. A computer program product comprising computer program instructions which, when run on a computer, cause the computer to perform the method of any of claims 1-9.